WAF is able to multithread tasks in subdirectories

Short presentation

WAF is a python framework for configuring, compiling and installing applications. One of the feature of WAF is the automatic parallelization of tasks.

Context method recurse

The recurse context method is used to call a script in a subdirectory (or subproject).

def build(bld):
  bld.recurse('sub_project_1')
  bld.recurse('sub_project_2')

WAF tool parallel_debug

The WAF tool parallell_debug is used to obtain detailed view of WAF execution. To use it in wscript :

def options(ctx):
  ctx.load('parallel_debug')

def configure(ctx):
  ctx.load('parallel_debug')

If WAF complains about missing tool parallel_debug, you must add the tool to your waf binary :

python waf-light –tools=parallell_debug,<your_favorite_tool>

Test recurse method and multithreading

A directory contains a top wscript and 3 subdirectories 1, 2 and 3. Each of them containing a wscript and some subdirectories.

The top wscript contains :

def options(opt):
  opt.load('parallel_debug')

def configure(conf):
  conf.load('parallel_debug')

def build(bld):
  bld(rule="sleep 1", color="BLUE", name=". sleep")
  bld.recurse("1")
  bld.recurse("2")
  bld.recurse("3")

A subdirectory wscript contains :

def build(bld):
  bld(rule="sleep 1", color="YELLOW", name="2 sleep")
  bld.recurse("1")
  bld.recurse("2")
  bld.recurse("3")

Issuing a :

waf distclean configure build -j4

produces an image :

On 4 threads, waf is able to parallelize each sleep task on subdirectories !