Distributed pipeline inside and beyond the cloud


ChunkFlow.jl ChunkFlow logo

Build Status

Note that this package was deprecated in favor of python chunkflow.


the computation was defined as a node. The kinds of node was defined in a task file


Find help

julia main.jl --help
julia main.jl -h

Running pipeline using local task configuration json file

julia main.jl --task=task.json --deviceid=0
julia main.jl -t task.json -d 0

Using AWS Simple Queue Service

Produce a bunch of tasks

  • define task.json file, the file name can be only a prefix. Task producer will match all the files and produce a bunch of tasks
  • produce tasks: julia taskproducer.jl --task=task.json --awssqs=chunkflow-tasks

find help:

julia taskproducer.jl -h

Fetch tasks from AWS SQS

launch a number of instances to deal with the tasks. run the same commands in each instance

julia -j number_of_processes main.jl -n number_of_processes -w wait_seconds_of_each_launch -q queue-name

First Commit


Last Touched

4 months ago


2050 commits

Used By: