dummy-link

ChunkFlow

Distributed pipeline inside and beyond the cloud

Readme

ChunkFlow.jl ChunkFlow logo

Build Status

Note that this package was deprecated in favor of python chunkflow.

Introduction

the computation was defined as a node. The kinds of node was defined in a task file

Usage

Find help

julia main.jl --help
julia main.jl -h

Running pipeline using local task configuration json file

julia main.jl --task=task.json --deviceid=0
julia main.jl -t task.json -d 0

Using AWS Simple Queue Service

Produce a bunch of tasks

  • define task.json file, the file name can be only a prefix. Task producer will match all the files and produce a bunch of tasks
  • produce tasks: julia taskproducer.jl --task=task.json --awssqs=chunkflow-tasks

find help:

julia taskproducer.jl -h

Fetch tasks from AWS SQS

launch a number of instances to deal with the tasks. run the same commands in each instance

julia -j number_of_processes main.jl -n number_of_processes -w wait_seconds_of_each_launch -q queue-name

First Commit

10/19/2015

Last Touched

4 months ago

Commits

2050 commits

Used By: