dummy-link

ChunkFlow

Distributed pipeline inside and beyond the cloud

Readme

ChunkFlow.jl ChunkFlow logo

Build Status

Introduction

the computation was defined as a node. The kinds of node was defined in a task file

Usage

Find help

julia main.jl --help
julia main.jl -h

Running pipeline using local task configuration json file

julia main.jl --task=task.json --deviceid=0
julia main.jl -t task.json -d 0

Using AWS Simple Queue Service

Produce a bunch of tasks

  • define task.json file, the file name can be only a prefix. Task producer will match all the files and produce a bunch of tasks
  • produce tasks: julia taskproducer.jl --task=task.json --awssqs=chunkflow-tasks

find help:

julia taskproducer.jl -h

Fetch tasks from AWS SQS

launch a number of instances to deal with the tasks. run the same commands in each instance

julia -j number_of_processes main.jl -n number_of_processes -w wait_seconds_of_each_launch -q queue-name

First Commit

10/19/2015

Last Touched

2 months ago

Commits

2047 commits

Used By: