AWS Lambda interface for Julia



AWS Lambda Interface for Julia

Build Status

Start by creating a basic AWSCore configuration...

using AWSCore
using AWSLambda
aws = aws_config(region = "us-east-1")

Deploy the base jl_lambda_eval function...


Execute a function on Lambda...

@lambda_call(aws, readdir)("/var/task/bin")

1-element Array{ByteString,1}:

@lambda_call(aws, sum)([1,2,3])


Evaluate an expression on Lambda...

@lambda_eval aws ENV["LAMBDA_TASK_ROOT"]


@lambda_eval aws run(`ls -l /var/task/bin/julia`)

-rwxr-xr-x 1 slicer 497 41748 Apr 20 05:33 /var/task/bin/julia

Evaluate an expression on Lambda...

@lambda_eval aws begin
    l = open(readlines, "/proc/cpuinfo")
    l = filter(i->ismatch(r"^model name", i), l)
    strip(split(l[1], ":")[2])

"Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz"

Deploy a Lambda function to count prime numbers...

λ = @λ aws function count_primes(low::Int, high::Int)
    count = length(primes(low, high))
    println("$count primes between $low and $high.")
    return count

The @λ macro creates an AWS Lambda named "count_primes". It wraps the body of the function with serialisation/deserialistion code and packages it into a .ZIP file. The .ZIP file deployed to AWS. The @λ macro returns an anonymous function that can be called to invoke the Lambda.

Run 20 instances of λ in parallel using amap()...

function count_primes(low::Int, high::Int)
    w = 500000000
    counts = amap(λ, [(i, min(high,i + w)) for i = low:w:high])
    count = sum(counts)
    println("$count primes between $low and $high.")
    return count

@test count_primes(10, 10000000000) == 455052507

Publish a verion with an alias...

lambda_publish_version(aws, "count_primes", "PROD")

Call a deployed Lambda by name with named arguments...

r = invoke_lambda(aws, "count_primes", low = 10, high = 100)
@test r[:jl_data] == "21"

Arguments and result are transfered as JSON.

Call a deployed Lambda and don't wait for the result...

r = async_lambda(aws, "count_primes", @SymDict(low = 10, high = 100))

Arguments are transfered as JSON.

Call a deployed Lambda by name...

r = invoke_jl_lambda(aws, "count_primes", 10 = 100)
@test r == 21

Arguments and result are transfered as serialised Julia objects.

Create a local module TestModule/TestModule.jl...

module TestModule

export test_function


test_function(x) = x * x


Use the module in a Lambda...

push!(LOAD_PATH, "TestModule")

λ = @λ aws function lambda_test(x)

    # Check that precompile cache is being used...
    @assert !Base.stale_cachefile("/var/task/TestModule/TestModule.jl",
                                  Base.LOAD_CACHE_PATH[1] * /TestModule.ji")
    using TestModule
    return test_function(x)

@test λ(4) == 16

The @λ macro sees the `julia-observer-quote-cut-paste-4workstatement and bundles the correspondingjulia-observer-quote-cut-paste-5work` files into the deployment .ZIP file. It then does a dry-run invocation to trigger module precompilation. The resulting '.ji' files are retrieved from the Lambda sandbox and added to the deployment .ZIP file. Subsequent calls do not have to wait for compilation.

First Commit


Last Touched

about 2 months ago


145 commits

Used By: