swift-serverless / aws-lambda-swift-sprinter Goto Github PK

Provide a Docker file to build Swift 5.0.3 on Docker amazonlinux:2018.03
The AWS Lambda Custom Runtime for Swift should be built on Docker amazonlinux:2018.03, the current solution uses the Swift docker images based on Ubuntu.

Solving this could help to:

• Reduce AWS Lambda Custom Runtime for Swift size
• Could solve the HTTPS issue with the CURL library
• Improve the debugging
• Improve the deployment

The solution is a fix to this broken Dockerfile:

Dockerfile:

FROM amazonlinux:2018.03 as builder

RUN yum update && yum -y group install "development tools" && \
    yum -y install ninja-build clang python \
    libuuid-devel libicu-devel libicu-devel libstdc++-devel \
    libedit-devel libxml2-devel sqlite-devel swig python26-devel ncurses-devel \
    pkgconfig openssl-devel systemtap-sdt-devel tzdata rsync git wget

#cmake
WORKDIR /tmp
RUN wget https://cmake.org/files/v3.15/cmake-3.15.0.tar.gz
RUN tar -xvf cmake-3.15.0.tar.gz
WORKDIR /tmp/cmake-3.15.0
RUN ./bootstrap
RUN make
RUN make install

RUN mkdir /swift-source
WORKDIR /swift-source
RUN git clone https://github.com/apple/swift.git
RUN ./swift/utils/update-checkout --clone
RUN ./swift/utils/update-checkout --tag swift-5.0.3-RELEASE
RUN ./swift/utils/build-script --preset=buildbot_linux installable_package=$ROOT_DIR/swift-package.tar.gz install_destdir=/tmp/swift

Error:

clang: error: unable to execute command: Killed
clang: error: linker command failed due to signal (use -v to see invocation)
[1683/3385] Building CXX object tools/llvm-lto2/CMakeFiles/llvm-lto2.dir/llvm-lto2.cpp.o
ninja: build stopped: subcommand failed.
./swift/utils/build-script: fatal error: command terminated with a non-zero exit status 1, aborting

Pull request details
The Docker file must be under the path docker/amazonlinux-2018.03

When the connection is slow it's hard to upload the package and the lambda

make upload_lambda_layer could fail because the connection is slow.

TODO:

• Add a new command upload_lambda_layer_with_s3 to use the s3 bucket to update the Lambda layer
• Add a new command create_lambda_with_s3 to create the lambda with the code and layer on s3
• Add a new command update_lambda_with_s3 to update the lambda from S3
• Fix the upload_build_to_s3 to upload only required files.

Describe the bug
make create_lambda doesn't behave correctly.

To Reproduce
make create_lambda

Expected behaviour

• It should create the execution role if doesn't exist.
• It should package the lambda

Add support for Travis CI
The minimal support requires to test that each example builds correctly.

Add support for Visual Studio Code Debugging
The minimal support requires:

• Enable the debugging inside Docker
• Enable the Unit Testing inside Docker
• Enable the code coverage inside Docker

Note the documentation for this will be added later

We want to provide some context on the internal performance when the Lambda is Warm

HelloWorld example:

import AsyncHTTPClient
import Foundation
#if canImport(FoundationNetworking)
    import FoundationNetworking
#endif
import LambdaSwiftSprinter
import LambdaSwiftSprinterNioPlugin

enum MyError: Error {
    case unableToConvertString
}

class Lambda: LambdaHandler {
    func commonHandler(event: Data, context: Context) -> LambdaResult {
        guard let data = "Hello World!".data(using: .utf8) else {
            return .failure(MyError.unableToConvertString)
        }
        return .success(data)
    }
}

let lambda = Lambda()

public func log(_ object: Any, flush: Bool = false) {
    fputs("\(object)\n", stderr)
    if flush {
        fflush(stderr)
    }
}

do {
    let sprinter = try SprinterNIO()
    sprinter.register(handler: "helloWorld", lambda: lambda)
    try sprinter.run()
} catch {
    log(String(describing: error))
}

Note Data and Context are not used, but are being passed to the lambda function.

Using AWS Lambda with 256MB the warm performance is around ~1.19ms.

We want to evaluate where the execution time is spent

The main loop when the lambda is warm is contained in the following file:
https://github.com/swift-sprinter/aws-lambda-swift-sprinter-core/blob/master/Sources/LambdaSwiftSprinter/Sprinter.swift

public func run() throws {
        while !cancel {
            let (event, responseHeaders) = try apiClient.getNextInvocation()
            counter += 1

            if let lambdaRuntimeTraceId = responseHeaders.rhk(key: .runtimeTraceId) {
                setenv(Context.AWSEnvironmentKey.xAmznTraceId.rawValue, lambdaRuntimeTraceId, 0)
            }

            guard let lambda = lambdas[handlerName] else {
                try apiClient.postInitializationError(error: SprinterError.missingEnvironmentVariables(.handler))
                return
            }

            let context = try Context(environment: environment, responseHeaders: responseHeaders)
            let result = lambda.commonHandler(event: event, context: context)

            switch result {
            case .success(let outputData):
                try apiClient.postInvocationResponse(for: context.awsRequestId, httpBody: outputData)
            case .failure(let error):
                try apiClient.postInvocationError(for: context.awsRequestId, error: error)
            }
        }
    }

The getNextInvocation() is implemented in https://github.com/swift-sprinter/aws-lambda-swift-sprinter-nio-plugin/blob/master/Sources/LambdaSwiftSprinterNioPlugin/LambdaApiNIO.swift

public func getNextInvocation() throws -> (event: Data, responseHeaders: [AnyHashable: Any]) {
        let result = try httpClient.execute(
            request: _nextInvocationRequest,
            deadline: nil
        ).wait()

        let httpHeaders = result.headers

        guard result.status.isValid() else {
            throw SprinterNIOError.invalidResponse(result.status)
        }

        if let body = result.body,
            let data = body.getData(at: 0,
                                    length: body.readableBytes,
                                    byteTransferStrategy: .noCopy) {
            return (event: data, responseHeaders: httpHeaders.dictionary)
        } else {
            throw SprinterNIOError.invalidBuffer
        }
    }

Note that the Lambda remains in waiting for the next event while calling getNextInvocation() in the run loop until is killed or the timeout of 3600s is triggered.

We want understand if some optimisation can improve the performance
Reference Golang with a performance at warm of ~0.44 ms.

package main

import "github.com/aws/aws-lambda-go/lambda"

type Response struct {
	StatusCode      int    `json:"statusCode"`
	Body            string `json:"body"`
	IsBase64Encoded bool   `json:"isBase64Encoded"`
}

func hello() (Response, error) {
	return Response{
		StatusCode:      200,
		Body:            "Hello world",
		IsBase64Encoded: false,
	}, nil
}

func main() {
	lambda.Start(hello)
}

Describe the bug
While sample code of syncLambda runs properly. Sample code for asyncLambda: AsyncCodableLambda throws following error

aws lambda invoke --function-name HelloWorld --profile default --payload "fileb://Examples/HelloWorld/event.json" ./.build/tmp/outfile && echo "\nResult:" && cat ./.build/tmp/outfile && echo "\n"
{
"FunctionError": "Unhandled",
"ExecutedVersion": "$LATEST",
"StatusCode": 200
}

Result:
{"errorType":"Runtime.ExitError","errorMessage":"RequestId: 3278a4df-68cb-45d0-afe9-8870a675dae5 Error: Runtime exited without providing a reason"}

To Reproduce
Steps to reproduce the behavior:
Clone this repository and change following line

//    sprinter.register(handler: "helloWorld", lambda: syncLambda)
//    sprinter.register(handler: "helloWorld2", lambda: syncDictLambda)
//    sprinter.register(handler: "helloWorld3", lambda: asyncLambda)
    sprinter.register(handler: "helloWorld4", lambda: asyncDictLambda)

Inside Example/HelloWorld/Sources/HelloWorld/Main.swift

Follow all the steps to make docker build, layer, lambda from terminal
Run make invoke_lambda
Expected behavior
Sample code given in asyncDictLambda: AsyncDictionaryLambda or asyncLambda: AsyncCodableLambda should run without any error

Desktop (please complete the following information):

• OS: MacOS 10.14.4

Security Update
https://support.apple.com/en-us/HT210647

The Makefile should allow to pass parameters from command line.

Investigation
It's required to better understand, in terms of performance, the AWS Lambda Custom Runtime with Swift.

It's required to provide solution to reduce the Initalization time and the Overhead time that you can monitor when X-Ray is enabled.

It could be helpful if you can comment this with ideas.

@Andrea-Scuderi - came over to your repo from your earlier update from tonisuter/aws-lambda-swift.

Your doc says AWS Linux does not support Swift ... but Ubuntu does.

Do you have a quick start to upload the Layer into Lambda to start using this without all the building required before use ?

Provide a Docker file to build Swift 5.1 on Docker amazonlinux:2018.03
The AWS Lambda Custom Runtime for Swift should be built on Docker amazonlinux:2018.03, the current solution uses the Swift docker images based on Ubuntu.

Solving this could help to:

• Reduce AWS Lambda Custom Runtime for Swift size
• Could solve the HTTPS issue with the CURL library
• Improve the debugging
• Improve the deployment

The solution is a fix to this broken Dockerfile:

Dockerfile:

FROM amazonlinux:2018.03 as builder

RUN yum update && yum -y group install "development tools" && \
    yum -y install ninja-build clang python \
    libuuid-devel libicu-devel libicu-devel libstdc++-devel \
    libedit-devel libxml2-devel sqlite-devel swig python26-devel ncurses-devel \
    pkgconfig openssl-devel systemtap-sdt-devel tzdata rsync git wget

#cmake
WORKDIR /tmp
RUN wget https://cmake.org/files/v3.15/cmake-3.15.0.tar.gz
RUN tar -xvf cmake-3.15.0.tar.gz
WORKDIR /tmp/cmake-3.15.0
RUN ./bootstrap
RUN make
RUN make install

RUN mkdir /swift-source
WORKDIR /swift-source
RUN git clone https://github.com/apple/swift.git
RUN ./swift/utils/update-checkout --clone
RUN ./swift/utils/update-checkout --tag swift-5.1-RELEASE
RUN ./swift/utils/build-script --preset=buildbot_linux installable_package=$ROOT_DIR/swift-package.tar.gz install_destdir=/tmp/swift

Pull request details
The Docker file must be under the path docker/amazonlinux-2018.03

I am using this example aws server-less environment.

I am able to deploy the code with given example on aws lambda. Now to use this in our actual project I need to use postgres and redis with swift.

I am unable to find example where i can use standalone postgresql with server-side swift without using other server framework like Vapor.

Can you please help us in finding the right-way to integrate swift example with postgres and redis.

Thank you.

Is your feature request related to a problem? Please describe.
Support the new release of swift 5.1

Describe the solution you'd like
The solution must be compatible with swift 5.*

Evaluate the performance of this library by comparing it with different languages
We need to compare the same example 'HelloWorld' by running it with Lambda using the same configuration but with different languages.

Swift, C#, Java, Go, NodeJS, Python, Ruby, PowerShell

Wanted solution
We need a CloudFormation or SAM template to build the test environment.
We need the code to run the performance test and obtain a csv or a JSON file with the results.
We need a tool to show the graphics of the result.
We should be able to change the example to evaluate the performances with a more complex one.
The solution should be automated.

Test Lambda invocation with Lambci and LocalStack
Add an automated test to check that the code works correctly with amazonlinux.

Add commands to test code locally

make build_lambda_local: Build the lambda and copy the artefacts under $(LOCAL_LAMBDA_PATH)

make invoke_lambda_local_once: Invoke the Lambda locally once with Docker and LambCI on port 9001

make start_lambda_local_env: Start the local environment with Docker and LambCI on port 9001

make invoke_lambda_local: Invoke the lambda locally using the endpoint `http://localhost:9001'

Use of Docker Compose
Add command in Makefile to build and test the code locally by using docker-compose on port 9001.

Add commands for local environment

make start_docker_compose_env: Start the docker-compose test environment

make stop_docker_compose_env: Stop the docker-compose test environment

make test_lambda_local_output: Verify the the output of the local invoke is the expected one.

Add the tests to Travis CI

Add tests for each Example

Update async-http-client to version 1.0.0

Improve the Lambda Handler for NIO Implementation
Add new lambda handlers to rely directly on SwiftNIO 2.0 EventLoop.

public typealias SyncCodableNIOLambda<Event: Decodable, Response: Encodable> = (Event, Context) throws -> EventLoopFuture<Response>
public typealias SyncDictionaryNIOLambda = ([String: Any], Context) throws -> EventLoopFuture<[String: Any]>
public typealias AsyncCodableNIOLambda<Event: Decodable, Response: Encodable> = (Event, Context, @escaping (Result<Response, Error>) -> Void) -> Void
public typealias AsyncDictionaryNIOLambda = ([String: Any], Context, @escaping (DictionaryResult) -> Void) -> Void