oxigraph-sparql-api-serverless's Introduction

oxigraph-sparql-api-serverless

This is the rust program which can deploy it as Web API(SPARQL Endpoint) using Oxigraph on AWS Lambda and API Gateway.

Build using Docker

$ git clone https://github.com/uedayou/oxigraph-sparql-api-serverless.git --recursive
$ cd oxigraph-sparql-api-serverless
$ cp YOUR_TURTLE_FILE_PATH rdf/dump.ttl 
$ docker image build -t oxigraph-build -f Dockerfile.build .
$ docker container run --rm -v $PWD:/code -v $HOME/.cargo/registry:/root/.cargo/registry -v $HOME/.cargo/git:/root/.cargo/git oxigraph-build

Deploy on AWS using SAM CLI

$ sam package --template-file template.yaml --output-template-file packaged.yaml --s3-bucket S3_BUCKET_NAME
$ sam deploy --template-file packaged.yaml --stack-name PROJECT_NAME --capabilities CAPABILITY_IAM

How to use it as SPARQL Endpoint

https://${ID}.execute-api.${Region}.amazonaws.com/Prod/sparql?query={sparql query with URL-encoded}&format={json(default) or xml}

oxigraph-sparql-api-serverless's People

Contributors

Stargazers

Watchers

oxigraph-sparql-api-serverless's Issues

Lambda package limit / パッケージの250MB制限

If the dump.ttl gets too big, it seems to easily hit the 250MB limit.

If we use Deploy by Container Image, it seems to reduce the limit to 10GB. How can I rewrite the code?

dump.ttl が大きくなると簡単に250MB制限に引っかかってしまうようです．

コンテナイメージによるデプロイを利用すれば，制限が10GBまで緩和されるようですが，どのようにコードを書き換えればいいでしょうか？

（参考）私の書いたコード

cf. .github/workflows/deploy_to_lambda.yml

name: Deploy to AWS Lambda
on:
  # Triggers the workflow on push or pull request events but only for the main branch
  push:
    branches: [ main ]
    paths:
      - 'data/turtle/**'

  # Allows you to run this workflow manually from the Actions tab
  workflow_dispatch:

# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
  Deploy-to-AWS-Lambda:
    runs-on: ubuntu-latest
    # Steps represent a sequence of tasks that will be executed as part of the job
    steps:
      # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
      - uses: actions/checkout@v2
      - uses: actions/setup-python@v2

      - name: install AWS SAM CLI
        run: |
          curl -OL https://github.com/aws/aws-sam-cli/releases/latest/download/aws-sam-cli-linux-x86_64.zip
          unzip aws-sam-cli-linux-x86_64.zip -d sam-installation
          sudo ./sam-installation/install
          sam --version

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v1
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: ${{ secrets.AWS_REGION }}

      # ttl をまとめて，dump.ttl として出力
      - name: Combine turtle files
        run: |
          pip install tqdm
          python dump.py

      # oxigraph-sparql-api-serverlessでコンパイルする
      - name: Clone repository
        run: |
          git clone https://github.com/uedayou/oxigraph-sparql-api-serverless.git --recursive
          mv dump.ttl oxigraph-sparql-api-serverless/rdf/dump.ttl 

      # pull docker image from dockerhub and compile
      - name: compile
        run: |
          docker pull ningensei848/oxigraph-sparql-api-serverless:latest
          docker container run --rm -v $PWD:/code -v $HOME/.cargo/registry:/root/.cargo/registry -v $HOME/.cargo/git:/root/.cargo/git ningensei848/oxigraph-sparql-api-serverless
        working-directory: oxigraph-sparql-api-serverless

      - name: Finally, deploy to lambda by AWS SAM CLI
        run: |
          sam package --template-file template.yaml --output-template-file packaged.yaml --s3-bucket $AWS_BUCKET_NAME --region $AWS_REGION
          sam deploy --template-file packaged.yaml --stack-name ML4Keiba --capabilities CAPABILITY_IAM --region $AWS_REGION
        env:
          AWS_REGION: ${{ secrets.AWS_REGION }}
          AWS_BUCKET_NAME: ${{ secrets.AWS_BUCKET_NAME }}
        working-directory: oxigraph-sparql-api-serverless

On last step: Finally, when I run deploy to lambda by AWS SAM CLI, I get the following error.

最後のステップ：Finally, deploy to lambda by AWS SAM CLI を実行すると以下のエラーが出る

Error: Failed to create/update the stack: ML4Keiba, Waiter StackUpdateComplete failed: Waiter encountered a terminal failure state: For expression "Stacks[].StackStatus" we matched expected path: "UPDATE_ROLLBACK_COMPLETE" at least once

Resource handler returned message: "Unzipped size must be smaller than 262144000 bytes (Service: Lambda, Status Code: 400, Request ID: XXXXXXX YYYYYYYY ZZZZZZZZ , Extended Request ID: null)" (RequestToken: XXXXX-YYYYY-ZZZZZ-WWWWW , HandlerErrorCode: InvalidRequest)

Recommend Projects