🔍 Data pipeline for crawling PDFs from the Web and transforming their contents into structured data using AWS Textract. Built with AWS CDK + TypeScript.

This is an example data pipeline that illustrates one possible approach for large-scale serverless PDF processing - it should serve as a good foundation to modify for your own purposes.

Getting Started

Run the following commands to install dependencies, build the CDK stack, and deploy the CDK Stack to AWS.

yarn install
yarn build
cdk bootstrap
cdk deploy

The following is an overview of each process performed by this CDK stack.

1. Scrape PDF download URLs from a website

   Scraping data from the COGCC website.

2. Store PDF download URL in DynamoDB

   

3. Download the PDF to S3

   A lambda fires off when a new PDF download URL has been created in DynamoDB.

4. Process the PDF with AWS Textract

   Another lambda fires off when a PDF has been downloaded to the S3 bucket.

5. Process the AWS Textract results

   When an SNS event is detected from AWS Textract, a lambda is fired off to process the result.

6. Save the processed Textract result to DynamoDB.

   After the full result is pruned down the the desired datastructure, we save the data in DynamoDB.

• yarn install - installs dependencies
• yarn build - builds the production-ready CDK Stack
• yarn test - runs Jest
• cdk bootstrap - bootstraps AWS Cloudformation for your CDK deploy
• cdk deploy - deploys the CDK stack to AWS

Notes

• If a PDF download URL has already been added to the pdfUrlsTable DynamoDB table, the pipeline will not re-execute for the PDF.

• Includes tests with Jest.

• Recommended to use Visual Studio Code with the Format on Save setting turned on.

Built with

• TypeScript
• Jest
• Puppeteer
• AWS CDK
• AWS Lambda
• AWS SNS
• AWS DynamoDB
• AWS S3

Additional Resources

• CDK API Reference
• Puppeteer
• Puppeteer Lambda
• CDK TypeScript Reference
• CDK Assertion Package
• Textract Pricing Chart
• awesome-cdk repo

License

Opens source under the MIT License.

Built with ❤️ by aeksco