Contains the core data pipeline resource setup in AWS which includes Glue, SQS and various S3 Buckets.

From the quest-data-pipeline/ directory, run the following commands:

npm install -g cdk
npm run cdk deploy -- --parameters bucketNamePrefix=<prefix>

If the aws account you're deploy to was never boostrapped before, run the following command before deploying:

npm run cdk bootstrap aws://<account-id>/us-east-1

Contains resources for bootstrapping a new vendor allowing the aws account to upload files to the data pipeline account.

From the quest-vendor-bootstrap/ directory, run the following commands:

npm install -g cdk
npm run cdk deploy -- --parameters accountId=<account> --parameters vendorName=<vendor>

Examples are located in the scripts/ directory. The following command can be used for uploading a file to the data pipeline (replacing parameters with actual resource names/urls):

sh uploadFile.sh test-vendor test-vendor-bucket https://sqs.us-east-1.amazonaws.com/1234567890/test-vendor-queue.fifo

To see a very basic example using connectors, execute:

sh uploadCSV.sh test-vendor test-vendor-bucket $replace_with_QuestDataPipelineStack.csvqueueUrl

The corresponding job will connect and transform the file from .csv to .json