• Features
• Precautions
• Deploy
• Clean up
• Authors
• Credits
• License

This solution provides ready-to-use code so you can start experimenting with a variety of Large Language Models, settings and prompts in your own AWS account.

Supported model providers:

• Amazon Bedrock
• Amazon SageMaker self-hosted models from Foundation, Jumpstart and HuggingFace.
• Third-party providers via API such as Anthropic, Cohere, AI21 Labs, OpenAI, etc. See available langchain integrations for a comprehensive list.

A workspace is a logical namespace where you can upload files for indexing and storage in one of the vector databases. You can select the embeddings model and text-splitting configuration of your choice.

The solution comes with several debugging tools to help you debug RAG scenarios:

• Run RAG queries without chatbot and analyse results, scores, etc.
• Test different embeddings models directly in the UI
• Test cross encoders and analyse distances from different functions between sentences.

The repository includes a CDK construct to deploy a full-fledged UI built with React to interact with the deployed LLMs as chatbots. Hosted on Amazon S3 and distributed with Amazon CloudFront.

Protected with Amazon Cognito Authentication to help you interact and experiment with multiple LLMs, multiple RAG engines, conversational history support and document upload/progress.

The interface layer between the UI and backend is built with API Gateway REST API for management requests and Amazon API Gateway WebSocket APIs for chatbot messages and responses.

Design system provided by AWS Cloudscape Design System.

Before you begin using the solution, there are certain precautions you must take into account:

• Cost Management with self-hosted models on SageMaker: Be mindful of the costs associated with AWS resources, especially with SageMaker models billed by the hour. While the sample is designed to be cost-effective, leaving serverful resources running for extended periods or deploying numerous LLMs can quickly lead to increased costs.

• Licensing obligations: If you choose to use any datasets or models alongside the provided samples, ensure you check the LLM code and comply with all licensing obligations attached to them.

• This is a sample: the code provided in this repository shouldn't be used for production workloads without further reviews and adaptation.

We recommend deploying with AWS Cloud9. If you'd like to use Cloud9 to deploy the solution, you will need the following before proceeding:

• select at least m5.large as Instance type.
• use Ubuntu Server 22.04 LTS as the platform.

If you have decided not to use Cloud9, verify that your environment satisfies the following prerequisites:

You have:

1. An AWS account
2. AdministratorAccess policy granted to your AWS account (for production, we recommend restricting access as needed)
3. Both console and programmatic access
4. NodeJS 16 or 18 installed
   • If you are using nvm you can run the following before proceeding

   • nvm install 16 && nvm use 16
     
     or
     
     nvm install 18 && nvm use 18
     

5. AWS CLI installed and configured to use with your AWS account
6. Typescript 3.8+ installed
7. AWS CDK CLI installed
8. Docker installed
   • N.B. buildx is also required. For Windows and macOS buildx is included in Docker Desktop
9. Python 3+ installed

1. Clone the repository

git clone https://github.com/aws-samples/aws-genai-llm-chatbot

2. Move into the cloned repository

cd aws-genai-llm-chatbot

If you use Cloud9, increase the instance's EBS volume to at least 100GB. To do this, run the following command from the Cloud9 terminal:

./scripts/cloud9-resize.sh

See the documentation for more details on environment resize.

3. Install the project dependencies and build the project by running this command

npm install && npm run build

4. Once done, run the magic-create CLI to help you set up the solution with the features you care most:

npm run create

You'll be prompted to configure the different aspects of the solution, such as:

• The LLMs to enable (we support all models provided by Bedrock, FalconLite, LLama 2 and more to come)
• Setup of the RAG system: engine selection (i.e. Aurora w/ pgvector, OpenSearch, Kendra..) embeddings selection and more to come.

When done, answer Y to create a new configuration.

Your configuration is now stored under bin/config.json. You can re-run the magic-create command as needed to update your config.json

5. (Optional) Bootstrap AWS CDK on the target account and region

Note: This is required if you have never used AWS CDK before on this account and region combination. (More information on CDK bootstrapping).

npx cdk bootstrap aws://{targetAccountId}/{targetRegion}

You can now deploy by running:

npx cdk deploy

Note: This step duration can vary greatly, depending on the Constructs you are deploying.

You can view the progress of your CDK deployment in the CloudFormation console in the selected region.

6. Once deployed, take note of the User Interface, User Pool and, if you want to interact with 3P models providers, the Secret that will, eventually, hold the various API_KEYS should you want to experiment with 3P providers.

...
Outputs:
GenAIChatBotStack.UserInterfaceUserInterfaceDomanNameXXXXXXXX = dxxxxxxxxxxxxx.cloudfront.net
GenAIChatBotStack.AuthenticationUserPoolLinkXXXXX = https://xxxxx.console.aws.amazon.com/cognito/v2/idp/user-pools/xxxxx_XXXXX/users?region=xxxxx
GenAIChatBotStack.ApiKeysSecretNameXXXX = ApiKeysSecretName-xxxxxx
...

7. Open the generated Cognito User Pool Link from outputs above i.e. https://xxxxx.console.aws.amazon.com/cognito/v2/idp/user-pools/xxxxx_XXXXX/users?region=xxxxx

8. Add a user that will be used to log into the web interface.

9. Open the User Interface Url for the outputs above, i.e. dxxxxxxxxxxxxx.cloudfront.net

10. Login with the user created in .8; you will be asked to change the password.

You can remove the stacks and all the associated resources created in your AWS account by running the following command:

npx cdk destroy

• Bigad Soleiman
• Sergey Pugachev

This sample was made possible thanks to the following libraries:

• langchain from LangChain AI
• unstructured from Unstructured-IO
• pgvector from Andrew Kane

This library is licensed under the MIT-0 License. See the LICENSE file.

• Changelog of the project.
• License of the project.
• Code of Conduct of the project.
• CONTRIBUTING for more information.