Private AI Server

securesilo is a private AI server that you can run on your own hardware or in the cloud.

Organizations that want to use AI to generate content, but are concerned about the privacy and security of their data, can use securesilo to create a private AI server that is completely under their control.

Using 100% open source software, securesilo provides a secure, end-to-end encrypted service that your users can access via a web-based bot, a command line client, mobile app, or via the securesilo API without having to worry about any of your prompt data or ai-generated content being shared with any third party.

securesilo uses end-to-end encryption to ensure that your data is secure at all times. All prompt data is encrypted on the client device before it is sent to the server, all responses are encrypted on the server before being sent back to the user's device, and all data stored on the server is encrypted at rest.

When you create a securesilo server instance, you generate a public/private key pair that is used to encrypt your data for storage on the file system or in the database. The private key is never stored on the server, so only you can access your data. The silo can write this encrypted data but cannot decrypt it once it is written. Users' private keys can decrypt their own data on their devices, and the administrator's private key can decrypt all data on the server but, again, the decryption happens on the administrator's client device, not on the server.

You can also create backup private keys that can be used to recover your data in the event that you lose access to your primary private key. Backup private keys are unique keys that can be stored in secure offsite locations or with trusted third parties. You can create as many backup private keys as you like to ensure that you can always recover your data. Any backup private key can be revoked at any time.

You control who can access all prompts and ai-generated content stored on the server, and you can revoke user access at any time. Since all data is encrypted both in transit and at rest, even if your server is compromised, all stored data remains encrypted and can only be recovered using the private key or a backup private key.

If you lose your private key and do not have backup private keys, your data will be lost forever! Due to the nature of public key encryption, there is no way to recover your data without one of the private keys that were used to encrypt it.

Therefore, if your data has significant financial value or business operational risk, it is strongly recommended that you create at least three backup private keys and store them in different secure offsite locations (such as a on a thumb drive in a safety deposit box at a bank) and/or with trusted third parties such as your attorney, accountant, audit firm, professional key escrow service, professional security firm or other trusted individuals. Private keys can also be printed out on paper or stored securely on a hardware security device such as a Yubikey or Trezor.

Each user that you create on your server instance is assigned a unique device-based passkey that is used to encrypt their data in transit. User passkeys are generated on the user's device and the private key portion is never sent to the server. The server only stores the public key portion of the user's passkey. In addition, user passkeys are rotated on a regular basis to ensure that even if a user's device is compromised, the risk of data loss is limited. You can also require users to use two-factor authentication to further secure their account.

securesilo uses only LLM's (large language models) that have been released as open source software, or with a license that allows for commercial use, that can be run entirely on your own hardware or a cloud server that you completely control, without having to rely on any third party service which could use your data for their own purposes.

Currently securesilo supports:

llama-2 is a powerful engine that can be used to generate a wide variety of content. While it is not quite as powerful as OpenAI's GPT-4, it already outperforms the GPT-3.5, Bing and Bard engines on most tasks. Notably, llama-2 does not (yet!) generate images or computer language code, but it generates text responses to natural language queries very well, and is quite suitable for generating text-based documents such as technical reports, articles, scientific research, contracts and other legal documents.

Additional language models are the roadmap such as

• Stable Diffusion, StableLM (and StabilityCode!) https://github.com/stability-AI/stableLM/
• other Open LLMs https://github.com/eugeneyan/open-llms

And we plan to add support for non-text LLMs soon such as

• text-to-image models such as DALL-E, CLIP, VQGAN, and others
• code generation models such as Codex and Copilot

securesilo is 100% free, open source software. It is licensed under the MIT license, so you can use it for any purpose you like.

The Llama 2 language model can be licensed for commercial use from Meta as long as you agree to their License Agreement https://ai.meta.com/resources/models-and-libraries/Llama-downloads/ and Acceptable Use Policy https://ai.meta.com/llama/use-policy/

Anyone can use securesilo for any purpose they like, including commercial use. Distribution of the securesilo software is subject to the open source terms of the MIT license.

The Llama 2 language model can be licensed for commercial use from Meta, at no cost as long as you have fewer than 700 million active users per month 😮 and do not use it to train other LLM's.

For research use, there is no limit on the number of users.

securesilo is designed to use pre-trained language models, See PLUGINS (and RELATED PROJECTS) below for links to the software, tools and and hardware needed to train your own language models.

securesilo supports plugins that can be used to extend its functionality. Plugins can be used to add new language models, user interfaces, structured document types and data formats, storage methods locations, encryption methods, and visualization tools that we could never dream up on our own.

You can create your own plugins, use plugins contributed by the community or purchase proprietary plugins from third parties. You can also share your plugins, or sell them to other securesilo users on the soon-to-be-released securesilo plugin marketplace.

There are two ways to get a securesilo server instance up and running:

For a turn-key solution, you can use securesilo's Done For You service to have a securesilo server instance created and professionally hosted for you on a secure cloud server. You can be up and running in minutes.

Enterprises can also use the DFY service to have a securesilo server instance created, professionally managed, and maintained on their own private cloud or on-premises infrastructure.

If you want to run your own server, you can use the instructions below to get started.

For dev servers:

• server with at least 32GB of RAM
• NVIDIA T4 GPU (16GB RAM) or preferably a 40GB A100 (or 2)
• 1TB of free disk space, preferably on a fast NVMe device

Examples of bare minimum cloud instances types/sizes that meet these requirements are:

• Amazon AWS EC2 g4dn.2xlarge instance (8 vCPU, 32GB RAM, 1x 125GB NVMe SSD, 1x NVIDIA T4 GPU)
• Google Cloud GCE n1-standard-8 instance (8 vCPU, 30GB RAM, 1x 100GB SSD, 1x NVIDIA T4 GPU)
• Microsoft Azure NCasT4_v3 instance (8 vCPU, 32GB RAM, 1x 100GB SSD, 1x NVIDIA T4 GPU)

For production deployments capable of real-time inference using the most capable llama2 70b models currently at maximum accuracy for multiple users, over 280GB of GPU RAM is required (so 8 x A100's @ 40GB) or more to support multiple concurrent users:

• AWS EC2 p4d.24xlarge instance (96 vCPU, 1,024GB RAM, 8x 1.9TB NVMe SSD, 8x NVIDIA A100 GPU) for 320GB HBM2 RAM)
• AWS EC2 p4de.24xlarge instance (96 vCPU, 1,024GB RAM, 8x 1.9TB NVMe SSD, 8x NVIDIA A100 GPU) for 640GB HBM2e RAM

securesilo has been tested on the following platforms:

• Rocky Linux 8.4
• CentOS 8.4
• Ubuntu 20.04 LTS
• MacOS 11.5.2

securesilo requires the following software to be installed on your server:

• Docker
• Docker Compose
• Git
• Python 3.8 or later
• pyTorch 1.9 or later
• CUDA 11.1 or later
• cuDNN 8.2 or later
• NVIDIA GPU driver 470.57.02 or later

See the Docker installation guide for instructions on how to install Docker on your server.

See the Docker Compose installation guide for instructions on how to install Docker Compose on your server.

See the Git installation guide for

See the Python installation guide for instructions on how to

See the pyTorch installation guide for instructions on how to

See the CUDA installation guide

See the cuDNN installation guide

See the NVIDIA GPU driver installation guide

git clone [email protected]:GigawattDigital/securesilo.git

Copy the env.example file in the project root directory to .env and edit the values to provide your Meta llama-2 license information.

(please use your own values for the license keys, not the invalid sample values shown below)

# meta llama2 license info
SILO_META_Key_Pair_Id=K15QTJFJEIFSLZ
SILO_META_Policy=eyJTdGF0ZW1lbnRiOlt7InVuaXF1ZV9eYXNoIjoiXHUwMDAxP1Q%7EVT8jWyIsIlJlc791cmNlIjoiaHR0cHM6XC9oL2Rvd25sb2FkLmxsNW1hbWV0YS5uZXRcLyoiLCJDb25khXRpb24iOnsiRGF0ZUxlc3NUaGFuIjp7IkFXUzpFcG9jaFRpbWUiOjE2ODk4OTg4NjF9fX1dfQ__
SILO_META_Signature=jKj8w2K5uNW4fuDyj9QMSvLsOMZMydJk5-sAlHwIK40EctKgFpP-bHvGIgEwPnfdc%7EMxh85b8olqPtK4bYk0Jiuv1Q3ZIeg-j0wrtQAvozi8w5OyB9G-lwjiwQx2A-kCjbgdkWP0DZSXM-gCapk4sskfzMkjRwiveqT2Yw7hP16Oy04RHbK6Jgglaxc4hdiSM0X9aMhZN1ExyD%7EX8arqanP8HkBVb9exSiavtWhsn%7EyzratAEA2soVZApb07H74GU2S%7EPh3xIHk8I%7EtsZsgeVihGa1w6jZB7ayugmigz9z%7E%7EPAdK3x6WXSeLj4cjgwciiFXKZIqH-uOkCXHDY-e71g__

# Optionally, change the hostname/ip and/or and port you'd like to use to access your development instance
SILO_HOST=localhost
SILO_PORT=6006

docker-compose up

Open a browser and navigate to http://localhost:6006/ (or whatever host/port you configured in your .env file)

Create your admin account and setup its userid and password.

To complete the admin account setup, you'll be prompted to generate a public/private key pair, and to download the private key file.

Be sure to save the private key file in a safe place. You will need it to create backup private keys, administer the server, invite users and decrypt your backups.

Only the public key will be stored on the server.

That's it! You're ready to start creating AI-generated content.

All of your prompt and ai-generated data will be end-to-end encrypted so that, even if the server is compromised, your data will be safe.

securesilo includes a web app named gigabot that you can use to produce ai-generated content that only you have access to.

gigabot is a clone of the popular OpenAPI bot, but connects only to your securesilo server instance, instead of relying on third-party services such as OpenAI, Google and Microsoft.

All communication between your browser and your securesilo server instance is encrypted, your prompts are encrypted before they are sent to the server, stored encrypted on the server, and all ai-generated responses are encrypted on the server before beng sent back to your browser.

securesilo also includes a command line client named silo that you can use to create and manage ai-generated files.

You can also access your securesilo server via your instance's API endpoint. The securesilo api is compatible with the OpenAI API, so you can use the same tools you use with OpenAI to access your securesilo inference engine and data.

Since securesilo is designed to use end-to-end encryption, deploying to a publicly accessible server is nearly identical to setting up development instance.

The only differences are that you will need to configure the .env file with the URL of your server, and will probably want to enforce the use of SSL.

See the securesilo user guide for more information on how to require users to use two-factor authentication, device-based passkeys, and control other user-related security settings. New users can also be required to register using an email address with a specified domain name and/or only allow users to be created via an invitation link.

# allow only SSH access to securesilo instance
SILO_SSL_MODE=1

# set the public URL of your server
SILO_URL=https://mysilo.mydomain.com

If your silo is running behind a reverse proxy that provides SSL termination (like a local nginx server, AWS ELB load balancer or remote reverse proxy service such as Cloudflare), you can set SILO_URL to an https URL e.g. https://mysilo.mydomain.com along with a SILO_SSL=0 setting. Silo will detect the https:// protocol of the URL and ask you to confirm that you want to use https URL's even though it is running in http mode. While this could be considered slightly less secure, as traffic between your silo host and the (possibly distant) reverse proxy will be unencrypted at the network level, the risk is not significant because all traffic between your users' devices and the securesilo instance is still end-to-end encrypted at the application level, in transit, and at rest on the server. So the risk of a person-in-the-middle attack is limited to that person (e.g. staff at your hosting provider, your users' internet providers, Cloudflare, or any of the untrusted networks in-between) only being able to see that communication is occurring between your silo host server and the reverse proxy. The content of that communication will still be encrypted.

See the securesilo deployment guide for other advanced deployment configuration options.

Other Awesome, similar and related open source projects:

• Langchain - Pipeline tooling
  https://github.com/langchain-ai/langchain

• Chroma - embed a corpus of PDF documents for search and retrieval
  https://github.com/chroma-core/gpt4-pdf-chatbot-langchain-chroma

• Llama 2 Local Vector Embeddings and Conversational Agents
  https://www.youtube.com/watch?v=6iHVJyX2e50
  https://www.youtube.com/watch?v=yfHHvmaMkcA

• Llama Index - Connect LLMs to external data sources
  https://github.com/jerryjliu/llama_index

• ChatGPT Retrieval Plugin
  https://github.com/chroma-core/chatgpt-retrieval-plugin

• Copilot for lawyers
  https://www.harvey.ai/

• GPT4All - federated inference using consumer grade CPUs
  https://github.com/nomic-ai/gpt4all

• Falcon MPT - Powerful Open-Source Language Model
  https://www.mosaicml.com/blog/mpt-30b

• Vicuna - Open Source Chat Bot tuned for conversation
  https://github.com/eddieali/Vicuna-AI-LLM

• Stable Diffusion Image models
  https://github.com/Stability-AI/stablediffusion

• Awesome Transformers - A curated list of awesome transformers and pre-trained models
  https://github.com/huggingface/transformers/blob/main/awesome-transformers.md

• privateGPT - Train a local LLM based on GPT4All-J or LlamaCpp with your documents
  https://github.com/imartinez/privateGPT

• localGPT - Vicuna-7B with InstructorEmbeddings that can train on Apple Silicon
  https://github.com/PromtEngineer/localGPT

securesilo is licensed under the MIT License. You can use it for any purpose; you can modify it freely, and you can distribute your modifications without restrictions or cost.