samagra-development / warpsql Goto Github PK

Currently the CI on GHA is failing.

Allows for finding and building new combinations of Postgres through GHA.

Features could include

• #64
• #65
• #66
• #67
• Show existing images for the combinations already built and pushed to GitHub Packages using a sequence of dropdowns.
• Allow for selecting an extension and its version and then narrowing it down based on compatibility of the combination with other extensions. (Initially can be just a small subset of extensions we support)
• Use GHA APIs to build the image and upload it to the GitHub packages and share the link when done. Check limits on GitHub Packages.

Stack

Static website deployed on GitHub pages and APIs routed through a thin proxy for hiding GHA keys and managing statuses for builds. The wrapper over GHA APIs can be deployed to a free dyno on Heroku.

https://github.com/features/codespaces

Cloning into 'zombodb'...
Error: 
   0: `pgrx-0.9.3` shouldn't be used with `cargo-pgrx-0.8.3`, please use `pgrx = "~0.8.3"` in your `Cargo.toml`.

Location:
   /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/cargo-pgrx-0.8.3/src/metadata.rs:42

  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ SPANTRACE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

   0: cargo_pgrx::metadata::validate
      at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/cargo-pgrx-0.8.3/src/metadata.rs:27
   1: cargo_pgrx::command::install::execute
      at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/cargo-pgrx-0.8.3/src/command/install.rs:51

Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it.
Run with RUST_BACKTRACE=full to include source snippets.
The command '/bin/sh -c apk add --no-cache --virtual .zombodb-build-deps     git 	curl 	bash 	ruby-dev 	ruby-etc 	musl-dev 	make 	gcc 	coreutils 	util-linux-dev 	musl-dev 	openssl-dev     clang15 	tar     && gem install --no-document fpm     && curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs/ | bash -s -- -y     && PATH=$HOME/.cargo/bin:$PATH     && cargo install cargo-pgrx --version 0.8.3     && cargo pgrx init --${PG_VER}=$(which pg_config)     && git clone https://github.com/zombodb/zombodb.git     && cd ./zombodb     && cargo pgrx install --release     && cd ..     && rm -rf ./zombodb     && apk del .zombodb-build-deps' returned a non-zero code: 1
make: *** [Makefile:62: .build_main_pg15] Error 1

Error: Process completed with exit code 2.

This CI

Currently, smoke tests are part of the workflow which can lead to a issues in parsing when updating it.

Create a .sql file for smoke tests and add a husky pre-commit and test if it's a valid SQL file.

As a part of deployment config, user should be able to specify number of read replicas they want and requests should be automatically load balanced over read repicas.

User, at any point, should also be able to update number of read replicas and accordingy system should adjust.

Future Scopes:

Dynamic Scaling Configuration, where in user can specify minimum and maximum replicas and system should auto scale.

Reference: https://severalnines.com/blog/comparing-load-balancers-postgresql/

Currently, our Docker Compose configuration contains sensitive information such as the PostgreSQL username and password. This information is directly written in the docker-compose.yml file, which is not a secure practice and could lead to accidental exposure of these details.

To enhance the security of our setup, we should move these sensitive details to a separate .env.sample file. Docker Compose automatically reads from a .env.sample file in the same directory as the docker-compose.yml file, if it exists.

Tasks:

1. Create a .env.sample file in the same directory as the docker-compose.yml file.
2. Move the POSTGRES_USER and POSTGRES_PASSWORD environment variables from the docker-compose.yml file to the .env.sample file.
3. Update the docker-compose.yml file to reference these variables from the .env.sample file.
4. Update the documentation to instruct users to create their own .env.sample file with their specific details.

Acceptance Criteria:

1. The docker-compose.yml file no longer contains the POSTGRES_USER and POSTGRES_PASSWORD environment variables directly.
2. The .env.sample file is created and contains the POSTGRES_USER and POSTGRES_PASSWORD environment variables.

Hypopg allows for enabling Dexter.

Currently, the GHA builds a single image with all extensions. It should be parametric to allow for the selection of any combination of extensions.

Description

There is a need for a commenting system that will allow users to interact with the blog posts. This will increase user engagement and provide feedback to the blog authors.

Goals

• Add a comment section to each blog post
• Implement a moderation system for comments
• Ensure comments are tied to a registered user
• Implement spam and bot protection for the comments
• Add notification system for new comments

Expected Outcome

• Users must be registered and logged in to leave a comment.
• Each comment should include the user's name, profile picture (if available), comment text, and timestamp.
• Comments should be displayed in chronological order, with the most recent comment at the top.
• An email notification should be sent to the blog post author when a new comment is made.
• Blog post authors and administrators should be able to moderate comments (approve, deny, delete).

Acceptance Criteria

• A registered user can submit a comment on a blog post.
• The comment appears on the blog post after approval from the post author or administrator.
• An email notification is sent to the blog post author when a new comment is posted.
• Comments can be moderated by the blog post author and administrators.
• Commenting system is resistant to spam and bot attacks.

Implementation Details

• Leverage Django's built-in commenting framework (if applicable)
• Use JavaScript and AJAX for real-time comment posting and updates
• Consider integrating with a service like Akismet for spam protection
• Use Django's built-in email function for the notification system

Mockups / Wireframes

(Here, you can link to any visual aids, mockups, wireframes, or diagrams that help illustrate what the final product should look like. This is not always necessary, but can be very helpful in many cases.)

Project

OpenBlog Platform

Organization Name:

The name of the organization proposing the project.

Mentor(s)

@ChakshuGautam @Shruti3004 @sukhpreetssekhon

Tech Skills Needed

Java

Complexity

High

Category

CI/CD

Sub Category

API

Domain

Education

Currently, WarpSQL's versioning lacks a proper structure. It would be nice to formalize this process to alert users of breaking changes etc.

Description:

Objective:
The goal of this issue is to design and implement a user interface for the WarpSQL web application that allows users to build customized Docker images for WarpSQL.

Details:
The user should have the option to select the extensions they want to include in their Docker image. After the user has made their selections, the web application should generate the Docker image accordingly. Once the image is generated, the user should be given the option to either pull the image or download it directly.

Requirements:

1. The UI should be intuitive and easy to navigate.
2. The UI should provide a clear list of available extensions for the user to select from.
3. The UI should provide real-time feedback on the status of the Docker image generation process.
4. The UI should provide clear instructions on how to pull or download the generated Docker image.

Additional Context:
This feature will greatly enhance the user experience of the WarpSQL web application by providing a simple and intuitive way for users to create customized Docker images.

Suggested Collaboration Tools:

1. Figma

Given this will increase in size over time, it would be important for us to be conscious about how much we are adding to the base image.

Project Details

Opinionated extensions to Postgres packaged as a single docker deployment. It allows for a quick stable installation of all plugins to speed up installation and management of your Postgres instance.

Features to be implemented

• 1. #2
• 2. #22
• 3. #49
• 4. #52
• 5. Benchmarking
• 6. #63

Learning Path

Complexity

Medium

Skills Required

Dockerfile; Shell Scripting; Linux; Ansible;

Name of Mentors:

@singhalkarun

Project size

8 Weeks

Product Set Up

Gitpod

C4GT

This issue is nominated for Code for GovTech (C4GT) 2023 edition.
C4GT is India's first annual coding program to create a community that can build and contribute to global Digital Public Goods. If you want to use Open Source GovTech to create impact, then this is the opportunity for you! More about C4GT here: https://codeforgovtech.in/

Ideally the extension should be available here - /usr/local/share/postgresql/extension
PgVector's Dockerfile - https://github.com/pgvector/pgvector/blob/master/Dockerfile

Update the smoke tests -

1. Rewrite the value of WarpSQL
2. Make a contribution section that includes both Gitpod and Codespaces.
3. Steps for contribution
   1. CI should pass
   2. How to publish, etc.

WarpSQL wants to support the following extensions

• PgVector
• TimescaleDB
• #3
• #4
• #5
• #6
• #7
• #8
• #9
• #10
• #34
• #61

Every single issue should have the following

1. Precise steps on how to add the extension to the Dockerfile
2. How to enable the extension
3. Actual smoke tests described in English
4. CI Updated with smoke tests
5. Build pipeline updates needed

Description

Create Terraform scripts that leverage images created by Packer to facilitate the one-click deployment of WarpSQL on each of these cloud platforms.

• Contribution.md to include everything related to contribution. Gitpod, Codespaces, how to contribute. Link to GFIs.
• Readme.md - Should answer - What is WarpSQL, What problem does it solve, Maintainers. Link to Contribution.md.

Readme Content to be shown on the home page for WarpSQL.

• Evaluate postgresml and see if we can add this to WarpSQL.
• If yes for the above, add this.

In order to maintain consistent code quality and identify problems early on, we should incorporate a linters into our workflow. This will assist in streamlining the code review process as well.

There is a workflow failure due to a version conflict. We need to investigate and fix the issue. Details of the failure and its associated job can be found at this GitHub link.

+ apk add --no-cache --virtual .build-deps 'gdal-dev~=3.6.4-r4' 'geos-dev~=3.11' 'proj-dev~=9.2.0-r0' autoconf automake clang-dev clang15 cunit-dev file g++ gcc gettext-dev git json-c-dev libtool libxml2-dev llvm-dev llvm15 make pcre-dev perl protobuf-c-dev
fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/community/x86_64/APKINDEX.tar.gz
ERROR: unable to select packages:
  proj-dev-9.2.1-r0:
    breaks: .build-deps-20230603.170454[proj-dev~9.2.0-r0]
  .build-deps-20230603.170454:
    masked in: cache
    satisfies: world[.build-deps=20230603.170454] 

Description

New WarpSQL Logo

Goals

Currently, the logo only works with a dark theme and should work with both light and dark.

Project

WarpSQL

Organization Name:

Samagra | Transforming Governance

Mentor(s)

@ChakshuGautam

Tech Skills Needed

Figma

Complexity

Low

Category

UI/UX enhancements

Sub Category

Configuration

Domain

Education

Reference: https://github.com/cybertec-postgresql/pgwatch2

This plugin needs to be integrated to the project.

• Define DR Management strategy for this.
• Implementation.

References:

1. https://github.com/EnterpriseDB/barman
2. https://github.com/pgbackrest/pgbackrest

Add a table with X and Y axis being extensions and ticks and cross emojis to figure out what extension combinations would work for someone. Any subset of a row or column can be a valid combination.

Update in the PostgreSQL apt repository has disrupted our scripts and workflow again. Following @jayanth-kumar-morem's advice in this comment, we should transition to building PostGIS from source.

Tasks:

• Modify Bitnami Dockerfile to build PostGIS from source.

• Apply as a Vendor. For this ticket create your own account and document the process.
• Create a 1-Click App
• Add a button to launch the 1-Click App on the Readme

Currently, the CI takes up ~20ish minutes which is too slow for my liking.

Every single issue should have the following

• Precise steps on how to add the extension to the Dockerfile
• How to enable the extension
• Actual smoke tests described in English
• CI Updated with smoke tests

Reference: https://developer.hashicorp.com/packer/docs/templates/hcl_templates

Details:

1. A packer template that allows us to build WarpSQL image for multiple platforms like Docker, AWS EC2 AMI, Azure VMI, etc.
2. The template should be flexible enough. For ex. User should be able to configure extensions users want to enable may be in ENV or Config, and only those extensions should be setup.
3. Both of the above should be done for extensions which are currently enabled.

pg_cron

Description

Document Smoke Tests

Goals

Currently, there is no good way to manage smoke tests. They are all added to the same file. The project should define all smoke tests in separate files for every extension. Update GHA to include the merging of all of these files and then running the CI for smoke test.

Product

WarpSQL

Organization Name:

Samagra | Transforming Governance

Mentor(s)

@ChakshuGautam

Tech Skills Needed

GitHub Action Scripts, Yaml

Complexity

Low

Category

CI/CD

Sub Category

API

Domain

Education

@CurrentBehaviour
The CITUS_VERSION is hardcoded in Dockerfiles.

@ExpectedBehaviour
The CITUS_VERSION should be passed as an argument from Makefiles-s to Dockerfile-s.

Why the change is needed ?
In future, we would like to test various versions of extension being compatible with various psql versions through CI.

CC: @ChakshuGautam

• Move the disaster recovery and packer code to their own branches and also clean up the readme to focus on the extension support in WarpSQL.
• Clean up the tests
• Update the CI Workflow to save the images in the github packages

Add a file called related projects that would include

1. Projects - https://github.com/ankane/pghero#related-projects
2. Open source that can use WarpSQL - pghero, supabase, hasura, ... with caveats.
3. Reference to Awesome

• With Hasura
• With Langchain
• With Supabase
• With pgwatch2

The Tutorials will be published through a static website (docusaurus)