Giter Club home page Giter Club logo

nucleuscloud / neosync Goto Github PK

View Code? Open in Web Editor NEW
644.0 5.0 38.0 58.26 MB

Open source data anonymization and synthetic data orchestration for developers. Create high fidelity synthetic data and sync it across your environments.

Home Page: https://www.neosync.dev

License: MIT License

Makefile 0.08% Go 38.53% TypeScript 56.10% CSS 0.18% Smarty 0.11% JavaScript 0.03% Shell 0.21% Starlark 0.18% MDX 4.54% Python 0.04%
docker golang protobuf synthetic-data benthos etl orchestration testing typescript open-source

neosync's Introduction

neosyncbanner

Open Source Synthetic Data Orchestration


Introduction

Neosync is a developer-first way to create anonymized or synthetic data and sync it across all environments.

Companies use Neosync to:

  1. Unblock local development - Give developers the ability to self-serve de-identified and synthetic data whenever they need it
  2. Fix broken staging environments - Catch bugs before they hit production when you hydrate your staging and QA environments with production-like data
  3. Keep environments in sync - Keep your environments in sync with the latest synthetic data so you never hear "it works for me locally" again
  4. Get frictionless security, privacy and compliance - Easily comply with laws like HIPAA, GDPR, and DPDP with de-identified and synthetic data
  5. Seed development databases - Easily seed development databases with synthetic data for unit testing, demos and more

Features

  • Generate synthetic data based on your schema
  • Anonymize existing production-data to protect data
  • Subset your production database for local and CI testing by filtering on an object, id or custom query
  • Complete async pipeline that automatically handles job retries, failures and playback using an event-sourcing model
  • Referential integrity for your data automatically - never worry about broken foreign keys again
  • Declarative, GitOps based configs as a step in your CI pipeline to hydrate your CI DB
  • Pre-built data transformers for all major data types
  • Custom data transformers
  • Pre-built integrations with Postgres, Mysql, S3

Getting started

Neosync is a fully dockerized setup which makes it easy to get up and running.

We provide a compose.yml file that contains production image references that allow you to get up and running with just a few commands without having to build anything on your system.

Neosync uses the newer docker compose command, so be sure to have that installed on your machine.

To start Neosync, clone the repo into a local directory, be sure to have docker insalled and running, and then run:

make compose-up

To stop, run:

make compose-down

Neosync will now be available on http://localhost:3000.

Neosync is also pre-seeded with connections and jobs to get you started! Simply run the generate and sync job to watch Neosync in action!

Kubernetes, Auth Mode and more

For more in-depth details on environment variables, Kubernetes deployments, and a production-ready guide, check out the Deploy Neosync section of our Docs.

Resources

Some resources to help you along the way:

  • Docs for comprehensive documentation and guides: Note these are still a work in progress.
  • Discord for discussion with the community and Neosync team
  • X for the latest updates

Contributing

We love contributions big and small. Here are just a few ways that you can contribute to Neosync.

Mission

Our mission is to help developers build better, more resilient applications while protecting sensitive data. To do that, we built Neosync to give teams three things:

  1. A world-class developer experience that fits into any workflow and follows modern developer best practices such as GitOps
  2. A platform that can anonymize sensitive data or automatically generate synthetic data from a schema and sync that across all environments
  3. An open source approach that allows you to keep your most sensitive data in your infrastructure

Licensing

We strongly believe in free and open source software and make this repo is available under the MIT expat license.

neosync's People

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

neosync's Issues

[NEOS-457] Bug with user not in account

This happened when alisha switched the database out.

the frontend was trying to update the session storage to include the latest account but it kept being overwritten to the old one.

This is doubled by the fact that SWR keps trying to reach out to the jobs api and was met with a 403. It was doing it endlessly though. Why was it not backing off? Need to look into this.

From SyncLinear.com | NEOS-457

[NEOS-507] Refactor user_identity_provider_associations to be more generic

We want to reduce our dependency on auth0 in favor or any OIDC compliant provider.

To do so, we need to refactor this table to be more generic.

Perhaps the new table should follow this schema: provider_name, provider_identifier

The auth0_provider_id column would be replaced by the following:

provider_name | provider_identifier
----------------------------------
        auth0 | google|[email protected]
        github| nickzelei
        google| [email protected]

From SyncLinear.com | NEOS-507

[NEOS-508] Reduce Auth0 Requirement on Frontend

This is already pretty minimal thanks to next-auth.

However, we load in specific Auth0 named env vars today.

We should make this more generic and just ask for one set.

Next-auth technically supports multiple providers, but usually an OIDC provider is used as the front for those.

From SyncLinear.com | NEOS-508

bare metal and fly.io

Is your feature request related to a problem?

docs refer to baremetal and would like this.

Describe the solution you'd like

I like to use Fly.io, which uses a single dockerFile.

https://fly.io/docs/app-guides/multiple-processes/

So what we do is as part of the Docker boiler, you stuff in your binaries, which in neosync 2 or 3 I think.
Fly has the concept of Apps, and it will boost each App ( which is a binary ).

The other way that we can use Overwind, to run the binaries outside docker or inside docker.

https://github.com/DarthSim/overmind

Describe alternatives you've considered

Additional context

https://fly.io/docs/hands-on/install-flyctl/ is golang.

Fly can scale to zero and also scale per region
https://fly.io/docs/apps/scale-count/#:~:text=Scale%20an%20app%27s%20regions,its%20changes%20across%20these%20regions.

Thank you for your feature request โ€“ we love each and every one!

[NEOS-498] When on a team account, clicking on Settings routes you back to your personal account

The link is correct, but somehow you are re-routed back to the default account.

Must have something to do with this useeffect:

  useEffect(() => {
    if (isSystemAppConfigDataLoading || isAccountLoading) {
      return;
    }
    if (systemAppConfigData?.isAuthEnabled && account?.name) {
      return router.push(`/${account?.name}/settings/members`);
    } else {
      return router.push('/personal/settings/temporal');
    }
  }, [
    account?.id,
    isAccountLoading,
    systemAppConfigData?.isAuthEnabled,
    isSystemAppConfigDataLoading,
  ]);

From SyncLinear.com | NEOS-498

[NEOS-497] Auth isn't working quite right on Staging

I havent been able to reproduce this on localhost, just staging for some reason.

account switcher goes away when you click on the home button (but none of the other pages?)

auth redirect doesnt always happen

Clicking around enough eventually logs you in.

need to figure out what the deal is here

Home page seems to be totally broken

From SyncLinear.com | NEOS-497

[NEOS-442] Update job status

job status currently doesn't support enough states and map correctly to temporal.

  1. Remove status field from Jobs db table.
  2. Map Neosync status to temporal status (pauseStatus, scheduleStatus)
  3. Update FE to support pause status and schedule status

From SyncLinear.com | NEOS-442

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.