• Make the system planning (provisioning, table placement) ideas more concrete
  • How to generalize the overall planning
  • What signals do we need
  • What initial experiments to run
• Make a plan for the initial implementation

Right now the daemon process receives queries as they are submitted to BRAD. We will need to record these queries for the offline planning pass.

Transactional logging should be done with sampling. But it should be fine to log every analytical query.

We should remove the codepath where we send queries to the daemon - this is a scaling bottleneck. Instead the frontend server should just log to its own files.

It would also be useful to write code to load the log format into a Workload class (depends on #97).

We currently use sqlglot for SQL parsing and manipulation. It's convenient and has a user-friendly API. The problem is that it is written in Python and could be a bottleneck on the critical path. It's worth investigating:

• Whether sqlglot ends up being a bottleneck
• If there are alternatives written in native code (with Python bindings) that support the features we need (parsing, AST manipulation, conversion back to SQL)
• Whether sqlglot supports enough of the Postgres SQL dialect that we care about

See #25.

So that we keep a consistent terminology.

Part of the data sync engine.

Call into one of the three systems

See the comments in #75 for context.

• Launch the planner and schedule it to run in the background
• Accept new blueprints and "deploy" them
  • For now, the deploy action will be a no-op (we will fill in the details later)

For convenience, we currently use a ThreadPoolExecutor to handle client requests. But because of the Python GIL, we will not be able to process multiple requests in parallel (even if the underlying server has multiple cores). We will likely need to revisit this design decision depending on the experiments we plan to run.

Some options

• A multiprocess architecture (e.g., use ProcessPoolExecutor) - we cannot easily share state among requests in this way
• Switch to a non-Python front end (e.g., C++) and embed a Python interpreter to run Python code
• Move computationally heavy code into C++ and release the GIL

• Two logical services:
  • The actual front end which handles queries from the user
  • Background daemon that runs policy engine, orchestration, forecasting
• Reasonable defaults for the development environment

See #25. We will want to change the schema slightly to capture data transformations across tables.

• Set up the schema and make sure we can load data effectively across the BRAD engines
• Transfer the workload executor over to BRAD

We'll use pytest to run tests. We should also hook it into the CI.

We already have a data blueprint. Probably the implementation that makes the most sense is to modify the DataBlueprint into a Blueprint. It makes sense to have access to the overall blueprint where we are currently using DataBlueprints.

We need to extend the brad admin tooling with a bulk load utility. We need this tool to be able to load datasets into the underlying engines before running any experiments.

Right now we are manually setting up our AWS deployment (via the web console). BRAD will need to automatically make provisioning changes. We need to set up abstractions and utilities within BRAD to initiate these provisioning changes.

• Explore AWS SDK integration (possibly boto3)
• Decorating the DB engines with the appropriate permissions for S3 export/import
• How to represent the provisioning details for system planning

We should restructure the BRAD architecture such that the daemon is the main component. It should be responsible for launching frontend servers and shutting them down. Right now it is the other way around because the system organically evolved with the frontend server being implemented first.

Why do this? Because the daemon forms BRAD's control plane. It handles system planning and mesh restructuring. The frontend server is just meant to handle user requests, and later we may have multiple of them.

See #17 for context. BRAD will have a scalability ceiling since it processes requests in a single-threaded event loop, but this is unfortunately a limitation of the Python runtime. If we need to scale beyond our current limits, we should run multiple instances of our front end (currently BradServer). This will require some light refactoring, but should be possible.

We currently manually implement the REPL interface (iohtap cli). But we should leverage Python's standard library implementation instead (cmd). We should then automatically get goodies like command history.

We use pyodbc, which runs all statements in a transaction. This ends up giving end-users a weird experience when using iohtap cli because they need to issue a COMMIT to commit writes (even if they never issued a BEGIN).

We can fix this by turning on autocommit when establishing connections to the DBMSes. But we need to double check that we were not relying on this functionality elsewhere inside the server.

Depends on #63.

See config/config_sample.yml for an example configuration file. As we add more configuration values, the file will get messier.

Currently, the only way to submit queries to IOHTAP is via the CLI. We need programmatic access for performance benchmarking purposes.

Representing table names with a custom type (for type checking) is more trouble than it is worth.

Beyond maintaining metrics for RDS as a whole, maintain a separate set for Aurora writer instances.

No need to go overboard in generality. But we need to have a way to run workloads against BRAD and the underlying engines individually.

• Issue queries via ODBC and the BRAD RPC interface
• Scale a workload up in size and number of clients
  • Workloads can be hardcoded - no need for excessive generality here
• Measure query latency (p50, p99)
• Measure workload throughput

Needed for scoring.

Use the existing physical operators in the data_sync module if possible.

Integrate the forecasted levels of system metrics into the overall system planning process

Needed for filtering and scoring

The sooner we do this the better.

• Extend the schema description configuration file to include data dependencies
• Implement a data placement plan abstraction (serializable)
• Add transformation specifications to the schema description config file
• Ensure routing is location-aware (not all tables are going to be available on all engines)
• Update Aurora table extraction to work with the transforms we need to specify
• Implement data sync plan logical abstractions
• Implement the physical data sync operators
• Implement the data sync planner
• Implement a function that lowers the logical plan into a physical plan
• Extend the data sync executor to run the data sync plan (execution operators, etc.)
• Handle skipped extractions (b/c there were no changes made to the table)
• Verify (or handle) cases of delete-insert-delete

• Blueprint neighborhood definitions
• Blueprint enumeration
• Filter framework
• Scoring function
  • Extract instance costs
  • #97
  • #98
  • #99

We need a "data mesh" workload (analytical queries and transactions). We should implement 2 - 3 workloads to showcase different parts of the system.

Possible workloads to adapt

• IMDB dataset and queries (@wuziniu and Dean are exploring this option) (#36)
  • Need to expand this workload to include some plausible transactions
• Adapt the HATtrick benchmark from Wisconsin (#37)
  • May need to alter the schemas to make the workload more transactionally friendly (it's based on SSB) and to showcase ETLs
  • May need to alter the analytical queries to make them more interesting
• TPC-DS
  • Already includes a data maintenance segment
  • Already includes analytical queries (should see if they expose interesting routing decisions)
  • Need to add plausible transactions

They are somewhat "left hanging" if the server shuts down first.

• One fixed baseline scenario (used to test the initial scale down)
• One scale up scenario where we increase the number of clients hitting the system (used to test re-planning for scale up)

See the comments in #16.

We currently believe that we will use metrics signals to know when to trigger re-provisioning. We also plan to use system metrics to detect poor provisioning decisions. We should see if these metrics are easily queryable from AWS, and explore how to best process them.

Needed to estimate transition costs. Also useful for forecasting changes in the dataset size.

We will eventually need connection pooling. We should look into options for any pooling solutions we can re-use, or write our own if needed.

We have them separate because data on S3 (for Athena) can also be read by Redshift (though we don't leverage this capability now). To simplify the code, we should just unify these two enums - having them separate adds unneeded code complexity at this stage in the project.

We will likely start with decision trees. This should be something simple and fast to start.

Beyond just forecasting that future values of the cloudwatch metrics will match the most recent value, we should explore using a moving average of their recent values, or fitting a linear model to predict the next value.

Later on, we can also add more complicated forecasting techniques that take seasonality into account, like Prophet by Meta.

Treat tables with data in it as "loaded". Only apply the bulk load to tables that are empty. This is useful because sometimes the bulk load gets "stuck" in the middle of processing Redshift data (I suspect some kind of synchronization bug in aioodbc).

We currently open a socket and implement a simple request-response protocol. This works for now, but we should avoid complicating the protocol if we need to add in more features. Instead, we should implement something more robust.

• Server/daemon interaction: Consider implementing RPCs (e.g., with gRPC)
• Client/server interaction: Ideally via the PostgreSQL wire protocol
  • gRPC can also be a stopgap method

• Install non-Python dependencies
• Install iohtap in development mode
• Provision a starting set of resources on AWS (Aurora, Redshift, S3 bucket, set up an IAM user for Athena)
• Generate a config.yml for development