An extremely simple DataSource that supports sequential reads (i.e.: on just one executor) from ExampleDB. It only supports reads from a single, pre-defined table with a pre-defined schema. This DataSource is probably about a simple as one that reads from a remote database can get.

Another simple DataSource that supports sequential reads (i.e.: on just one executor) from ExampleDB. It gets a table name from its configuration and infers a schema from that table.

Another simple DataSource that supports parallel reads (i.e.: on multiple executors) from ExampleDB. It gets a table name from its configuration and infers a schema from that table. If a number of partitions is specified in properties, it is used. Otherwise, the table's default partition count (always 4 in ExampleDB) is used.

This also supports parallel reads (i.e.: on multiple executors) from the ExampleDB. The interesting feature of this example is that it supports informing the Spark SQL optimizer whether the table is partitioned in the right way to avoid shuffles in certain queries. One example is grouping queries, where shuffles can be avoided if the table is clustered in such a way that each group (cluster) is fully contained in a single partition. Since ExampleDB only supports clustered indexes on single columns, in practice a shuffle can be avoided if the table is clustered on one of the grouping (In ExampleDB clustered tables, splits always respect clustering.)

This data source adds the ability to write data, and does so in parallel. The various classes for reading are identical to those of ParallelRowDataSource. All four values of SaveMode are supported. Each task writes to its own temporary table, and on global commit all of these temporary tables are copied into the destination table in a single ExampleDB transaction.

Simplest example that uses direct ExampleDB calls to populate a table and then uses the SimpleRowDataSource to query it from Spark. Since that data source is sequential the resulting Dataset has just one partition. Since the data source reads from a single, hard coded table with a hard coded schema, the table name is not specified int he Spark code.

Instead uses the FlexibleRowDataSource to infer the schema of a specified table and query it, again sequentially, again resulting in a Dataset with a single partition.

Uses the ParallelRowDataSource to infer the schema of a specified table and query it, this time in parallel, resulting in Datasets with multiple partitions. The example shows both taking the default number of partitions and specifying a partition count.

Uses the PartitioningRowDataSource to avoid a shuffle in a grouping/aggregation query against a table that is clustered ont he grouping column. It achieves this by using the SupportsReportPartitioning mixin for the DataSourceReader interface.

This uses the same data source as the previous example but doesn't cluster the table, thus illustrating the shuffle that takes place. .

This illustrates updates using the simplest update-capable data source example, the ParallelRowReadWriteDataSource.

First a dataframe is created that is used to populate a table for the first time. At that point the newly created table's database schema is calculated from the dataframe schema. Notice that even though we create a dataframe with 6 partitions, later when we read from the table we always obtain dataframes with 4 partitions. This is because all tables in ExampleDB advertise 4 partitions by default, and we would have to override that default when reading to obtain different partitioning. However, the partitioning of the dataframe DOES impact update parallelism -- notice from the log output that six tasks write to six temporary tables -- and these would have run in parallel had we not specified only 4 executors as we do in all these examples.

We then put all four settings of SaveMode through their paces and see their impact.