gingersnap-project / db-syncer Goto Github PK

View Code? Open in Web Editor NEW

4.0 4.0 8.0 413 KB

License: Apache License 2.0

Java 48.00% HTML 51.60% Shell 0.25% TSQL 0.15%

db-syncer's People

Contributors

Stargazers

Watchers

Forkers

wburns jabolina ryanemerson fax4ever rigazilla karesti pruivo pminz

db-syncer's Issues

Unable to connect to Postgres DB

I followed the steps in the README and I get the following ConnectException on startup:

__  ____  __  _____   ___  __ ____  ______ 
 --/ __ \/ / / / _ | / _ \/ //_/ / / / __/ 
 -/ /_/ / /_/ / __ |/ , _/ ,< / /_/ /\ \   
--\___\_\____/_/ |_/_/|_/_/|_|\____/___/   
2022-11-02 11:19:58,783 INFO  [org.inf.HOTROD] (Quarkus Main Thread) ISPN004021: Infinispan version: Infinispan 'Flying Saucer' 14.0.1.Finalio.netty.incubator.channel.uring.IOUring
2022-11-02 11:19:58,892 INFO  [org.gin.cdc.ManagedEngine] (Quarkus Main Thread) Starting service
2022-11-02 11:19:59,196 INFO  [org.inf.HOTROD] (Quarkus Main Thread) ISPN004021: Infinispan version: Infinispan 'Flying Saucer' 14.0.1.Final
2022-11-02 11:19:59,231 INFO  [org.inf.HOTROD] (HotRod-client-async-pool-2-1) ISPN004006: Server sent new topology view (id=1827126186, age=0) containing 2 addresses: [172.17.0.4/<unresolved>:11222, 172.17.0.3/<unresolved>:11222]
2022-11-02 11:19:59,240 INFO  [org.inf.HOTROD] (HotRod-client-async-pool-2-1) ISPN004014: New server added(172.17.0.4/<unresolved>:11222), adding to the pool.
2022-11-02 11:19:59,243 INFO  [org.inf.HOTROD] (HotRod-client-async-pool-2-1) ISPN004014: New server added(172.17.0.3/<unresolved>:11222), adding to the pool.
2022-11-02 11:19:59,245 INFO  [org.inf.HOTROD] (HotRod-client-async-pool-2-1) ISPN004016: Server not in cluster anymore(127.0.0.1/<unresolved>:11222), removing from the pool.
2022-11-02 11:19:59,459 INFO  [org.inf.HOTROD] (HotRod-client-async-pool-2-4) ISPN004006: Server sent new topology view (id=161664359, age=0) containing 2 addresses: [172.17.0.4/<unresolved>:11222, 172.17.0.3/<unresolved>:11222]
2022-11-02 11:19:59,461 INFO  [org.inf.HOTROD] (HotRod-client-async-pool-2-4) ISPN004014: New server added(172.17.0.4/<unresolved>:11222), adding to the pool.
2022-11-02 11:19:59,461 INFO  [org.inf.HOTROD] (HotRod-client-async-pool-2-4) ISPN004014: New server added(172.17.0.3/<unresolved>:11222), adding to the pool.
2022-11-02 11:19:59,462 INFO  [org.inf.HOTROD] (HotRod-client-async-pool-2-4) ISPN004016: Server not in cluster anymore(127.0.0.1/<unresolved>:11222), removing from the pool.
2022-11-02 11:19:59,660 INFO  [org.inf.HOTROD] (HotRod-client-async-pool-2-6) ISPN004006: Server sent new topology view (id=161664359, age=0) containing 2 addresses: [172.17.0.4/<unresolved>:11222, 172.17.0.3/<unresolved>:11222]
2022-11-02 11:19:59,661 INFO  [org.inf.HOTROD] (HotRod-client-async-pool-2-6) ISPN004014: New server added(172.17.0.4/<unresolved>:11222), adding to the pool.
2022-11-02 11:19:59,661 INFO  [org.inf.HOTROD] (HotRod-client-async-pool-2-6) ISPN004014: New server added(172.17.0.3/<unresolved>:11222), adding to the pool.
2022-11-02 11:19:59,662 INFO  [org.inf.HOTROD] (HotRod-client-async-pool-2-6) ISPN004016: Server not in cluster anymore(127.0.0.1/<unresolved>:11222), removing from the pool.
2022-11-02 11:19:59,766 INFO  [org.apa.kaf.con.jso.JsonConverterConfig] (Quarkus Main Thread) JsonConverterConfig values: 
	converter.type = key
	decimal.format = BASE64
	schemas.cache.size = 1000
	schemas.enable = false


2022-11-02 11:19:59,768 INFO  [org.apa.kaf.con.jso.JsonConverterConfig] (Quarkus Main Thread) JsonConverterConfig values: 
	converter.type = value
	decimal.format = BASE64
	schemas.cache.size = 1000
	schemas.enable = false


2022-11-02 11:19:59,772 INFO  [io.deb.tra.Filter] (Quarkus Main Thread) Using language 'jsr223.groovy' to evaluate expression 'value.source.table == 'customer' && valueSchema.name ==~ /us-east\.debezium\.customer\..*/'
2022-11-02 11:20:00,302 INFO  [io.deb.emb.EmbeddedEngine$EmbeddedConfig] (Quarkus Main Thread) EmbeddedConfig values: 
	access.control.allow.methods = 
	access.control.allow.origin = 
	admin.listeners = null
	bootstrap.servers = [localhost:9092]
	client.dns.lookup = use_all_dns_ips
	config.providers = []
	connector.client.config.override.policy = All
	header.converter = class org.apache.kafka.connect.storage.SimpleHeaderConverter
	key.converter = class org.apache.kafka.connect.json.JsonConverter
	listeners = [http://:8083]
	metric.reporters = []
	metrics.num.samples = 2
	metrics.recording.level = INFO
	metrics.sample.window.ms = 30000
	offset.flush.interval.ms = 60000
	offset.flush.timeout.ms = 5000
	offset.storage.file.filename = 
	offset.storage.partitions = null
	offset.storage.replication.factor = null
	offset.storage.topic = 
	plugin.path = null
	response.http.headers.config = 
	rest.advertised.host.name = null
	rest.advertised.listener = null
	rest.advertised.port = null
	rest.extension.classes = []
	ssl.cipher.suites = null
	ssl.client.auth = none
	ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
	ssl.endpoint.identification.algorithm = https
	ssl.engine.factory.class = null
	ssl.key.password = null
	ssl.keymanager.algorithm = SunX509
	ssl.keystore.certificate.chain = null
	ssl.keystore.key = null
	ssl.keystore.location = null
	ssl.keystore.password = null
	ssl.keystore.type = JKS
	ssl.protocol = TLSv1.3
	ssl.provider = null
	ssl.secure.random.implementation = null
	ssl.trustmanager.algorithm = PKIX
	ssl.truststore.certificates = null
	ssl.truststore.location = null
	ssl.truststore.password = null
	ssl.truststore.type = JKS
	task.shutdown.graceful.timeout.ms = 5000
	topic.creation.enable = true
	topic.tracking.allow.reset = true
	topic.tracking.enable = true
	value.converter = class org.apache.kafka.connect.json.JsonConverter


2022-11-02 11:20:00,304 WARN  [org.apa.kaf.con.run.WorkerConfig] (Quarkus Main Thread) Variables cannot be used in the 'plugin.path' property, since the property is used by plugin scanning before the config providers that replace the variables are initialized. The raw value 'null' was used for plugin scanning, as opposed to the transformed value 'null', and this may cause unexpected results.
2022-11-02 11:20:00,306 INFO  [org.apa.kaf.con.jso.JsonConverterConfig] (Quarkus Main Thread) JsonConverterConfig values: 
	converter.type = key
	decimal.format = BASE64
	schemas.cache.size = 1000
	schemas.enable = true


2022-11-02 11:20:00,307 INFO  [org.apa.kaf.con.jso.JsonConverterConfig] (Quarkus Main Thread) JsonConverterConfig values: 
	converter.type = value
	decimal.format = BASE64
	schemas.cache.size = 1000
	schemas.enable = true


2022-11-02 11:20:00,310 INFO  [org.gin.cdc.rem.RemoteOffsetStore] (engine) Configuring offset store io.debezium.embedded.EmbeddedEngine$EmbeddedConfig@2d0d2104
2022-11-02 11:20:00,312 INFO  [org.gin.cdc.rem.RemoteOffsetStore] (engine) Starting remote offset store
2022-11-02 11:20:00,348 ERROR [io.deb.con.com.BaseSourceTask] (engine) The 'snapshot.mode' value 'when_needed' is invalid: Value must be one of always, never, initial_only, initial, custom
2022-11-02 11:20:00,356 INFO  [io.deb.con.com.BaseSourceTask] (engine) Stopping down connector
2022-11-02 11:20:00,359 INFO  [org.gin.cdc.rem.RemoteOffsetStore] (engine) Stopping remote offset store.
2022-11-02 11:20:00,359 ERROR [io.deb.emb.EmbeddedEngine] (engine) Unable to initialize and start connector's task class 'io.debezium.connector.postgresql.PostgresConnectorTask' with config: {connector.class=io.debezium.connector.postgresql.PostgresConnector, converter.schemas.enable=true, schema.history.internal.remote.uri=hotrod://admin:[email protected]:11222?sasl_mechanism=SCRAM-SHA-512, offset.storage.remote.uri=hotrod://admin:[email protected]:11222?sasl_mechanism=SCRAM-SHA-512, transforms=filter, schema.include.list=debezium, tombstones.on.delete=false, topic.prefix=us-east, offset.storage.file.filename=, errors.retry.delay.initial.ms=300, value.converter=org.apache.kafka.connect.json.JsonConverter, key.converter=org.apache.kafka.connect.json.JsonConverter, publication.autocreate.mode=filtered, database.user=gingersnap_user, database.dbname=debeziumdb, offset.storage.remote.topic=us-east, offset.storage=org.gingesnap.cdc.remote.RemoteOffsetStore, database.server.name=gingersnap-eager, offset.flush.timeout.ms=5000, errors.retry.delay.max.ms=10000, transforms.filter.type=io.debezium.transforms.Filter, transforms.filter.language=jsr223.groovy, plugin.name=pgoutput, database.port=5432, offset.flush.interval.ms=60000, schema.history.internal=org.gingesnap.cdc.remote.RemoteSchemaHistory, transforms.filter.condition=value.source.table == 'customer' && valueSchema.name ==~ /us-east\.debezium\.customer\..*/, errors.max.retries=-1, database.hostname=localhost, database.password=********, name=engine, schema.history.internal.remote.topic=us-east, table.include.list=debezium.customer, snapshot.mode=when_needed}: org.apache.kafka.connect.errors.ConnectException: Error configuring an instance of PostgresConnectorTask; check the logs for details
	at io.debezium.connector.common.BaseSourceTask.start(BaseSourceTask.java:125)
	at io.debezium.embedded.EmbeddedEngine.run(EmbeddedEngine.java:797)
	at io.debezium.embedded.ConvertingEngineBuilder$2.run(ConvertingEngineBuilder.java:195)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)


2022-11-02 11:20:00,373 INFO  [io.quarkus] (Quarkus Main Thread) debezium-poc 0.1-SNAPSHOT on JVM (powered by Quarkus 2.14.0.CR1) started in 11.050s. Listening on: http://localhost:8080
2022-11-02 11:20:00,374 INFO  [io.quarkus] (Quarkus Main Thread) Profile pgsql activated. Live Coding activated.
2022-11-02 11:20:00,375 INFO  [io.quarkus] (Quarkus Main Thread) Installed features: [cdi, infinispan-client, resteasy-reactive, smallrye-context-propagation, vertx]
2022-11-02 11:20:16,348 INFO  [org.gin.cdc.ManagedEngine] (Shutdown thread) Service shutting down
2022-11-02 11:20:16,349 INFO  [io.deb.emb.EmbeddedEngine] (Shutdown thread) Stopping the embedded engine

Cache manager shutdown before flushing offset

Currently, we stop the cache manager in a few ways, including by listening to the ShutdownEvent. When the application is shutting down and the Debezium engine stops, it will try to flush the current offset. It is possible to stop the RemoteCacheManager before flushing the offset to the remote cache. This will raise an exception during shutdown:

2022-12-28 16:49:36,320 INFO  [io.gin.cdc.rem.RemoteOffsetStore] (engine) Setting {java.nio.HeapByteBuffer[pos=0 lim=29 cap=29]=java.nio.HeapByteBuffer[pos=0 lim=54 cap=54]}
2022-12-28 16:49:36,321 INFO  [org.inf.HOTROD] (engine) ISPN004002: Cannot perform operations on a cache associated with an unstarted RemoteCacheManager. Use RemoteCacheManager.start before using the remote cache.
2022-12-28 16:49:36,323 INFO  [io.gin.cdc.rem.RemoteOffsetStore] (engine) Stopping remote offset store.
2022-12-28 16:49:36,325 INFO  [io.deb.emb.EmbeddedEngine] (main) Stopping the embedded engine
2022-12-28 16:49:36,325 INFO  [io.deb.emb.EmbeddedEngine] (main) Waiting for PT5M for connector to stop
2022-12-28 16:49:36,325 ERROR [io.gin.hea.ConnectorReadiness] (engine) Engine rule1 failed [Error Occurred After Shutdown]: org.infinispan.client.hotrod.exceptions.RemoteCacheManagerNotStartedException:: Cannot perform operations on a cache associated with an unstarted RemoteCacheManager. Use RemoteCacheManager.start before using the remote cache.
	at org.infinispan.client.hotrod.impl.RemoteCacheImpl.assertRemoteCacheManagerIsStarted(RemoteCacheImpl.java:587)
	at org.infinispan.client.hotrod.impl.RemoteCacheImpl.putAllAsync(RemoteCacheImpl.java:255)
	at org.infinispan.client.hotrod.impl.RemoteCacheSupport.putAllAsync(RemoteCacheSupport.java:53)
	at io.gingersnapproject.cdc.cache.hotrod.HotRodOffsetBackend.set(HotRodOffsetBackend.java:38)
	at io.gingersnapproject.cdc.remote.RemoteOffsetStore.set(RemoteOffsetStore.java:51)
	at org.apache.kafka.connect.storage.OffsetStorageWriter.doFlush(OffsetStorageWriter.java:175)
	at io.debezium.embedded.EmbeddedEngine.commitOffsets(EmbeddedEngine.java:1059)
	at io.debezium.embedded.EmbeddedEngine.run(EmbeddedEngine.java:910)
	at io.debezium.embedded.ConvertingEngineBuilder$2.run(ConvertingEngineBuilder.java:195)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)

We should handle this scenario and only stop the cache manager after the Debezium engine stopped and the offset was flushed. We can use the RemoteOffsetStore#stop and issue an event while the cache manager waits for an event from all created caches before stopping.

Revisit connector metrics

Split metrics between Gauge and TimeGauge
Improve description (use the description provided in Debezium docs)
Add base unit if needed
Avoid metrics names with count or total at the end to avoid confusion. (Prometheus appends those words for counter and timer metric types)

Oracle Database Support

Add support to Oracle Database

Add connector type to Micrometer tags

Show the connector type in Micrometer tags. It may be helpful for more complex queries.

Investigate removal of Jackson as a dependency

Debezium has a lot of dependencies on kafka classes. One of these is for the JsonConverter code which uses Jackson to do so. Jackson has been historically a pain in regards to CVEs causing additional respins. We should implement our own JsonConverter class that uses the Json class we have in ISPN. This should allow us to remove all deps on Jackson.

SQL Server exposes streaming metrics different

While working on adding support for SQL Server in the tests (#95) we had some issues with the metrics tests. The problem is that the streaming metrics for SQL Server are exposed in a different way, some of the properties are contained within another object, identified by the database name:

Some properties are in the streaming level while others are in the debezium level.

Right now, it is retrieving only the properties within the debezium level, we need to fix it to recover all streaming properties.

For comparison, here is how MySQL (and other DBs) is exposed:

Add metrics for Oracle Database Connector

Debezium exposes more information in the Oracle Connector. Expose those metrics with micrometer.

Exception thrown when connecting to empty DB

This only occurs if you forget to execute populate.sql before quarkus dev.

2022-11-02 12:45:15,414 INFO  [io.deb.con.mys.MySqlStreamingChangeEventSource] (debezium-mysqlconnector-us-east-change-event-source-coordinator) Waiting for keepalive thread to start
2022-11-02 12:45:15,414 INFO  [io.deb.uti.Threads] (blc-localhost:3306) Creating thread debezium-mysqlconnector-us-east-binlog-client
2022-11-02 12:45:15,415 ERROR [io.deb.con.mys.MySqlStreamingChangeEventSource] (blc-localhost:3306) Error during binlog processing. Last offset stored = null, binlog reader near position = binlog.000002/2772
2022-11-02 12:45:15,415 INFO  [com.git.shy.mys.bin.BinaryLogClient] (blc-keepalive-localhost:3306) threadExecutor is shut down, terminating keepalive thread
2022-11-02 12:45:15,416 ERROR [io.deb.pip.ErrorHandler] (blc-localhost:3306) Producer failure: io.debezium.DebeziumException: Client requested master to start replication from position > file size Error code: 1236; SQLSTATE: HY000.
	at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.wrap(MySqlStreamingChangeEventSource.java:1229)
	at io.debezium.connector.mysql.MySqlStreamingChangeEventSource$ReaderThreadLifecycleListener.onCommunicationFailure(MySqlStreamingChangeEventSource.java:1274)
	at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:1079)
	at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:631)
	at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:932)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: com.github.shyiko.mysql.binlog.network.ServerException: Client requested master to start replication from position > file size
	at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:1043)
	... 3 more

Modifying key view when it is not primary key will cause duplicate entry

A user may update a column that is in the key view. In this case we need to detect this change and remove the old entry in addition to adding the new cache entry.

For example modifying the name in the default table we have will cause a second user in the cache.

Micrometer naming convention

Micrometer naming convection suggests using . to separate words in the metrics name.

My initial implementation uses _ and it should be fixed.

Multiple cache-manager pods

Registering @wburns' idea to deal with multiple cache-manager pods.

We'll do a re-work mapping Debezium's engines to a single pod. Instead of creating one engine for each rule, we'll have each engine handling all rules. We have a limitation here of not handling more than one rule mapping to the same database table, but I expect us to limit that already.

Adding a new rule means the engines will be stopped, the new table added, and the engine restarted. On a stop initiated, all work is flushed. Debezium itself handles the different offsets. It is capable of adding/removing a table and continuing from where it left off (if the pod still has the offset information).

Adding or removing a new pod means starting or stopping a Debezium engine.

Incorporate Config Yaml extension

With the extension, we can run locally a the prod mode of the db syncer with the config in yaml
https://quarkus.io/guides/config-yaml

BatchConsumer throws NPE for MSSQL

2023-02-16 15:24:22,454 INFO  [io.deb.con.sql.SqlServerChangeEventSourceCoordinator] (debezium-sqlserverconnector-e2e-rule-10.244.0.92-change-event-source-coordinator) Starting streaming
2023-02-16 15:24:22,454 INFO  [io.deb.con.sql.SqlServerStreamingChangeEventSource] (debezium-sqlserverconnector-e2e-rule-10.244.0.92-change-event-source-coordinator) Last position recorded in offsets is 00000025:00000418:0001(NULL)[1]
2023-02-16 15:24:22,918 INFO  [io.gin.cdc.con.BatchConsumer] (engine) Processing 2 entries
2023-02-16 15:24:22,918 INFO  [io.gin.cdc.cha.CacheBackendLink] (engine) BEFORE -> null
2023-02-16 15:24:22,918 INFO  [io.gin.cdc.cha.CacheBackendLink] (engine) BEFORE -> null
2023-02-16 15:24:22,918 INFO  [io.gin.cdc.cha.CacheBackendLink] (engine) AFTER -> null
2023-02-16 15:24:22,918 INFO  [io.gin.cdc.cha.CacheBackendLink] (engine) AFTER -> {"id":1,"email":"[email protected]","fullname":"Alice"}
2023-02-16 15:24:22,919 INFO  [io.gin.cdc.con.BatchConsumer] (engine) Exception encountered writing updates for engine e2e-rule: java.util.concurrent.CompletionException: java.lang.NullPointerException: Cannot invoke "org.infinispan.commons.dataconversion.internal.Json.asString()" because the return value of "org.infinispan.commons.dataconversion.internal.Json.at(String)" is null
	at io.gingersnapproject.cdc.util.CompletionStages.join(CompletionStages.java:86)
	at io.gingersnapproject.cdc.consumer.BatchConsumer.handleBatch(BatchConsumer.java:54)
	at io.debezium.embedded.ConvertingEngineBuilder.lambda$notifying$2(ConvertingEngineBuilder.java:86)
	at io.debezium.embedded.EmbeddedEngine.run(EmbeddedEngine.java:881)
	at io.debezium.embedded.ConvertingEngineBuilder$2.run(ConvertingEngineBuilder.java:195)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.NullPointerException: Cannot invoke "org.infinispan.commons.dataconversion.internal.Json.asString()" because the return value of "org.infinispan.commons.dataconversion.internal.Json.at(String)" is null
	at io.gingersnapproject.cdc.chain.CacheBackendLink.processAsync(CacheBackendLink.java:39)
	at io.gingersnapproject.cdc.chain.CacheBackendLink.process(CacheBackendLink.java:28)
	at io.gingersnapproject.cdc.chain.EventProcessingChain.processNext(EventProcessingChain.java:23)
	at io.gingersnapproject.cdc.chain.EventFilterLink.process(EventFilterLink.java:37)
	at io.gingersnapproject.cdc.consumer.BatchConsumer.lambda$handleBatch$1(BatchConsumer.java:59)
	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768)
	... 3 more

Debezium Event JSON:

{
	"source": {
		"version": "2.0.0.Final",
		"connector": "sqlserver",
		"name": "e2e-rule-10.244.0.101",
		"ts_ms": 1676561650277,
		"snapshot": "true",
		"db": "gingersnap",
		"schema": "gingersnap",
		"table": "customer",
		"commit_lsn": "00000025:00000410:0003"
	},
	"ts_ms": 1676561650277,
	"databaseName": "gingersnap",
	"schemaName": "gingersnap",
	"tableChanges": [{
		"type": "CREATE",
		"id": "\"gingersnap\".\"gingersnap\".\"customer\"",
		"table": {
			"primaryKeyColumnNames": ["id"],
			"columns": [{
				"name": "id",
				"jdbcType": -5,
				"typeName": "bigint",
				"typeExpression": "bigint",
				"length": 19,
				"scale": 0,
				"position": 1,
				"optional": false,
				"autoIncremented": false,
				"generated": false
			}, {
				"name": "email",
				"jdbcType": 12,
				"typeName": "varchar",
				"typeExpression": "varchar",
				"length": 255,
				"position": 2,
				"optional": true,
				"autoIncremented": false,
				"generated": false
			}, {
				"name": "fullname",
				"jdbcType": 12,
				"typeName": "varchar",
				"typeExpression": "varchar",
				"length": 255,
				"position": 3,
				"optional": true,
				"autoIncremented": false,
				"generated": false
			}]
		}
	}]
}

When a rule is removed we need to clean up

When a rule is removed we need to make sure to delete the corresponding cache and the index offset. We may have to evaluate the schema cache, but I think that is going to be a lot tougher to clean up properly.

There may be other things I haven't thought of right now, but either way we should make sure this is investigated.