mpchadwick / dbanon Goto Github PK

Include a subcommand. E.g.

dbanon -config=myconfig.yml validate

Should check that none of the fields are anonymized with values that aren't known (e.g. typo on "fistname")

Potentially should also run the validation before running other commands.

Hello,

I encounter errors during the anonymized dump on a magento 1.9.4.5.

STEP 1 :

mysqldump --complete-insert -uroot -proot mydb eav_entity_type eav_attribute | ./dbanon -config=./magento1.yml map-eav > ./magento1-mapped.yml

STEP2 :

mysqldump --complete-insert -uroot -proot mydb | ./dbanon -log-level=trace -log-file=./dbanonlog.log -config=./magento1-mapped.yml | gzip > mydb.sql.gz

ERROR:

goroutine 1 [running]:
github.com/mpchadwick/dbanon/src.LineProcessor.processInsert(0x16eef36, 0x9, 0xc0002f6ea0, 0x17bb220, 0xc000124188, 0xc000334580, 0xc00067c000, 0xe762d, 0xe762d, 0xe762d)
/Users/maxchadwick/go/src/github.com/mpchadwick/dbanon/src/processor.go:32 +0x1086
github.com/mpchadwick/dbanon/src.LineProcessor.ProcessLine(0x16eef36, 0x9, 0xc0002f6ea0, 0x17bb220, 0xc000124188, 0xc000334580, 0xc00067c000, 0xe762d, 0xc0005565c0, 0x35)
/Users/maxchadwick/go/src/github.com/mpchadwick/dbanon/src/processor.go:22 +0x9b
main.main()
/Users/maxchadwick/go/src/github.com/mpchadwick/dbanon/main.go:112 +0x706
mysqldump: Got errno 32 on write

I tried on a magento 2.3.6 and I get the same error

STEP 1 :

mysqldump -uroot -proot mydbm2 eav_entity_type eav_attribute | dbanon -config=magento2 map-eav > ./magento2-mapped.yml

STEP2 :

mysqldump --complete-insert -uroot -proot mydbm2 | ./dbanon -log-level=trace -log-file=./dbanonlog.log -config=./magento2-mapped.yml | gzip > mydb.sql.gz

ERROR:

goroutine 1 [running]:
github.com/mpchadwick/dbanon/src.LineProcessor.processInsert(0xaedb67, 0x9, 0xc0002b4e70, 0xbb93c0, 0xc00000e198, 0xc0002ee930, 0xc0010b4000, 0x15bc, 0x15bc, 0x15bc)
/Users/maxchadwick/go/src/github.com/mpchadwick/dbanon/src/processor.go:32 +0x1086
github.com/mpchadwick/dbanon/src.LineProcessor.ProcessLine(0xaedb67, 0x9, 0xc0002b4e70, 0xbb93c0, 0xc00000e198, 0xc0002ee930, 0xc0010b4000, 0x15bc, 0xc00007de40, 0x3d)
/Users/maxchadwick/go/src/github.com/mpchadwick/dbanon/src/processor.go:22 +0x9b
main.main()
/Users/maxchadwick/go/src/github.com/mpchadwick/dbanon/main.go:112 +0x706
mysqldump: Got errno 32 on write

For the M2 and the M1, I ran the command on MySQL 5.7 docker container

This error occurs when exporting the core_session table.
I dumped the "core_session" table and got the same error when exporting the "rating_option_vote" table with the "remote_ip_long" column.
I set the "remote_ip_long" column to null and the export now fails on the "sales_payment_transaction" table.

I think the anonymization script does not take into account serialized variables and special characters
Once I empty all the problematic tables, the script completes fine but no data is anonymized.

We would no longer need to maintain requests like this #6

Hello,
I am getting an error when I run dbanon. Do you know what this error could relate to? The dbanon.log file is empty.

root@af1ae37f315b:/app# mysqldump -h mariadb -B oastest | dbanon -config=application/configs/anonymize.yml -log-level=debug > anonymized.sql
panic: interface conversion: sqlparser.Statement is nil, not *sqlparser.CreateTable

goroutine 1 [running]:
github.com/mpchadwick/dbanon/src.findNextTable(0xc00037ae80, 0x39)
        /Users/maxchadwick/go/src/github.com/mpchadwick/dbanon/src/create_table.go:27 +0x417
github.com/mpchadwick/dbanon/src.LineProcessor.ProcessLine(0xaedb67, 0x9, 0xc000272ea0, 0xbb93c0, 0xc0000a8130, 0xc0000839d0, 0xc00037ae80, 0x39, 0xc00037ae40, 0x3b)
        /Users/maxchadwick/go/src/github.com/mpchadwick/dbanon/src/processor.go:25 +0xdd
main.main()
        /Users/maxchadwick/go/src/github.com/mpchadwick/dbanon/main.go:112 +0x706
mysqldump: Got errno 32 on write

I've tried running it on both a MySQL 5.6 and MariaDB 10.4 inside a docker container, same result for both.

https://twitter.com/SchumacherFM/status/1117775634950950913?s=20

Per #2 (comment) there are limitations such as inability to parse CREATE TABLE.

This looks like a potential alternative:

https://github.com/blastrain/vitess-sqlparser

See: xwb1989/sqlparser#60

Per #2 (comment) there are limitations such as inability to parse CREATE TABLE.

This looks like a potential alternative:

https://github.com/blastrain/vitess-sqlparser

See: xwb1989/sqlparser#60

First of all, thanks for this great project!

I'd like to ask whether you're planning to handle databases with prefixes at any time in the future and/or you accept PRs in this matter?

Cheers!

There are of course many databases for which this tool could be useful. As a a second supported backend will look to add Posgresql support

Currently dbanon is silent about things like this in stdout. We can't say anything about this in stdout to make sure we don't corrupt the dump file. However we really should leave some trace of the fact that it did this. We should generate a log file for each invocation where we record things like this. We should also support the ability to specify a logging level at run time so that we can additionally log debug level messages

Would like to gather some data on how much time this adds when running the mysqldump command. It would also be interesting, although maybe difficult, to set target in terms of maximum percentage it could add to time to run mysqldump and fail the build if it exceeds. Should also do some tests with various databases sizes (small, medium, large) to confirm it amount of time added is linear or otherwise. Would also be nice to update the README with some details on performance benchmarks.

Hello,
I am getting an error when I run dbanon on Magento 2 database. It is simmiar to #14

root@3355fca9767a:/home/dev# mysqldump --complete-insert dev | ./dbanon -config=magento2-mapped.yml > anonymized.sql
panic: interface conversion: sqlparser.Statement is nil, not *sqlparser.Insert

goroutine 1 [running]:
github.com/mpchadwick/dbanon/src.LineProcessor.processInsert(0xaedb67, 0x9, 0xc0002f8e70, 0xbb93c0, 0xc00013a180, 0xc000336920, 0xc0015b4000, 0x35f8b, 0x35f8b, 0x35f8b)
        /Users/maxchadwick/go/src/github.com/mpchadwick/dbanon/src/processor.go:32 +0x1086
github.com/mpchadwick/dbanon/src.LineProcessor.ProcessLine(0xaedb67, 0x9, 0xc0002f8e70, 0xbb93c0, 0xc00013a180, 0xc000336920, 0xc0015b4000, 0x35f8b, 0xc0003ea190, 0x42)
        /Users/maxchadwick/go/src/github.com/mpchadwick/dbanon/src/processor.go:22 +0x9b
main.main()
        /Users/maxchadwick/go/src/github.com/mpchadwick/dbanon/main.go:112 +0x706
mysqldump: Got errno 11 on write