Giter Club home page Giter Club logo

mysqldumpsplitter's Introduction

mysqldumpsplitter - MySQL Dump splitter to split / extract databases, tables, list from mysqldump with plenty of more funtionality.

Usage:

Download the utility from website or git repository.


************ Usage ************

sh mysqldumpsplitter.sh --source filename --desc --extract [DB|TABLE|DBTABLES|ALLDBS|ALLTABLES|REGEXP] --match_str string --compression [gzip|pigz|bzip2|none] --decompression [gzip|pigz|bzip2|none] --output_dir [path to output dir] [--config /path/to/config]

Options:

--source: mysqldump filename to process. It could be a compressed or regular file.
--desc: This option will list out all databases and tables.
--extract: Specify what to extract. Possible values DB, TABLE, ALLDBS, ALLTABLES, REGEXP
--match_str: Specify match string for extract command option.
--compression: gzip/pigz/bzip2/none (default: gzip). Extracted file will be of this compression.
--decompression: gzip/pigz/bzip2/none (default: gzip). This will be used against input file.
--output_dir: path to output dir. (default: ./out/)
--config: path to config file. You may use --config option to specify the config file that includes following variables.
	SOURCE=
	EXTRACT=
	COMPRESSION=
	DECOMPRESSION=
	OUTPUT_DIR=
	MATCH_STR=

Ver. 5.0


mysqldumpsplitter recipe:

  1. Extract single database from mysqldump:

sh mysqldumpsplitter.sh --source filename --extract DB --match_str database-name

Above command will create sql for specified database from specified "filename" sql file and store it in compressed format to database-name.sql.gz.

  1. Extract single table from mysqldump:

sh mysqldumpsplitter.sh --source filename --extract TABLE --match_str table-name

Above command will create sql for specified table from specified "filename" mysqldump file and store it in compressed format to database-name.sql.gz.

  1. Extract tables matching regular expression from mysqldump:

sh mysqldumpsplitter.sh --source filename --extract REGEXP --match_str regular-expression

Above command will create sqls for tables matching specified regular expression from specified "filename" mysqldump file and store it in compressed format to individual table-name.sql.gz.

  1. Extract all databases from mysqldump:

sh mysqldumpsplitter.sh --source filename --extract ALLDBS

Above command will extract all databases from specified "filename" mysqldump file and store it in compressed format to individual database-name.sql.gz.

  1. Extract all table from mysqldump:

sh mysqldumpsplitter.sh --source filename --extract ALLTABLES

Above command will extract all tables from specified "filename" mysqldump file and store it in compressed format to individual table-name.sql.gz.

  1. Extract list of tables from mysqldump:

sh mysqldumpsplitter.sh --source filename --extract REGEXP --match_str '(table1|table2|table3)'

Above command will extract tables from the specified "filename" mysqldump file and store them in compressed format to individual table-name.sql.gz.

  1. Extract a database from compressed mysqldump:

sh mysqldumpsplitter.sh --source filename.sql.gz --extract DB --match_str 'dbname' --decompression gzip

Above command will decompress filename.sql.gz using gzip, extract database named "dbname" from "filename.sql.gz" & store it as out/dbname.sql.gz

  1. Extract a database from compressed mysqldump in an uncompressed format:

sh mysqldumpsplitter.sh --source filename.sql.gz --extract DB --match_str 'dbname' --decompression gzip --compression none

Above command will decompress filename.sql.gz using gzip and extract database named "dbname" from "filename.sql.gz" & store it as plain sql out/dbname.sql

  1. Extract alltables from mysqldump in different folder:

sh mysqldumpsplitter.sh --source filename --extract ALLTABLES --output_dir /path/to/extracts/

Above command will extract all tables from specified "filename" mysqldump file and extracts tables in compressed format to individual files, table-name.sql.gz stored under /path/to/extracts/. The script will create the folder /path/to/extracts/ if not exists.

  1. Extract one or more tables from one database in a full-dump:

Consider you have a full dump with multiple databases and you want to extract few tables from one database.

Extract single database:
`sh mysqldumpsplitter.sh --source filename --extract DB --match_str DBNAME --compression none`

Extract all tables
`sh mysqldumpsplitter.sh --source out/DBNAME.sql --extract REGEXP --match_str "(tbl1|tbl2)"`

though we can use another option to do this in single command as follows:

sh mysqldumpsplitter.sh --source filename --extract DBTABLE --match_str "DBNAME.(tbl1|tbl2)" --compression none

Above command will extract both tbl1 and tbl2 from DBNAME database in sql format under folder "out" in current directory.

You can extract single table as follows:

sh mysqldumpsplitter.sh --source filename --extract DBTABLE --match_str "DBNAME.(tbl1)" --compression none

  1. Extract all tables from specific database:

mysqldumpsplitter.sh --source filename --extract DBTABLE --match_str "DBNAME.*" --compression none

Above command will extract all tables from DBNAME database in sql format and store it under "out" directory.

  1. List content of the mysqldump file

mysqldumpsplitter.sh --source filename --desc

Above command will list databases and tables from the dump file.

mysqldumpsplitter's People

Contributors

definiteiymaybe avatar kedarvj avatar wngmnheiko avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

mysqldumpsplitter's Issues

Broken

I cloned the latest version (6.2) and attempted to use it and I get no errors and no output when it runs. Then I used version 6.1 and it works as expected. Used the following command:

sh mysqldumpsplitter.sh --source db.sql --extract DB --match_str zoom_dev3_menu

I think something is broken in the new version.

Feature request: Remove indexes from table

Large tables with indexes take a lot longer to import than without index. The net time to import without index, then recreate after data load, is about 1/3 of the. One scenario this script could solve is to finf the indexes of a table and just remove them.

What is DBTABLES?

There is an undocumented method called DBTABLES. I was hoping it would only extract tables from the specified DB ( one by one )

But it doesn't look like this does anything in the code?

grep detects binary data in dump and does not output any data

if there is any binary data detected in the mysqldump output, the grep command that searches for databases and tables will not output any data. On RedHat Linux system you can add the "-a" option to grep to treat the data as text and then the script will work normally.

Bzip2 Decompression Failing

root@backup:/root# ./mysqldumpsplitter.sh --decompression bzip2 --desc --source mysql-dump-20170119.bz2

Results IN the following error:

ERROR: Source file not specified or does not exist. (Entered: )

Usage: file [-bcEhikLlNnprsvzZ0] [--apple] [--extension] [--mime-encoding] [--mime-type]
            [-e testname] [-F separator] [-f namefile] [-m magicfiles] file ...
       file -C [-m magicfiles]
       file [--help]
/bin/bzip2
File  is a regular dump.
 Default decompression method for source is gzip.
 The input file  does not appear a compressed dump.
 We will try using no decompression. Please consider specifying --decompression none argument.
Setting output directory: out.
Processing: Extract DESCRIBE  from  with compression option as gzip and output location as out
-------------------------------
Database                Tables
-------------------------------

Using custom compression command (gzip --rsyncable)

I'm evaluating using this script together with mysqldump to do consistent backups that are splitted to several files for convenience. It looks incredibly well, I only found one missing feature.

I want to use the --rsyncable command-line option in gzip and pigz. This option is useful when the files are going to be transfered or backed up with any solution based on rsync. It can reduce the amount of data to transfer/store on consecutive dumps.

Now I think it's not possible to use this option without modifying the script.

multiple file processing? why?

as I see the script process source file many times, is it correct?
for ALLTABLES that would be number of tables + 1?

look, I have 74Gb gzipped mysqldump file (350+ Gb ungzipped size) with 615 tables in it ...

Add ALLDBS-ALLTABLES extract

ALLDBS extracts one file per database.
ALLTABLES extracts one file per table.

Can you add ALLDBS-ALLTABLES which extracts one file per database and table?
(We can run the script twice to do this but an option would be welcome to save time... ^^)

Add option: Random number or string to filenames when tables have same name

Some idiot before me dumped a massive DB (120gb dump file) with multiple tables that have the same name, into the same file. Dumping with this tool works, but overwrites the previous tables when extracting. I don't know sh at all, or even bash really, but I ended up changing it to bash and throwing $RANDOM until I got it to add a random number to the filename.

EDIT: Looks like it dumps the same table in the dump file multiple times. Not each file.

sed: RE error: illegal byte sequence

On OSX, seeing "sed: RE error: illegal byte sequence" when processing a dump. All of the tables in the dump have charset set to utf-8, and the cli has lang set to utf-8.

Terribly slow for big files

I'm trying to use this script on a 70GB compressed file and it's terribly slow. It's decompressing the whole source file again and again for every table and it does a sed over the whole file everytime.

Current version is broken

When running the current version with e.g. sh mysqldumpsplitter.sh --source /absolute/path/to/filename then only the help as well as some messages containing "not found" are rendered and nothing is done.

$ sh mysqldumpsplitter.sh --source /path/to/my/devdump.sql --extract ALLTABLES
: not foundlitter.sh: 2:
: not foundlitter.sh: 10:
: not foundlitter.sh: 34:
: not foundlitter.sh: 48:
: not foundlitter.sh: 50:
: not foundlitter.sh: 51:
: not foundlitter.sh: 52:
: not foundlitter.sh: 53:
: not foundlitter.sh: 54:
: not foundlitter.sh: 55:
: not foundlitter.sh: 56:
: not foundlitter.sh: 57:
: not foundlitter.sh: 58:
: not foundlitter.sh: 60:
: not foundlitter.sh: 63: {

                                                        ************ Usage ************

: not foundlitter.sh: 64:
sh mysqldumpsplitter.sh --source filename --extract [DB|TABLE|DBTABLES|ALLDBS|ALLTABLES|REGEXP] --match_str string --compression [gzip|pigz|bzip2|xz|pxz|none] --decompression [gzip|pigz|bzip2|xz|pxz|none] --output_dir [path to output dir] [--config /path/to/config]

OPTIONS:

  --source: mysqldump filename to process. It could be a compressed or regular file.
  --desc: This option will list out all databases and tables.
  --extract: Specify what to extract. Possible values DB, TABLE, ALLDBS, ALLTABLES, REGEXP
  --match_str: Specify match string for extract command option.
  --compression: gzip/pigz/bzip2/xz/pxz/none (default: gzip). Extracted file will be of this compression.
  --decompression: gzip/pigz/bzip2/xz/pxz/none (default: gzip). This will be used against input file.
  --output_dir: path to output dir. (default: ./out/)
  --config: path to config file. You may use --config option to specify the config file that includes following variables.
                SOURCE=
                EXTRACT=
                COMPRESSION=
                DECOMPRESSION=
                OUTPUT_DIR=
                MATCH_STR=


Ver. 6.1

It works for me perfectly fine when manually checking out 4792da5.

Environment is Ubuntu 22.04 on WSL.

How to extract tables without headers

Having dump without any headers, just pure data, so dump looks like

INSERT INTO members10 VALUES (...
INSERT INTO members11 VALUES (...
INSERT INTO members12 VALUES (...
INSERT INTO members13 VALUES (...

as I see when I choose option "--match_str members10" it search for "-- Table structure for table members10" but it's not in dump. Dump is just pure data without database structure. Structure created from other dump. File is 100gb sized and I need to split it in files members10 members11 members12 etc.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.