colinnewell / pcap2mysql-log Goto Github PK

View Code? Open in Web Editor NEW

3.0 3.0 2.0 513 KB

Extracting mysql traffic from a pcap (tcpdump) file.

License: MIT License

Makefile 1.45% Go 94.37% Dockerfile 0.81% Shell 1.31% Perl 2.07%

mysql pcap pcap-analyzer golang

pcap2mysql-log's Introduction

pcap2mysql-log

This program takes a packet capture file and turns it into a json transcript of the MySQL communications.

pcap2mysql-log test/captures/dump00.pcap --server-ports 3306

It is currently being developed as a quick tool to aid development, and is only really being developed as needed. To develop it properly it really needs a lot of effort, and so far that isn't being expended on this.

Rather than try to be clever with threads and synchronisation this is making a first pass at the data and breaking it into the MySQL packets. Then it processes them in the order they appeared on the wire. That does mean it holds onto large chunks of data in memory. Very large capture files may be a struggle to process.

There is also a quick tool for turning the data from the tool into a quick summary.

pcap2mysql-log test/captures/big-data.pcap | pcap2mysql-summaries

This embeds a go template that translates the json output to something more textual. This is rather rough right now.

Building

This program requires libpcap to build and run. On Linux you typically install a development version of the library like this on Debian and Ubuntu variants:

sudo apt install libpcap-dev

On Windows download and install npcap from https://nmap.org/npcap/. The regular installer is sufficient, you shouldn't need the SDK.

On Mac's/BSD the library bindings required should be there out of the box (no further action required).

For tests jq is used. See https://stedolan.github.io/jq/download/ for info on installing jq.

Note that it's assumed you have Go installed, and also make (without make look at the commands in the Makefile, that is mostly being used for convenience rather than because things are particularly complex).

git clone https://github.com/colinnewell/pcap2mysql-log.git
cd pcap2mysql-log
make
sudo make install

Usage of pcap2mysql-log:

    --server-ports int32Slice   Server ports (default [])
    --version                   Display program version

Reading a pcap file:

sudo tcpdump port 3306 -w packets.dump
./pcap2mysql-log packets.dump

Note that the --server-ports option is useful for narrowing down the traffic the program process from the packet capture. If you've captured web traffic as well as MySQL you can speed it up and ensure it won't get confused by the other traffic.

Known issues

Memory usage can be quite high. The code is very much not optimised.
Lots of features haven't been implemented.
The output format is very clunky. It largely matches the actual data structures it encounters and can take a fair amount of effort to interpret.
The json output is indented, but not perfectly as the output is partly manual for efficiency.

General notes

Since creating pcap2har seemed so easy, mostly just wiring up existing libraries, I figured how hard could this be? A fair bit more it turns out.

For context pcap2har reads HTTP requests and responses and uses the existing libraries. With MySQL we need to do the same kind of thing, looking at both sides of the communication, but libraries tend to be sending one way, and receiving the other, rather than reading both ways.

Where is the program so far? The current goal is to read a file containing the packet data and emit a quick transcript that demonstrates we've grokked it's contents correctly. It's very crude so far, but seems to be heading in the right direction.

Wireshark is as ever a useful reference for looking at what's going on. Both by looking at packet captures, and by looking at it's code.

Thanks

A big thank you to Nadja for helping figure out the MySQL traffic and starting some of the foundational code when it was at a nascent stage. Her help got it going at a point where I was feeling blocked.

The gopacket library and the people involved in creating that. This program uses that library and also contains a minor variation on their tcpreader code.

Wireshark has provided a useful reference point for checking my results.

The Lottip program also gave me confidence that this sort of thing could be done.

References

Wireshark's packet-mysql.c
https://github.com/orderbynull/lottip it's code got me started.
https://mariadb.com/kb/en/result-set-packets/
https://dev.mysql.com/doc/internals/en/integer.html#packet-Protocol::FixedLengthInteger

pcap2mysql-log's People

Contributors

Stargazers

Watchers

Forkers

cxaqhq hai008

pcap2mysql-log's Issues

Not detecting login packet

Not sure how best to detect it. Ideally detect that a greeting came first, but not absolutely sure if we can?

Could check the last string on packet 1 (could be 2, except if it's ssl that doesn't seem realistic) for the auth plugin.

Ensure we deal with different character sets in a reasonable way

Ensure that latin1 etc. work as you'd hope (not absolutely sure how that is ;) ).

Improve/add non query packet support and interpretation

The greeting and other negotiations/maintenance of mysql connections is only roughly sketched out so far. It would be good to ensure that can all be correctly interpretted.

Add godoc

This may end up suggesting restructuring the code a little.

Store capabilities from Greetings for use in decoding

OK packet in particular can do different things based on what capabilities were set in the greeting. We should maintain that greetings state so we know what is possible.

We can also hopefully display the capabilities nicely in the greetings output.

https://dev.mysql.com/doc/internals/en/capability-flags.html#flag-CLIENT_SESSION_TRACK
https://dev.mysql.com/doc/internals/en/packet-OK_Packet.html

Crash decoding pcap file

Got this crash with a capture:

panic: runtime error: makeslice: len out of range

goroutine 35 [running]:
github.com/colinnewell/pcap2mysql-log/pkg/mysql/decoding.readLenEncBytes(0x62ab40, 0xc00058e150, 0x6306a0, 0xcc0eb0, 0x593020, 0xc00058011a, 0x0)
	/home/cnewell/git/pcap2mysql-log/pkg/mysql/decoding/packet.go:40 +0x75
github.com/colinnewell/pcap2mysql-log/pkg/mysql/decoding.(*RequestDecoder).decodeExecute(0xc0004c0040, 0xc000594194, 0x49, 0x353, 0xc0004cf9b8, 0x4746f4, 0xc0004ba150)
	/home/cnewell/git/pcap2mysql-log/pkg/mysql/decoding/request.go:239 +0x565
github.com/colinnewell/pcap2mysql-log/pkg/mysql/decoding.(*RequestDecoder).Write(0xc0004c0040, 0xc000594194, 0x49, 0x353, 0xc0004c3000, 0x4c9419, 0xc000594194)
	/home/cnewell/git/pcap2mysql-log/pkg/mysql/decoding/request.go:135 +0x305
github.com/colinnewell/pcap2mysql-log/pkg/mysql/packet.(*MySQLPacketWriter).Write(0xc0004cfa80, 0xc000594194, 0x49, 0x353, 0x49, 0x0, 0x0)
	/home/cnewell/git/pcap2mysql-log/pkg/mysql/packet/split.go:39 +0xe2
github.com/colinnewell/pcap2mysql-log/pkg/mysql/packet.Copy(0x62ad20, 0xc0004ba140, 0x62acc0, 0xc0004c0040, 0x64, 0x7f0d357d1108, 0x0)
	/home/cnewell/git/pcap2mysql-log/pkg/mysql/packet/copy.go:24 +0x1dc
github.com/colinnewell/pcap2mysql-log/pkg/mysql/decoding.(*MySQLConversationReaders).ReadRequestDecoder(0xc00000e2c0, 0x62ad20, 0xc0004ba140, 0xc0004b8270, 0x1, 0x4, 0x4, 0x20016ad, 0x0, 0x40016ad, ...)
	/home/cnewell/git/pcap2mysql-log/pkg/mysql/decoding/reader.go:163 +0x1da
github.com/colinnewell/pcap2mysql-log/pkg/mysql/decoding.(*MySQLConversationReaders).ReadStream(0xc00000e2c0, 0x62c0a0, 0xc0004b8240, 0x1, 0x4, 0x4, 0x20016ad, 0x0, 0x40016ad, 0x0, ...)
	/home/cnewell/git/pcap2mysql-log/pkg/mysql/decoding/reader.go:103 +0x468
github.com/colinnewell/pcap2mysql-log/pkg/tcp.(*StreamFactory).New.func1(0xc00000e2e0, 0xc0004b8240, 0x1, 0x4, 0x4, 0x20016ad, 0x0, 0x40016ad, 0x0, 0x4, ...)
	/home/cnewell/git/pcap2mysql-log/pkg/tcp/factory.go:25 +0xc9
created by github.com/colinnewell/pcap2mysql-log/pkg/tcp.(*StreamFactory).New
	/home/cnewell/git/pcap2mysql-log/pkg/tcp/factory.go:24 +0x22e

Fuzz processing

Once we have enough functionality we should fuzz the code to spot problems. Make sure it's easy to run. Note that I'm not sure if we can do a quick run via github actions?

Text output?

It might be good to have a more readable text format that isn't machine readable. Machine readable is core, it allows someone to build whatever they want on top, but that doesn't mean this tool can't optionally produce a more easy to process format too.

Record coverage to point out untested code

A fair bit of the code is untested and that ought to be rectified. To help with that coverage ought to be generated and made obvious.

This ideally should take both the go tests and the e2e tests into consideration. The latter can be done with a specially built binary, and then combined to produce an overall coverage report.

For bonus points prevent a reduction in coverage. It might be that coveralls (the service) is the answer, or some other mechanism.

Understand prepared statements and responses

The code so far may understand a little of this traffic, but it's quite likely that returns a binary response format for efficiency and we should aim to grok that.

Understand responses made over multiple mysql packets

A large set of results is likely to span multiple packets and we should aim to deal with that.

Note that there are probably 2 possible scenarios here:

there are so many fields they span multiple packets
lots of data that spans multiple packets.

Deal with EOF deprecated mode

When getting field info there is an EOF deprecated mode that presumably means no EOF packet and you just use the field count. We should add support for that.

Enable more linters

Some of them have been commented out because I didn't fancy dealing with the problems. That ought to be resolved.

Improve test suite

Initial development was very test light. Hopefully we start adding more as we add new features. Fill in any gaps in the test coverage that make sense.

Ensure we have some decent end to end tests.

Ensure github actions continue to work with libpcap in the build

The out of the box go environment doesn't have the library, so on the pcap2har-go project that means I haven't got linting/testing automatically so far. We need to ensure we can keep the automated testing/linting when we do start using that.

Note that we may have to consider whether we can continue to automate testing on all the other platforms when we do. In theory it ought to be possible to build on Windows and Mac, but I haven't figured that out thus far.

Update error generation and ensure it's consistent and usable

Turn on the go error checking linter probably and conform with that.

Also just check that how errors are dealt with, and how a user can deal with them makes sense.

Ideally we ought to be able to extract the maximum information despite having issues (note that I'm not sure I want to try to deal with errors per se, but make sure I don't lose stuff I've already processed).

Create a test 'harness' or environment for creating packet captures reliably

It would be useful to be able to make new packet captures for testing purposes quickly and easily. Ideally allowing for a range of them to be re-created to spot if new details are added to new versions of MySQL for example.

We could create a docker-compose environment that allows for the spinning up of MySQL, and tcpdump and then connecting some sorts of client to generate traffic.

One really easy way to run the tcpdump is within a container and also on the same interface as the mysql/client, although there are probably more ways to do that.

I wonder if clients from different languages would be worth while to achieve that.

This would require the following parts:

mysql container
tcpdump container
client container (e.g. a test program that emits various mysql queries. This could actual be a Go test library or something like that).
docker-compose environment for running it.
documentation on how to use it and expand it.

Tidy up README and documentation

Ensure the README explains how to build/use the software and most of the rambling is gone.

Add thank you's to people and projects that have helped. Figure out correct attribution for help provided.

Produce decent json output

We should come up with a reasonable json format for the traffic that allows us to read it easily as a human, but also process it efficiently with computers.

We need to include address information about the connections, and we should also providing timing information (we get timing info for each data burst).

We should also note that traffic can be client -> server, or server -> client (as in the greeting).

A connection can have multiple interactions and ideally we should make it as easy as possible to understand them.

Add back flags to process raw files

They could be something like --raw-client-requests and just dump the json for that. That ought to be reasonably usable. Does raise the question of whether anything would need to be joined up. That something that isn't happening right now (the joining up), but might become more important later.

Ingest pcap files

This is the mission of this program, and it feels like we have enough in place to start ingesting files.

Ideally allow us to continue to ingest raw files too for now as I would like to encourage us to be able to ingest multiple sources. One of the things I would like to do is ensure that the decoding/output is reasonably independant of the pcap reading so that we can wire in other mechanisms. eBPF springs to mind.

Ensure we handle non MySQL input gracefully

When we are presented with pcap files they could have traffic unrelated to MySQL connections. We ought to be able to ignore that.

I wonder if it's worth having a mode where we note the connections but note they didn't appear to be MySQL traffice?

Support cursors

I'm not sure I've tried checking that at all and I wouldn't be surprised if that's going to fail in interesting ways.

Add output of packets the tool struggled to decode

We ought to be able to emit more info about them, and what the problem we thought we encountered with them.

Fix intermittent crash

The tests have exposed an intermittent crash.

fatal error: concurrent map iteration and map write

goroutine 6 [running]:
runtime.throw(0x62b4bb, 0x26)
	/snap/go/7013/src/runtime/panic.go:1116 +0x72 fp=0xc00004bce0 sp=0xc00004bcb0 pc=0x43a132
runtime.mapiternext(0xc00004bdd0)
	/snap/go/7013/src/runtime/map.go:853 +0x554 fp=0xc00004bd60 sp=0xc00004bce0 pc=0x4142d4
github.com/colinnewell/pcap2mysql-log/internal/reader.(*MySQLConversationReaders).GetConversations(0xc00004e890, 0x10, 0x0, 0x4)
	/home/colin/git/pcap2mysql-log/internal/reader/reader.go:40 +0xd6 fp=0xc00004be40 sp=0xc00004bd60 pc=0x59ba36
github.com/colinnewell/pcap2mysql-log/internal/streamfactory_test.TestHTTPStreamRead(0xc000001b00)
	/home/colin/git/pcap2mysql-log/internal/streamfactory/factory_test.go:36 +0x445 fp=0xc00004bf80 sp=0xc00004be40 pc=0x59ffc5
testing.tRunner(0xc000001b00, 0x62fbf8)
	/snap/go/7013/src/testing/testing.go:1123 +0xef fp=0xc00004bfd0 sp=0xc00004bf80 pc=0x4d576f
runtime.goexit()
	/snap/go/7013/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc00004bfd8 sp=0xc00004bfd0 pc=0x46eea1
created by testing.(*T).Run
	/snap/go/7013/src/testing/testing.go:1168 +0x2b3

goroutine 1 [chan receive]:
testing.(*T).Run(0xc000001b00, 0x622aac, 0x12, 0x62fbf8, 0x4837c6)
	/snap/go/7013/src/testing/testing.go:1169 +0x2da
testing.runTests.func1(0xc000001980)
	/snap/go/7013/src/testing/testing.go:1439 +0x78
testing.tRunner(0xc000001980, 0xc00014dde0)
	/snap/go/7013/src/testing/testing.go:1123 +0xef
testing.runTests(0xc00000e240, 0x78de10, 0x1, 0x1, 0xc002332b0b4bc8c6, 0x8bb30a4974, 0x7a9be0, 0x410f10)
	/snap/go/7013/src/testing/testing.go:1437 +0x2fe
testing.(*M).Run(0xc000150000, 0x0)
	/snap/go/7013/src/testing/testing.go:1345 +0x1eb
main.main()
	_testmain.go:45 +0x138

goroutine 22 [runnable]:
github.com/colinnewell/pcap2mysql-log/internal/gopacket.(*TCPReaderStream).Read(0xc00008e180, 0xc00009e000, 0x800, 0x800, 0x20, 0xc000446000, 0x0)
	/home/colin/git/pcap2mysql-log/internal/gopacket/tcpreader.go:112 +0x91
github.com/colinnewell/pcap2mysql-log/internal/reader.(*TimeCaptureReader).Read(0xc00008e1b0, 0xc00009e000, 0x800, 0x800, 0xc00000f801, 0x0, 0xc0000b9b40)
	/home/colin/git/pcap2mysql-log/internal/reader/timecapture.go:15 +0x62
io.(*teeReader).Read(0xc000094120, 0xc00009e000, 0x800, 0x800, 0xc00004e930, 0x5c82a0, 0xc00004e760)
	/snap/go/7013/src/io/io.go:542 +0x55
github.com/colinnewell/pcap2mysql-log/internal/reader.(*SavePointReader).Read(0xc0000960f0, 0xc00009e000, 0x800, 0x800, 0x0, 0x65c0c0, 0xc00000f8e0)
	/home/colin/git/pcap2mysql-log/internal/reader/savepointreader.go:32 +0x52
github.com/colinnewell/pcap2mysql-log/internal/mysql/packet.Copy(0x65bd60, 0xc0000960f0, 0x65bd00, 0xd03738, 0x41b065, 0x7fb458, 0x203000)
	/home/colin/git/pcap2mysql-log/internal/mysql/packet/copy.go:37 +0x110
github.com/colinnewell/pcap2mysql-log/internal/reader.(*MySQLConversationReaders).ReadMySQLRequest(0xc00004e890, 0xc0000960f0, 0xc00008e1b0, 0x1, 0x4, 0x4, 0x320a8c0, 0x0, 0x220a8c0, 0x0, ...)
	/home/colin/git/pcap2mysql-log/internal/reader/reader.go:108 +0x65
github.com/colinnewell/pcap2mysql-log/internal/reader.(*MySQLConversationReaders).ReadStream(0xc00004e890, 0x65cfc0, 0xc00008e180, 0x1, 0x4, 0x4, 0x320a8c0, 0x0, 0x220a8c0, 0x0, ...)
	/home/colin/git/pcap2mysql-log/internal/reader/reader.go:70 +0x2fc
github.com/colinnewell/pcap2mysql-log/internal/streamfactory.(*MySQLStreamFactory).New.func1(0xc00000e260, 0xc00008e180, 0x1, 0x4, 0x4, 0x320a8c0, 0x0, 0x220a8c0, 0x0, 0x4, ...)
	/home/colin/git/pcap2mysql-log/internal/streamfactory/factory.go:26 +0xc9
created by github.com/colinnewell/pcap2mysql-log/internal/streamfactory.(*MySQLStreamFactory).New
	/home/colin/git/pcap2mysql-log/internal/streamfactory/factory.go:25 +0x22e

goroutine 23 [runnable]:
github.com/colinnewell/pcap2mysql-log/internal/gopacket.(*TCPReaderStream).Read(0xc00008e1e0, 0xc00009e800, 0x800, 0x800, 0x20, 0xc000446000, 0x0)
	/home/colin/git/pcap2mysql-log/internal/gopacket/tcpreader.go:112 +0x91
github.com/colinnewell/pcap2mysql-log/internal/reader.(*TimeCaptureReader).Read(0xc00008e210, 0xc00009e800, 0x800, 0x800, 0xc00000f901, 0x0, 0xc0000b7b40)
	/home/colin/git/pcap2mysql-log/internal/reader/timecapture.go:15 +0x62
io.(*teeReader).Read(0xc000094140, 0xc00009e800, 0x800, 0x800, 0xc00004e940, 0x5c82a0, 0xc00004e760)
	/snap/go/7013/src/io/io.go:542 +0x55
github.com/colinnewell/pcap2mysql-log/internal/reader.(*SavePointReader).Read(0xc000096140, 0xc00009e800, 0x800, 0x800, 0x0, 0x65c0c0, 0xc00000f940)
	/home/colin/git/pcap2mysql-log/internal/reader/savepointreader.go:32 +0x52
github.com/colinnewell/pcap2mysql-log/internal/mysql/packet.Copy(0x65bd60, 0xc000096140, 0x65bd00, 0xd03738, 0x41b065, 0x7fb458, 0x203000)
	/home/colin/git/pcap2mysql-log/internal/mysql/packet/copy.go:37 +0x110
github.com/colinnewell/pcap2mysql-log/internal/reader.(*MySQLConversationReaders).ReadMySQLRequest(0xc00004e890, 0xc000096140, 0xc00008e210, 0x1, 0x4, 0x4, 0x220a8c0, 0x0, 0x320a8c0, 0x0, ...)
	/home/colin/git/pcap2mysql-log/internal/reader/reader.go:108 +0x65
github.com/colinnewell/pcap2mysql-log/internal/reader.(*MySQLConversationReaders).ReadStream(0xc00004e890, 0x65cfc0, 0xc00008e1e0, 0x1, 0x4, 0x4, 0x220a8c0, 0x0, 0x320a8c0, 0x0, ...)
	/home/colin/git/pcap2mysql-log/internal/reader/reader.go:70 +0x2fc
github.com/colinnewell/pcap2mysql-log/internal/streamfactory.(*MySQLStreamFactory).New.func1(0xc00000e260, 0xc00008e1e0, 0x1, 0x4, 0x4, 0x220a8c0, 0x0, 0x320a8c0, 0x0, 0x4, ...)
	/home/colin/git/pcap2mysql-log/internal/streamfactory/factory.go:26 +0xc9
created by github.com/colinnewell/pcap2mysql-log/internal/streamfactory.(*MySQLStreamFactory).New
	/home/colin/git/pcap2mysql-log/internal/streamfactory/factory.go:25 +0x22e

goroutine 29 [runnable]:
github.com/colinnewell/pcap2mysql-log/internal/reader.(*MySQLConversationReaders).addRequest(0xc00004e890, 0x1, 0x4, 0x4, 0x220a8c0, 0x0, 0x320a8c0, 0x0, 0x4, 0x2, ...)
	/home/colin/git/pcap2mysql-log/internal/reader/reader.go:130 +0x485
github.com/colinnewell/pcap2mysql-log/internal/reader.(*MySQLConversationReaders).ReadMySQLRequest(0xc00004e890, 0xc000096320, 0xc00008e5d0, 0x1, 0x4, 0x4, 0x220a8c0, 0x0, 0x320a8c0, 0x0, ...)
	/home/colin/git/pcap2mysql-log/internal/reader/reader.go:111 +0x145
github.com/colinnewell/pcap2mysql-log/internal/reader.(*MySQLConversationReaders).ReadStream(0xc00004e890, 0x65cfc0, 0xc00008e5a0, 0x1, 0x4, 0x4, 0x220a8c0, 0x0, 0x320a8c0, 0x0, ...)
	/home/colin/git/pcap2mysql-log/internal/reader/reader.go:70 +0x2fc
github.com/colinnewell/pcap2mysql-log/internal/streamfactory.(*MySQLStreamFactory).New.func1(0xc00000e260, 0xc00008e5a0, 0x1, 0x4, 0x4, 0x220a8c0, 0x0, 0x320a8c0, 0x0, 0x4, ...)
	/home/colin/git/pcap2mysql-log/internal/streamfactory/factory.go:26 +0xc9
created by github.com/colinnewell/pcap2mysql-log/internal/streamfactory.(*MySQLStreamFactory).New
	/home/colin/git/pcap2mysql-log/internal/streamfactory/factory.go:25 +0x22e

goroutine 54 [semacquire]:
sync.runtime_SemacquireMutex(0xc00004e894, 0xc000062000, 0x1)
	/snap/go/7013/src/runtime/sema.go:71 +0x47
sync.(*Mutex).lockSlow(0xc00004e890)
	/snap/go/7013/src/sync/mutex.go:138 +0x105
sync.(*Mutex).Lock(...)
	/snap/go/7013/src/sync/mutex.go:81
github.com/colinnewell/pcap2mysql-log/internal/reader.(*MySQLConversationReaders).addRequest(0xc00004e890, 0x1, 0x4, 0x4, 0x320a8c0, 0x0, 0x220a8c0, 0x0, 0x4, 0x2, ...)
	/home/colin/git/pcap2mysql-log/internal/reader/reader.go:118 +0x55d
github.com/colinnewell/pcap2mysql-log/internal/reader.(*MySQLConversationReaders).ReadMySQLRequest(0xc00004e890, 0xc000456230, 0xc000454480, 0x1, 0x4, 0x4, 0x320a8c0, 0x0, 0x220a8c0, 0x0, ...)
	/home/colin/git/pcap2mysql-log/internal/reader/reader.go:111 +0x145
github.com/colinnewell/pcap2mysql-log/internal/reader.(*MySQLConversationReaders).ReadStream(0xc00004e890, 0x65cfc0, 0xc0004542d0, 0x1, 0x4, 0x4, 0x320a8c0, 0x0, 0x220a8c0, 0x0, ...)
	/home/colin/git/pcap2mysql-log/internal/reader/reader.go:70 +0x2fc
github.com/colinnewell/pcap2mysql-log/internal/streamfactory.(*MySQLStreamFactory).New.func1(0xc00000e260, 0xc0004542d0, 0x1, 0x4, 0x4, 0x320a8c0, 0x0, 0x220a8c0, 0x0, 0x4, ...)
	/home/colin/git/pcap2mysql-log/internal/streamfactory/factory.go:26 +0xc9
created by github.com/colinnewell/pcap2mysql-log/internal/streamfactory.(*MySQLStreamFactory).New
	/home/colin/git/pcap2mysql-log/internal/streamfactory/factory.go:25 +0x22e

goroutine 55 [semacquire]:
sync.runtime_SemacquireMutex(0xc00004e894, 0x7f8d02328f00, 0x1)
	/snap/go/7013/src/runtime/sema.go:71 +0x47
sync.(*Mutex).lockSlow(0xc00004e890)
	/snap/go/7013/src/sync/mutex.go:138 +0x105
sync.(*Mutex).Lock(...)
	/snap/go/7013/src/sync/mutex.go:81
github.com/colinnewell/pcap2mysql-log/internal/reader.(*MySQLConversationReaders).addRequest(0xc00004e890, 0x1, 0x4, 0x4, 0x220a8c0, 0x0, 0x320a8c0, 0x0, 0x4, 0x2, ...)
	/home/colin/git/pcap2mysql-log/internal/reader/reader.go:118 +0x55d
github.com/colinnewell/pcap2mysql-log/internal/reader.(*MySQLConversationReaders).ReadMySQLRequest(0xc00004e890, 0xc000506000, 0xc000504000, 0x1, 0x4, 0x4, 0x220a8c0, 0x0, 0x320a8c0, 0x0, ...)
	/home/colin/git/pcap2mysql-log/internal/reader/reader.go:111 +0x145
github.com/colinnewell/pcap2mysql-log/internal/reader.(*MySQLConversationReaders).ReadStream(0xc00004e890, 0x65cfc0, 0xc000454300, 0x1, 0x4, 0x4, 0x220a8c0, 0x0, 0x320a8c0, 0x0, ...)
	/home/colin/git/pcap2mysql-log/internal/reader/reader.go:70 +0x2fc
github.com/colinnewell/pcap2mysql-log/internal/streamfactory.(*MySQLStreamFactory).New.func1(0xc00000e260, 0xc000454300, 0x1, 0x4, 0x4, 0x220a8c0, 0x0, 0x320a8c0, 0x0, 0x4, ...)
	/home/colin/git/pcap2mysql-log/internal/streamfactory/factory.go:26 +0xc9
created by github.com/colinnewell/pcap2mysql-log/internal/streamfactory.(*MySQLStreamFactory).New
	/home/colin/git/pcap2mysql-log/internal/streamfactory/factory.go:25 +0x22e

goroutine 38 [runnable]:
github.com/colinnewell/pcap2mysql-log/internal/gopacket.(*TCPReaderStream).Read(0xc000012d50, 0xc0000d0800, 0x800, 0x800, 0x20, 0xc000446000, 0x0)
	/home/colin/git/pcap2mysql-log/internal/gopacket/tcpreader.go:112 +0x91
github.com/colinnewell/pcap2mysql-log/internal/reader.(*TimeCaptureReader).Read(0xc000012d80, 0xc0000d0800, 0x800, 0x800, 0xc00000ed01, 0x0, 0xc0000d7b40)
	/home/colin/git/pcap2mysql-log/internal/reader/timecapture.go:15 +0x62
io.(*teeReader).Read(0xc00000e5e0, 0xc0000d0800, 0x800, 0x800, 0xc00004e8c0, 0x5c82a0, 0xc00004e760)
	/snap/go/7013/src/io/io.go:542 +0x55
github.com/colinnewell/pcap2mysql-log/internal/reader.(*SavePointReader).Read(0xc00006c280, 0xc0000d0800, 0x800, 0x800, 0x0, 0x65c0c0, 0xc00000ed80)
	/home/colin/git/pcap2mysql-log/internal/reader/savepointreader.go:32 +0x52
github.com/colinnewell/pcap2mysql-log/internal/mysql/packet.Copy(0x65bd60, 0xc00006c280, 0x65bd00, 0xd03738, 0x66, 0x7fb458, 0x203000)
	/home/colin/git/pcap2mysql-log/internal/mysql/packet/copy.go:37 +0x110
github.com/colinnewell/pcap2mysql-log/internal/reader.(*MySQLConversationReaders).ReadMySQLRequest(0xc00004e890, 0xc00006c280, 0xc000012d80, 0x1, 0x4, 0x4, 0x220a8c0, 0x0, 0x320a8c0, 0x0, ...)
	/home/colin/git/pcap2mysql-log/internal/reader/reader.go:108 +0x65
github.com/colinnewell/pcap2mysql-log/internal/reader.(*MySQLConversationReaders).ReadStream(0xc00004e890, 0x65cfc0, 0xc000012d50, 0x1, 0x4, 0x4, 0x220a8c0, 0x0, 0x320a8c0, 0x0, ...)
	/home/colin/git/pcap2mysql-log/internal/reader/reader.go:70 +0x2fc
github.com/colinnewell/pcap2mysql-log/internal/streamfactory.(*MySQLStreamFactory).New.func1(0xc00000e260, 0xc000012d50, 0x1, 0x4, 0x4, 0x220a8c0, 0x0, 0x320a8c0, 0x0, 0x4, ...)
	/home/colin/git/pcap2mysql-log/internal/streamfactory/factory.go:26 +0xc9
created by github.com/colinnewell/pcap2mysql-log/internal/streamfactory.(*MySQLStreamFactory).New
	/home/colin/git/pcap2mysql-log/internal/streamfactory/factory.go:25 +0x22e
FAIL	github.com/colinnewell/pcap2mysql-log/internal/streamfactory	0.020s
FAIL

Move the decoding code to allow for use in a public library

The code has all been developed withing the internal package to make it clear that it's unstable and under development. This code ought to be generally useful though, so some thought to moving it out should be done.

This may involve further refactoring and renaming.

Documentation should also be checked to ensure it is comprehensible.