DKVP: pair separator in value breaks parsing about miller HOT 8 CLOSED

The RFC-compliant CSV reader handles embedded separators within double quotes; readers for the other formats do not at all. This is a non-ideal situation, for sure.

This is a dup of #52 and all four of the current on-deck or active tasks are what I'm actively working on for v2.2.0.

from miller.

@johnkerl While proper quoting can solve this issue, I'm not sure it is the only solution and I'm not certain that it is preferable in this case. Having miller make a best-effort to parse unquoted-but-still-parsable data is a bug, IMO.

Given the log file that I am currently working with, for example: adding quotes to all the fields would make it a LOT harder to read (as a human reading plaintext, I mean).

from miller.

Good feedback. I'll dig harder into your request this evening.

from miller.

Thanks for looking into it further, @johnkerl -- I have not spent long looking into this, but I'm wondering if the problem isn't in /c/input/lrec_reader_mmap_dkvp.c within the lrec_parse_mmap_dkvp function: whenever it matches on ips, it changes the ips to a null and then sets the value to the byte after the ips. This seemingly would result in the behavior that is being seen (i.e. the key ends at the first ips, and the value begins at the last ips).

I wonder if this couldn't be fixed by tweaking that code to ensure that you only match on ips once per field, which should result in only the first ips being matched.

from miller.

My apologies for the hasty read; I've got double-quotes on the brain & assigned too much weight to the double-quotes in your data. :^/

This is (was) definitely a bug; fixed in c2e11c0.

Thank you for the report!! :)

from miller.

No problem at all, @johnkerl -- thanks very much for the quick fix!

from miller.

Nice fix, @johnkerl -- have confirmed that with it in-place, miller can now deal very nicely with things like a web-server access.log file in dkvp format (using = as the PS now works well, even though it is used in some field values as well).

from miller.

Awesome!!

from miller.