I just gave a talk about this at SCaLE 17x. Presentation lives here.
Vnlog (pronounced “vanillog”) is a trivially-simple log format:
- A whitespace-separated table of ASCII human-readable text
- Lines beginning with
#
are comments - The first line that begins with a single
#
(not##
or#!
) is a legend, naming each column - Empty fields reported as
-
Example:
#!/usr/bin/whatever # a b c 1 2 3 ## another comment 4 5 6
Such data works very nicely with normal UNIX tools (awk
, sort
, join
), can
be easily read by fancier tools (numpy
, matlab
, excel
, etc), and can be
plotted with feedgnuplot
. This tookit provides some tools to manipulate
vnlog
data and a few libraries to read/write it. The core philosophy is to
avoid creating new knowledge as much as possible. Consequently, the toolkit
relies heavily on existing (and familiar!) tools and workflows. As such,
the toolkit is small, light, and has a very friendly learning curve.
I have two sets of historical stock data, from the start of 2018 until now (2018/11):
$ < dji.vnl head -n 4 # Date Open High Low Close AdjClose Volume 2018-11-15 25061.48 25354.56 24787.79 25289.27 25289.27 383292840 2018-11-14 25388.08 25501.29 24935.82 25080.50 25080.50 384240000 2018-11-13 25321.21 25511.03 25193.78 25286.49 25286.49 339690000 $ < tsla.vnl head -n 4 # Date Open High Low Close AdjClose Volume 2018-11-15 342.33 348.58 339.04 348.44 348.44 4486339 2018-11-14 342.70 347.11 337.15 344.00 344.00 5036300 2018-11-13 333.16 344.70 332.20 338.73 338.73 5448600
I can pull out the closing prices:
$ < dji.vnl vnl-filter -p Close | head -n4 # Close 25289.27 25080.50 25286.49
And I can plot them:
$ < dji.vnl vnl-filter -p Close | feedgnuplot --lines --terminal 'dumb 80,30' --unset grid 27000 +-------------------------------------------------------------------+ | * + + + + | | * * | | *** * | 26500 |-+ * *** * +-| | * * * * | | * * ** * * | 26000 |-* * *** * * ** +-| | * * **** * * | | * ** ** * * | | * *** ** * * * * | 25500 |*+* *** *** * * * * +-| |* * **** ** ** ** * ** * * | |* *** * ** * ** ** * *** * * | |* ** * * ** * * * *** * * | 25000 |-+ ** *** * * * ** *** * * +-| | ** ** * * *** * ****** * * | | ** ** * ** ** * ** * ** | | ** * ** ** * * ** * ** | 24500 |-+ * * ** * * * ** * ** +-| | *** * **** *** * ** | | ** *** ***** ** | 24000 |-+ *** **** * +-| | * **** * | | *** | | + + + ** + | 23500 +-------------------------------------------------------------------+ 0 50 100 150 200 250
For this documentation I’m plotting in ascii (“dumb” terminal). Normally you’d
use the default terminal (by omitting --terminal ...
), and you’d get
interactive, graphical plots. Here I kept only the closing price column, so
the x-axis is just the row index. The data was in reverse chronological order,
so this plot is also in reverse chronological order. Let’s fix that:
$ < dji.vnl vnl-sort -k Date | vnl-filter -p Close | feedgnuplot --lines --terminal 'dumb 80,30' --unset grid 27000 +-------------------------------------------------------------------+ | + + + * + | | * * | | * * * | 26500 |-+ * ** * +-| | * ** * | | *** ** * * | 26000 |-+** * ***** * ** +-| | * * **** * ** | | * * ** ** ** | | * * * * ** ** ** | 25500 |-* * * * *** ** * * +-| | * * ** * ** ***** *** * * | |** * *** * ** *** ** * * * ** | |* * *** * *** *** * * * * * | 25000 |*+ * *** ** * * * *** *** +-| |* * ****** * * * * * * ** | | * * ** ** ** ** ** * * ** | | ** ** * ** * ** * * ** | 24500 |-+ ** * **** * ** * * * +-| | ** * ***** * * *** | | * * ** **** ** | 24000 |-+ * *** *** +-| | * *** * | | ** | | + ** + + + | 23500 +-------------------------------------------------------------------+ 0 50 100 150 200 250
This is in the correct order, but it’d be nice to see the actual dates on the x-axis:
$ < dji.vnl vnl-filter -p Date,Close | head -n4 # Date Close 2018-11-15 25289.27 2018-11-14 25080.50 2018-11-13 25286.49 $ < dji.vnl vnl-sort -k Date | vnl-filter -p Date,Close | feedgnuplot --lines --unset grid --terminal 'dumb 80,30' --timefmt %Y-%m-%d --domain 27000 +-------------------------------------------------------------------+ | + + + + + + + + * + | | * * | | * **** | 26500 |-+ ** ***** +-| | ** ** * | | *** ** * * | 26000 |-+** * *** ** * * +-| | * * **** * * | | ** * ** ** * | | * * * * * * *** ** | 25500 |-* * ** ***** *** ** +-| | * * ** * * ** ** **** *** | |* * * ** * ** ** * ** ** * * ** | |* * **** * *** ** * * * * * | 25000 |-+ * ******* * * * * * * * +-| | **** ** * * **** * * * ** | | *** ** ** * ****** ** ** ** | | *** * * ** * *** * * ** | 24500 |-+ *** * ***** * * ** * * +-| | ** * *** **** * *** | | * ** ** *** ** | 24000 |-+ * ***** *** +-| | * **** * | | * * | | + + * * + + + + + + + | 23500 +-------------------------------------------------------------------+ 01/01 02/01 03/01 04/01 05/01 06/01 07/01 08/01 09/01 10/01 11/01 12/01
What was the highest value of the Dow-Jones index, and when did it happen?
$ < dji.vnl vnl-sort -rgk Close | head -n2 | vnl-align # Date Open High Low Close AdjClose Volume 2018-10-03 26833.47 26951.81 26789.08 26828.39 26828.39 280130000
Alrighty. Looks like the high was in October. Let’s zoom in on that month:
$ < dji.vnl vnl-sort -k Date | vnl-filter 'Date ~ /2018-10/' -p Date,Close | feedgnuplot --lines --unset grid --terminal 'dumb 80,30' --timefmt %Y-%m-%d --domain 27000 +-------------------------------------------------------------------+ | ** + + + + | | ** * | | * * | 26500 |-+ * **** +-| | *** * | | * | | * | | * | 26000 |-+ * +-| | * ** | | * * * | | * * * | 25500 |-+ * * * ****** +-| | * ******* * * | | * * * ** | | ** * * | 25000 |-+ * * * *+-| | * ** * | | ** * * | | ** ***** * | | * * * | 24500 |-+ * +-| | | | | | + + + + | 24000 +-------------------------------------------------------------------+ 09/27 10/04 10/11 10/18 10/25 11/01
OK. Is this thing volatile? What was the largest single-day gain?
$ < dji.vnl | vnl-filter -p '.,d=diff(Close)' | head -n4 | vnl-align # Date Open High Low Close AdjClose Volume d 2018-11-15 25061.48 25354.56 24787.79 25289.27 25289.27 383292840 - 2018-11-14 25388.08 25501.29 24935.82 25080.50 25080.50 384240000 -208.77 2018-11-13 25321.21 25511.03 25193.78 25286.49 25286.49 339690000 205.99 $ < dji.vnl | vnl-filter -p '.,d=diff(Close)' | vnl-sort -rgk d | head -n2 | vnl-align # Date Open High Low Close AdjClose Volume d 2018-02-02 26061.79 26061.79 25490.66 25520.96 25520.96 522880000 1175.21
Whoa. So the best single-gain day was 2018-02-02: the dow gained 1175.21 points between closing on Feb 1 and Feb 2. But it actually lost ground that day! What if I looked at the difference between the opening and closing in a single day?
< dji.vnl | vnl-filter -p '.,d=Close-Open' | vnl-sort -rgk d | head -n2 | vnl-align # Date Open High Low Close AdjClose Volume d 2018-02-06 24085.17 24946.23 23778.74 24912.77 24912.77 823940000 827.6
I guess by that metric 2018-02-06 was better. Let’s join the Dow-jones index data and the TSLA data, and let’s look at them together:
$ vnl-join --vnl-autosuffix dji.vnl tsla.vnl -j Date | head -n4 | vnl-align # Date Open_dji High_dji Low_dji Close_dji AdjClose_dji Volume_dji Open_tsla High_tsla Low_tsla Close_tsla AdjClose_tsla Volume_tsla 2018-11-15 25061.48 25354.56 24787.79 25289.27 25289.27 383292840 342.33 348.58 339.04 348.44 348.44 4486339 2018-11-14 25388.08 25501.29 24935.82 25080.50 25080.50 384240000 342.70 347.11 337.15 344.00 344.00 5036300 2018-11-13 25321.21 25511.03 25193.78 25286.49 25286.49 339690000 333.16 344.70 332.20 338.73 338.73 5448600 $ vnl-join --vnl-autosuffix dji.vnl tsla.vnl -j Date | vnl-filter -p '^Close' | head -n4 | vnl-align # Close_dji Close_tsla 25289.27 348.44 25080.50 344.00 25286.49 338.73 $ vnl-join --vnl-autosuffix dji.vnl tsla.vnl -j Date | vnl-filter -p '^Close' | feedgnuplot --domain --points --unset grid --terminal 'dumb 80,30' 380 +---------------------------------------------------------------------+ | + + + + + + | | A A | | A | 360 |-+ A A A +-| | A A A A A AA | | A A AA AAAA A AA AA A | 340 |-+ A AA AAA A A AAA A A A A +-| | A A A A AAA AA A A A | | A A A A AAA A A A | | A A AA A A | 320 |-+ A A AAAAAA A AA A +-| | A A A A AAA AA | | A AA A A A A A A A AA A | | A AA AA A A A A A A | 300 |A+ AA A A A A AA A AAA A A AA A A +-| | A AA AA A A A A A A A | | A A A A AA A AA A | | AA A AAAA A A A AA A A | 280 |-+ A AAAAA A A +-| | A | | AA A | 260 |-+ AA AA A A +-| | A A A | | A A A | | + + + + + + | 240 +---------------------------------------------------------------------+ 23500 24000 24500 25000 25500 26000 26500 27000
Huh. Apparently there’s no obvious, strong correlation between TSLA and Dow-Jones closing prices. And we saw that with just a few shell commands, without dropping down into a dedicated analysis system.
Vnlog data is nicely readable by both humans and machines. Any time your
application invokes printf()
for either diagnostics or logging, consider
writing out vnlog-formatted data. You retain human readability, but gain the
power all the vnl-...
tools provide.
Vnlog tools are designed to be very simple and light. There exist a number of other tools that do vaguely the same thing. Some of these:
- https://www.gnu.org/software/datamash/
- https://csvkit.readthedocs.io/
- https://github.com/johnkerl/miller
- https://github.com/eBay/tsv-utils-dlang
- http://harelba.github.io/q/
- https://github.com/BatchLabs/charlatan
- https://github.com/dinedal/textql
- https://github.com/BurntSushi/xsv
- https://github.com/dbohdan/sqawk
- https://stedolan.github.io/jq/
- https://github.com/benbernard/RecordStream
Many of these provide facilities to run various analyses, and others focus on data types that aren’t just a table (json for instance). Vnlog by contrast doesn’t analyze anything, and targets the most trivial possible data format. This makes it very easy to run any analysis you like in any tool you like. The main envisioned use case is one-liners, and the tools are geared for that purpose. The above mentioned tools are much more powerful than vnlog, so they could be a better fit for some use cases. I claim that
- 90% of the time you want to do simple things, and vnlog is a great fit for the task
- If you really do need to do something complex, you really shouldn’t be in the shell writing oneliners anymore, and a fully-fledged analysis system (numpy, etc) is more appropriate
In the spirit of doing as little as possible, the provided tools are wrappers around tools you already have and are familiar with. The provided tools are:
vnl-filter
is a tool to select a subset of the rows/columns in a vnlog and/or to manipulate the contents. This is anawk
wrapper where the fields can be referenced by name instead of index. 20-second tutorial:
vnl-filter -p col1,col2,colx=col3+col4 'col5 > 10' --has col6
will read the input, and produce a vnlog with 3 columns: col1
and col2
from the input, and a column colx
that’s the sum of col3
and col4
in the
input. Only those rows for which both col5 > 10
is true and that have a
non-null value for col6
will be output. A null entry is signified by a
single -
character.
vnl-filter --eval '{s += x} END {print s}'
will evaluate the given awk program on the input, but the column names work as
you would hope they do: if the input has a column named x
, this would
produce the sum of all values in this column.
vnl-sort
,vnl-uniq
,vnl-join
,vnl-tail
,vnl-ts
are wrappers around the corresponding commandline tools. These work exactly as you would expect also: the columns can be referenced by name, and the legend comment is handled properly. These are wrappers, so all the commandline options those tools have “just work” (except options that don’t make sense in the context of vnlog). As an example,vnl-tail -f
will follow a log: data will be read byvnl-tail
as it is written into the log (just liketail -f
, but handling the legend properly). And you already know how to use these tools without even reading the manpages! Note: these were written for and have been tested with the Linux kernel and GNU Coreutilssort
,join
andtail
. Other kernels and tools probably don’t (yet) work. Send me patches.vnl-align
aligns vnlog columns for easy interpretation by humans. The meaning is unaffectedVnlog::Parser
is a simple perl library to read a vnlogvnlog
is a simple python library to read a vnlog. Both python2 and python3 are supportedlibvnlog
is a C library to simplify writing a vnlog. Clearly all you really need isprintf()
, but this is useful if we have lots of columns, many containing null values in any given row, and/or if we have parallel threads writing to a log. In my usage I have hundreds of columns, so this is handyvnl-make-matrix
converts a one-point-per-line vnlog to a matrix of data. I.e.
$ cat dat.vnl # i j x 0 0 1 0 1 2 0 2 3 1 0 4 1 1 5 1 2 6 2 0 7 2 1 8 2 2 9 3 0 10 3 1 11 3 2 12 $ < dat.vnl vnl-filter -p i,x | vnl-make-matrix --outdir /tmp Writing to '/tmp/x.matrix' $ cat /tmp/x.matrix 1 2 3 4 5 6 7 8 9 10 11 12
All the tools have manpages that contain more detail. And more tools will probably be added with time.
For most uses, these logfiles are simple enough to be generated with plain prints. But then each print statement has to know which numeric column we’re populating, which becomes effortful with many columns. In my usage it’s common to have a large parallelized C program that’s writing logs with hundreds of columns where any one record would contain only a subset of the columns. In such a case, it’s helpful to have a library that can output the log files. This is available. Basic usage looks like this:
In a shell:
$ vnl-gen-header 'int w' 'uint8_t x' 'char* y' 'double z' 'void* binary' > vnlog_fields_generated.h
In a C program test.c:
#include "vnlog_fields_generated.h"
int main()
{
vnlog_emit_legend();
vnlog_set_field_value__w(-10);
vnlog_set_field_value__x(40);
vnlog_set_field_value__y("asdf");
vnlog_emit_record();
vnlog_set_field_value__z(0.3);
vnlog_set_field_value__x(50);
vnlog_set_field_value__w(-20);
vnlog_set_field_value__binary("\x01\x02\x03", 3);
vnlog_emit_record();
vnlog_set_field_value__w(-30);
vnlog_set_field_value__x(10);
vnlog_set_field_value__y("whoa");
vnlog_set_field_value__z(0.5);
vnlog_emit_record();
return 0;
}
Then we build and run, and we get
$ cc -o test test.c -lvnlog $ ./test # w x y z binary -10 40 asdf - - -20 50 - 0.2999999999999999889 AQID -30 10 whoa 0.5 -
The binary field in base64-encoded. This is a rarely-used feature, but sometimes you really need to log binary data for later processing, and this makes it possible.
So you
- Generate the header to define your columns
- Call
vnlog_emit_legend()
- Call
vnlog_set_field_value__...()
for each field you want to set in that row. - Call
vnlog_emit_record()
to write the row and to reset all fields for the next row. Any fields unset with avnlog_set_field_value__...()
call are written as null:-
This is enough for 99% of the use cases. Things get a bit more complex if we have have threading or if we have multiple vnlog ouput streams in the same program. For both of these we use vnlog contexts.
To support independent writing into the same vnlog (possibly by multiple threads; this is reentrant), each log-writer should create a context, and use it when talking to vnlog. The context functions will make sure that the fields in each context are independent and that the output records won’t clobber each other:
void child_writer( // the parent context also writes to this vnlog. Pass NULL to
// use the global one
struct vnlog_context_t* ctx_parent )
{
struct vnlog_context_t ctx;
vnlog_init_child_ctx(&ctx, ctx_parent);
while(records)
{
vnlog_set_field_value_ctx__xxx(&ctx, ...);
vnlog_set_field_value_ctx__yyy(&ctx, ...);
vnlog_set_field_value_ctx__zzz(&ctx, ...);
vnlog_emit_record_ctx(&ctx);
}
vnlog_free_ctx(&ctx); // required only if we have any binary fields
}
If we want to have multiple independent vnlog writers to different streams (with different columns and legends), we do this instead:
file1.c
:
#include "vnlog_fields_generated1.h"
void f(void)
{
// Write some data out to the default context and default output (STDOUT)
vnlog_emit_legend();
...
vnlog_set_field_value__xxx(...);
vnlog_set_field_value__yyy(...);
...
vnlog_emit_record();
}
file2.c
:
#include "vnlog_fields_generated2.h"
void g(void)
{
// Make a new session context, send output to a different file, write
// out legend, and send out the data
struct vnlog_context_t ctx;
vnlog_init_session_ctx(&ctx);
FILE* fp = fopen(...);
vnlog_set_output_FILE(&ctx, fp);
vnlog_emit_legend_ctx(&ctx);
...
vnlog_set_field_value__a(...);
vnlog_set_field_value__b(...);
...
vnlog_free_ctx(&ctx); // required only if we have any binary fields
vnlog_emit_record();
}
Note that it’s the user’s responsibility to make sure the new sessions go to a
different FILE
by invoking vnlog_set_output_FILE()
. Furthermore, note that
the included vnlog_fields_....h
file defines the fields we’re writing to; and
if we have multiple different vnlog field definitions in the same program (as in
this example), then the different writers must live in different source files.
The compiler will barf if you try to #include
two different
vnlog_fields_....h
files in the same source.
vnlog_printf(...)
andvnlog_printf_ctx(ctx, ...)
write to a pipe like
printf()
does. This exists primarily for comments.
vnlog_clear_fields_ctx(ctx, do_free_binary)
clears out the data in a context
and makes it ready to be used for the next record. It is rare for the user to have to call this manually. The most common case is handled automatically (clearing out a context after emitting a record). One area where this is useful is when making a copy of a context:
struct vnlog_context_t ctx1;
// .... do stuff with ctx1 ... add data to it ...
struct vnlog_context_t ctx2 = ctx1;
// ctx1 and ctx2 now both have the same data, and the same pointers to
// binary data. I need to get rid of the pointer references in ctx1
vnlog_clear_fields_ctx(&ctx1, false);
vnlog_free_ctx(ctx)
frees memory for an vnlog context. Do this before
throwing the context away. Currently this is only needed for context that have binary fields, but this should be called for all contexts anyway, in case this changes in a later revision
The C interface supports writing base64-encoded binary data using Chris Venter’s
libb64. The base64-encoder used here was slightly modified: the output appears
all on one line, making is suitable to appear in a vnlog field. If we’re writing
a vnlog with printf()
directly without using the vnlog.h
interface described
above, we allow this modified base64 encoder to be invoked by itself. Usage:
void* binary_buffer = ...;
int binary_buffer_len = ...;
char base64_buffer[vnlog_base64_dstlen_to_encode(binary_buffer_len)];
vnlog_base64_encode( base64_buffer, sizeof(base64_buffer),
binary_buffer, binary_buffer_len );
Clearly the above example allocates the base64 buffer on the stack, so it’s only suitable for small-ish data chunks. But if you have lots and lots of data, probably writing it as base64 into a vnlog isn’t the best thing to do.
The built-in numpy.loadtxt
numpy.savetxt
functions work well to read and
write these files. For example to write to standard output a vnlog with fields
a
, b
and c
:
numpy.savetxt(sys.stdout, array, fmt="%g", header="a b c")
Note that numpy automatically adds the #
to the header. To read a vnlog from a
file on disk, do something like
array = numpy.loadtxt('data.vnl')
These functions know that #
lines are comments, but don’t interpret anything
as field headers. That’s easy to do, so I’m not providing any helper libraries.
I might do that at some point, but in the meantime, patches are welcome.
A common use case is a complex application that produces several semi-related subsets of data at once. Example: a moving vehicle is reporting both its own position and the observed positions of other vehicles; at any given time any number of other vehicles may be observed. Two equivalent workflows are possible:
- a single unified vnlog stream for all the data
- several discrete vnlog streams for each data subset
Both are valid approaches
Here the application produces a single vnlog that contains all the columns,
from all the data subsets. In any given row, many of the columns will be empty
(i.e. contain only -
). For instance, a row describing a vehicle own position
will not have data about any observations, and vice versa. It is inefficient to
store all the extra -
but it makes many things much nicer, so it’s often worth
it. vnl-filter
can be used to pull out the different subsets. Sample
joint.vnl
:
# time x_self x_observation 1 10 - 2 20 - 2 - 100 3 30 - 3 - 200 3 - 300
Here we have 3 instances in time. We have no observations at time
1, one
observation at time
2, and two observations at time
3. We can use
vnl-filter
to pull out the data we want:
$ < joint.vnl vnl-filter -p time,self # time x_self 1 10 2 20 2 - 3 30 3 - 3 -
If we only care about our own positions, the +
modifier in picked columns in
vnl-filter
is very useful here:
$ < joint.vnl vnl-filter -p time,+self # time x_self 1 10 2 20 3 30 $ < joint.vnl vnl-filter -p time,+observation # time x_observation 2 100 3 200 3 300
Note that the default is --skipempty
, so if we’re only looking at x_self
for instance, then we don’t even need to +
modifier:
$ < joint.vnl vnl-filter -p self # x_self 10 20 30
Also, note that the vnlog
C interface works very nicely to produce these
datafiles:
- You can define lots and lots of columns, but only fill some of them before
calling
vnlog_emit_record()
. The rest will be set to-
. - You can create multiple contexts for each type of data, and you can populate
them with data independently. And when calling
vnlog_emit_record_ctx()
, you’ll get a record with data for just that context.
Conversely, the application can produce separate vnlog streams for each
subset of data. Depending on what is desired, exactly, vnl-join
can be used to
re-join them:
$ cat self.vnl # time x_self 1 10 2 20 3 30 $ cat observations.vnl # time x_observation 2 100 3 200 3 300 $ vnl-join -j time -a- self.vnl observations.vnl # time x_self x_observation 1 10 - 2 20 100 3 30 200 3 30 300
A common need is to compute basic statistics from your data. Many of the
alternative toolkits listed above provide built-in facilities to do this, but
vnlog does not: it’s meant to be unixy, where each tool has very limited scope.
Thus you can either do this with awk
like you would normally, or you can use
other standalone tools to perform the needed computations. For instance, I can
generate some data:
$ seq 2 100 | awk 'BEGIN {print "# x"} {print log($1)}' > /tmp/log.vnl
Then I can compute the mean with awk
:
$ < /tmp/log.vnl vnl-filter --eval '{sum += x} END {print sum/NR}' 3.67414
Or I can compute the mean (and other stuff) with a separate standalone tool:
$ < /tmp/log.vnl ministat x <stdin> +----------------------------------------------------------------------------+ | xx | | x xxxxxxx | | xx xxxxxxxxxxxx| | x x xxxxxxxxxxxxxxxxxxxxxxx| |x x x x x x x x x xx xx xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| | |_______________A____M___________| | +----------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 99 0.693147 4.60517 3.93183 3.6741353 0.85656382
ministat
is not a part of the vnlog toolkit, but the vnlog format is generic
so it works just fine.
Everything about vnlog is generic and simple, so it’s easy to use it to process
data that wasn’t originally meant to be used this way. For instance filtering
the output of ls -l
to report only file names and sizes, skipping directories,
and sorting by file sizes:
$ ls -l total 320 -rw-r--r-- 1 dima dima 5044 Aug 25 15:04 Changes -rw-r--r-- 1 dima dima 12749 Aug 25 15:04 Makefile -rw-r--r-- 1 dima dima 69789 Aug 25 15:04 README.org -rw-r--r-- 1 dima dima 33781 Aug 25 15:04 README.template.org -rw-r--r-- 1 dima dima 5359 Aug 25 15:04 b64_cencode.c drwxr-xr-x 4 dima dima 4096 Aug 25 15:04 completions drwxr-xr-x 3 dima dima 4096 Aug 25 15:04 lib drwxr-xr-x 3 dima dima 4096 Aug 25 15:04 packaging drwxr-xr-x 2 dima dima 4096 Aug 25 15:04 test -rwxr-xr-x 1 dima dima 5008 Aug 25 15:04 vnl-align -rwxr-xr-x 1 dima dima 56637 Aug 25 15:04 vnl-filter -rwxr-xr-x 1 dima dima 5678 Aug 25 15:04 vnl-gen-header -rwxr-xr-x 1 dima dima 29815 Aug 25 15:04 vnl-join -rwxr-xr-x 1 dima dima 3631 Aug 25 15:04 vnl-make-matrix -rwxr-xr-x 1 dima dima 8372 Aug 25 15:04 vnl-sort -rwxr-xr-x 1 dima dima 5822 Aug 25 15:04 vnl-tail -rwxr-xr-x 1 dima dima 4439 Aug 25 15:04 vnl-ts -rw-r--r-- 1 dima dima 559 Aug 25 15:04 vnlog-base64.h -rw-r--r-- 1 dima dima 8169 Aug 25 15:04 vnlog.c -rw-r--r-- 1 dima dima 12677 Aug 25 15:04 vnlog.h $ (echo '# permissions num_links user group size month day time name'; ls -l | tail -n +2) | vnl-filter 'permissions !~ "^d"' -p name,size | vnl-sort -gk size | vnl-align # name size vnlog-base64.h 559 vnl-make-matrix 3631 vnl-ts 4439 vnl-align 5008 Changes 5044 b64_cencode.c 5359 vnl-gen-header 5678 vnl-tail 5822 vnlog.c 8169 vnl-sort 8372 vnlog.h 12677 Makefile 12749 vnl-join 29815 README.template.org 33781 vnl-filter 56637 README.org 69789
With a bit of shell manipulation, these tools can be applied to a whole lot of different data streams that know nothing of vnlog.
The tools that wrap standard commandline utilities (vnl-sort
, vnl-uniq
,
vnl-join
, vnl-tail
, vnl-ts
) are written specifically to work with the
Linux kernel and the GNU coreutils. None of these have been tested with BSD
tools or with non-Linux kernels, and I’m sure things don’t just work. It’s
probably not too effortful to get that running, but somebody needs to at least
bug me for that. Or better yet, send me nice patches :)
These tools are meant to be simple, so some things are hard requirements. A big one is that columns are whitespace-separated. There is no mechanism for escaping or quoting whitespace into a single field. I think supporting something like that is more trouble than it’s worth.
At this time vnlog is a part of Debian/buster and Ubuntu/cosmic (18.10). On those boxes you can simply
$ sudo apt install vnlog libvnlog-dev libvnlog-perl python-vnlog
to get the binary tools, the C API, the perl and python2 interfaces respectively.
On a Debian (or Ubuntu) machine that’s too old to have the packages already available, you can build and install them:
$ git clone [email protected]:dkogan/vnlog.git $ cd vnlog $ cp -r packaging/debian . $ dpkg-buildpackage -us -uc -b $ sudo dpkg -i ../vnlog*.deb ../libvnlog-dev*.deb ../libvnlog-perl*.deb ../python-vnlog*.deb
With the exception of the C API, every part of the toolkit is written in an interpreted language, and there’s nothing to “install”. You can run everything directly from the source tree:
$ git clone [email protected]:dkogan/vnlog.git $ cd vnlog $ ./vnl-filter .....
If you do want to install to some location, do this:
$ make $ PREFIX=/usr/local make install
NAME vnl-filter - filters vnlogs to select particular rows, fields SYNOPSIS $ cat run.vnl # time x y z temperature 3 1 2.3 4.8 30 4 1.1 2.2 4.7 31 6 1 2.0 4.0 35 7 1 1.6 3.1 42 $ <run.vnl vnl-filter -p x,y,z | vnl-align # x y z 1 2.3 4.8 1.1 2.2 4.7 1 2.0 4.0 1 1.6 3.1 $ <run.vnl vnl-filter -p i=NR,time,'dist=sqrt(x*x + y*y + z*z)' | vnl-align # i time dist 1 3 5.41572 2 4 5.30471 3 6 4.58258 4 7 3.62905 $ <run.vnl vnl-filter 'temperature >= 35' | vnl-align # time x y z temperature 6 1 2.0 4.0 35 7 1 1.6 3.1 42 $ <run.vnl vnl-filter --eval '{s += temperature} END { print "mean temp: " s/NR}' mean temp: 34.5 $ <run.vnl vnl-filter -p x,y | feedgnuplot --terminal 'dumb 80,30' --unset grid --domain --lines --exit 2.3 +---------------------------------------------------------------------+ | + + *************** + | | ************** | | *******| 2.2 |-+ ************| | ******** | | ******** | 2.1 |-+ ********* +-| | ******** | | ******** | | **** | 2 |-+ * +-| | * | | * | | * | 1.9 |-+ * +-| | * | | * | | * | 1.8 |-+ * +-| | * | | * | 1.7 |-+ * +-| | * | | * | | * + + + + | 1.6 +---------------------------------------------------------------------+ 0.98 1 1.02 1.04 1.06 1.08 1.1 DESCRIPTION This tool is largely a frontend for awk to operate on vnlog files. Vnlog is both an input and an output. This tool makes it very simple to select specific rows and columns for output and to manipulate the data in various ways. This is a UNIX-style tool, so the input/output of this tool is strictly STDIN/STDOUT. Furthermore, in its usual form this tool is a filter, so the format of the output is *exactly* the same as the format of the input. The exception to this is when using "--eval", in which the output is dependent on whatever expression we're evaluating. This tool is convenient to process both stored data or live data; in the latter case, it's very useful to pipe the streaming output to "feedgnuplot --stream" to get a realtime visualization of the incoming data. This tool reads enough of the input file to get a legend, at which point it constructs an awk program to do the main work, and execs to awk (it's possible to use perl as well, but this isn't as fast). Input/output data format The input/output data is vnlog: a plain-text table of values. Any lines beginning with "#" are treated as comments, and are passed through. The first line that begins with "#" but not "##" or "#!" is a *legend* line. After the "#", follow whitespace-separated field names. Each subsequent line is whitespace-separated values matching this legend. For instance, this is a valid vnlog file: #!/usr/bin/something ## more comments # x y z -0.016107 0.004362 0.005369 -0.017449 0.006711 0.006711 -0.018456 0.014093 0.006711 -0.017449 0.018791 0.006376 "vnl-filter" uses this format for both the input and the output. The comments are preserved, but the legend is updated to reflect the fields in the output file. A string "-" is used to indicate an undefined value, so this is also a valid vnlog file: # x y z 1 2 3 4 - 6 - - 7 Filtering To select specific *columns*, pass their names to the "-p" option (short for "--print" or "--pick", which are synonyms). In its simplest form, to grab only columns "x" and "y", do vnl-filter -p x,y See the detailed description of "-p" below for more detail. To select specific *rows*, we use *matches* expressions. Anything on the "vnl-filter" commandline and not attached to any "--xxx" option is such an expression. For instance vnl-filter 'size > 10' would select only those rows whose "size" column contains a value > 10. See the detailed description of matches expressions below for more detail. Context lines "vnl-filter" supports the context output options ("-A", "-B" and "-C") exactly like the "grep" tool. I.e to print out all rows whose "size" column contains a value > 10 *but also* include the 3 rows immediately before *and* after such matching rows, do this: vnl-filter -C3 'size > 10' "-B" reports the rows *before* matching ones and "-A" the rows *after* matching ones. "-C" reports both. Note that this applies *only* to *matches* expressions: records skipped because they fail "--has" or "--skipempty" are *not* included in contextual output. Backend choice By default, the parsing of arguments and the legend happens in perl, which then constructs a simple awk script, and invokes "mawk" to actually read the data and to process it. This is done because awk is lighter weight and runs faster, which is important because our data sets could be quite large. We default to "mawk" specifically, since this is a simpler implementation than "gawk", and runs much faster. If for whatever reason we want to do everything with perl, this can be requested with the "--perl" option. Special functions For convenience we support several special functions in any expression passed on to awk or perl (named expressions, matches expressions, "--eval" strings). These generally maintain some internal state, and vnl-filter makes sure that this state is consistent. Note that these are evaluated *after* "--skipcomments" and "--has". So any record skipped because of a "--has" expression, for instance, will *not* be considered in "prev()", "diff()" and so on. * rel(x) returns value of "x" relative to the first value of "x". For instance we might want to see the time or position relative to the start, not relative to some absolute beginning. Example: $ cat tst.vnl # time x 100 200 101 212 102 209 $ <tst.vnl vnl-filter -p 't=rel(time),x=rel(x) # t x 0 0 1 12 2 9 * diff(x) returns the difference between the current value of "x" and the previous value of "x". The first row will always be -. Example: $ <tst.vnl vnl-filter -p x,'d1=diff(x),d2=diff(diff(x))' | vnl-align # x d1 d2 1 - - 8 7 7 27 19 12 64 37 18 125 61 24 * sum(x) returns the cumulative sum of "x". As diff(x) can be thought of as a derivative, sum(x) can be thought of as an integral. So "diff(sum(x))" would return the same value as "x" (except for the first row; "diff()" always returns - for the first row). Example: $ <tst.vnl vnl-filter -p 'x,s=sum(x),ds=diff(sum(x))' | vnl-align # x s ds 1 1 - 8 9 8 27 36 27 64 100 64 125 225 125 * prev(x) returns the previous value of "x". One could construct "sum()" and "rel()" using this, if they weren't already available. ARGUMENTS -p|--print|--pick expr These option provide the mechanism to select specific columns for output. For instance to pull out columns called "lat", "lon", and any column whose name contains the string "feature_", do vnl-filter -p lat,lon,'feature_.*' or, equivalently vnl-filter --print lat --print lon --print 'feature_.*' We look for exact column name matches first, and if none are found, we try a regex. If there was no column called exactly "feature_", then the above would be equivalent to vnl-filter -p lat,lon,feature_ This mechanism is much more powerful than just selecting columns. First off, we can rename chosen fields: vnl-filter -p w=feature_width would pick the "feature_width" field, but the resulting column in the output would be named "w". When renaming a column in this way regexen are *not* supported, and exact field names must be given. But the string to the right of the "=" is passed on directly to awk (after replacing field names with column indices), so any awk expression can be used here. For instance to compute the length of a vector in separate columns "x", "y", and "z" you can do: vnl-filter -p 'l=sqrt(x*x + y*y + z*z)' A single column called "l" would be produced. We can also *exclude* columns by preceding their name with "!". This works like you expect. Rules: * The pick/exclude directives are processed in order given to produce the output picked-column list * If the first "-p" item is an exclusion, we implicitly pick *all* the columns prior to processing the "-p". * The exclusion expressions match the *output* column names, not the *input* names. * We match the exact column names first. If that fails, we match as a regex Example. To grab all the columns *except* the temperature(s) do this: vnl-filter -p !temperature To grab all the columns that describe *something* about a robot (columns whose names have the string "robot_" in them), but *not* its temperature (i.e. *not* "robot_temperature"), do this: vnl-filter -p robot_,!temperature --has a,b,c,... Used to select records (rows) that have a non-empty value in a particular field (column). A *null* value in a column is designated with a single "-". If we want to select only records that have a value in the "x" column, we pass "--has x". To select records that have data for *all* of a given set of columns, the "--has" option can be repeated, or these multiple columns can be given in a whitespace-less comma-separated list. For instance if we want only records that have data in *both* columns "x" *and* "y" we can pass in "--has x,y" or "--has x --has y". If we want to combine multiple columns in an *or* (select rows that have data in *any* of a given set of columns), use a matches expression, as documented below. If we want to select a column *and* pick only rows that have a value in this column, a shorthand syntax exists: vnl-filter --has col -p col is equivalent to vnl-filter -p +col Note that just like the column specifications in "-p" the columns given to "--has" must match exactly *or* as a regex. In either case, a unique matching column must be found. Matches expressions Anything on the commandline not attached to any "--xxx" option is a *matches* expression. These are used to select particular records (rows) in a data file. For each row, we evaluate all the expressions. If *all* the expressions evaluate to true, that row is output. This expression is passed directly to the awk (or perl) backend. Example: to select all rows that have valid data in column "a" *or* column "b" *or* column "c" you can vnl-filter 'a != "-" || b != "-" || c != "-"' or vnl-filter --perl 'defined a || defined b || defined c' As with the named expressions given to "-p" (described above), these are passed directly to awk, so anything that can be done with awk is supported here. -A N|--after-context N Output N lines following each *matches* expression, even those lines that do not themselves match. This works just like the "grep" options of the same name. See "Context lines" -B N|--before-context N Output N lines preceding each *matches* expression, even those lines that do not themselves match. This works just like the "grep" options of the same name. See "Context lines" -C N|--context N Output N lines preceding and following each *matches* expression, even those lines that do not themselves match. This works just like the "grep" options of the same name. See "Context lines" --eval expr Instead of printing out all matching records and picked columns, just run the given chunk of awk (or perl). In this mode of operation, "vnl-filter" acts just like a glorified awk, that allows fields to be accessed by name instead of by number, as it would be in raw awk. Since the expression may print *anything* or nothing at all, the output in this mode is not necessarily itself a valid vnlog stream. And no column-selecting arguments should be given, since they make no sense in this mode. In awk the expr is a full set of pattern/action statements. So to print the sum of columns "a" and "b" in each row, and at the end, print the sum of all values in the "a" column vnl-filter --eval '{print a+b; suma += a} END {print suma}' In perl the arbitrary expression fits in like this: while(<>) # read each line { next unless matches; # skip non-matching lines eval expression; # evaluate the arbitrary expression } --function|--sub Evaluates the given expression as a function that can be used in other expressions. This is most useful when you want to print something that can't trivially be written as a simple expression. For instance: $ cat tst.vnl # s 1-2 3-4 5-6 $ < tst.vnl vnl-filter --function 'before(x) { sub("-.*","",x); return x }' \ --function 'after(x) { sub(".*-","",x); return x }' \ -p 'b=before(s),a=after(s)' # b a 1 2 3 4 5 6 See the CAVEATS section below if you're doing something sufficiently-complicated where you need this. --[no]skipempty Do [not] skip records where all fields are blank. By default we *do* skip all empty records; to include them, pass "--noskipempty" --skipcomments Don't output non-legend comments --perl By default all procesing is performed by "mawk", but if for whatever reason we want perl instead, pass "--perl". Both modes work, but "mawk" is noticeably faster. "--perl" could be useful because it is more powerful, which could be important since a number of things pass commandline strings directly to the underlying language (named expressions, matches expressions, "--eval" strings). Note that while variables in perl use sigils, column references should *not* use sigils. To print the sum of all values in column "a" you'd do this in awk vnl-filter --eval '{suma += a} END {print suma}' and this in perl vnl-filter --perl --eval '{$suma += a} END {say $suma}' The perl strings are evaluated without "use strict" or "use warnings" so I didn't have to declare $suma in the example. --dumpexprs Used for debugging. This spits out all the final awk (or perl) program we run for the given commandline options and given input. This is the final program, with the column references resolved to numeric indices, so one can figure out what went wrong. --unbuffered Flushes each line after each print. This makes sure each line is output as soon as it is available, which is crucial for realtime output and streaming plots. --stream Synonym for "--unbuffered" CAVEATS This tool is very lax in its input validation (on purpose). As a result, columns with names like %CPU and "TIME+" do work (i.e. you can more or less feed in output from "top -b"). The downside is that shooting yourself in the foot is possible. This tradeoff is currently tuned to be very permissive, which works well for my use cases. I'd be interested in hearing other people's experiences. Potential pitfalls/unexpected behaviors: * All column names are replaced in all eval strings without regard to context. The earlier example that reports the sum of values in a column: "vnl-filter --eval '{suma += a} END {print suma}'" will work fine if we *do* have a column named "a" and do *not* have a column named "suma". But will not do the right thing if any of those are violated. For instance, if a column "a" doesn't exist, then "awk" would see "suma += a" instead of something like "suma += $5". "a" would be an uninitialized variable, which evaluates to 0, so the full "vnl-filter" command would not fail, but would print 0 instead. It's the user's responsibility to make sure we're talking about the right columns. The focus here was one-liners so hopefully nobody has so many columns, they can't keep track of all of them in their head. I don't see any way to resolve this without seriously impacting the scope of the tool, so I'm leaving this alone. * It is natural to use vnlog as a database. You can run queries with something like vnl-filter 'key == 5' This works. But unlike a real database this is clearly a linear lookup. With large data files, this would be significantly slower than the logarithmic searches provided by a real database. The meaning of "large" and "significant" varies, and you should test it. In my experience vnlog "databases" scale surprisingly well. But at some point, importing your data to something like sqlite is well worth it. * When substituting column names I match *either* a word-nonword transition ("\b") *or* a whitespace-nonword transition. The word boundaries is what would be used 99% of the time. But the keys may have special characters in them, which don't work with "\b". This means that whitespace becomes important: "1+%CPU" will not be parsed as expected, which is correct since "+%CPU" is also a valid field name. But "1+ %CPU" will be parsed correctly, so if you have weird field names, put the whitespace into your expressions. It'll make them more readable anyway. * Strings passed to "-p" are split on "," *except* if the "," is inside balanced "()". This makes it possible to say things like "vnl-filter --function 'f(a,b) { ... }' -p 'c=f(a,b)'". This is probably the right behavior, although some questionable looking field names become potentially impossible: "f(a" and "b)" *could* otherwise be legal field names, but you're probably asking for trouble if you do that. * Currently there're two modes: a pick/print mode and an "--eval" mode. Then there's also "--function", which adds bits of "--eval" to the pick/print mode, but it feels maybe insufficient. I don't yet have strong feelings about what this should become. Comments welcome
NAME vnl-align - aligns vnlog columns for easy interpretation by humans SYNOPSIS $ cat tst.vnl # w x y z -10 40 asdf - -20 50 - 0.300000 -30 10 whoa 0.500000 $ vnl-align tst.vnl # w x y z -10 40 asdf - -20 50 - 0.300000 -30 10 whoa 0.500000 DESCRIPTION The basic usage is vnl-align logfile The arguments are assumed to be the vnlog files. If no arguments are given, the input comes from STDIN. This is very similar to "column -t", but handles "#" lines properly: 1. The first "#" line is the legend. For the purposes of alignment, the leading "#" character and the first column label are treated as one column 2. All other "#" lines are output verbatim.
NAME vnl-sort - sorts an vnlog file, preserving the legend SYNOPSIS $ cat a.vnl # a b AA 11 bb 12 CC 13 dd 14 dd 123 Sort lexically by a: $ <a.vnl vnl-sort -k a # a b AA 11 CC 13 bb 12 dd 123 dd 14 Sort lexically by a, ignoring case: $ <a.vnl vnl-sort -k a --ignore-case # a b AA 11 bb 12 CC 13 dd 123 dd 14 Sort lexically by a, then numerically by b: $ <a.vnl vnl-sort -k a -k b.n # a b AA 11 CC 13 bb 12 dd 14 dd 123 Sort lexically by a, then numerically by b in reverse: $ <a.vnl vnl-sort -k a -k b.nr # a b AA 11 CC 13 bb 12 dd 123 dd 14 Sort by month and then day: $ cat dat.vnl # month day March 5 Jan 2 Feb 1 March 30 Jan 21 $ <dat.vnl vnl-sort -k month.M -k day.n # month day Jan 2 Jan 21 Feb 1 March 5 March 30 DESCRIPTION Usage: vnl-sort [options] logfile logfile logfile ... < logfile This tool sorts given vnlog files in various ways. "vnl-sort" is a wrapper around the GNU coreutils "sort" tool. Since this is a wrapper, most commandline options and behaviors of the "sort" tool are present; consult the sort(1) manpage for detail. The differences from GNU coreutils "sort" are * The input and output to this tool are vnlog files, complete with a legend * The columns are referenced by name, not index. So instead of saying sort -k1 to sort by the first column, you say sort -k time to sort by column "time". * The fancy "KEYDEF" spec from "sort" is only partially supported. I only allow us to sort by full *fields*, so the start/stop positions don't make sense. I *do* support the "OPTS" to change the type of sorting in a given particular column. For instance, to sort by month and then by day, do this (see example above): vnl-sort -k month.M -k day.n * "--files0-from" is not supported due to lack of time. If somebody really needs it, talk to me. * "--output" is not supported due to an uninteresting technical limitation. The output always goes to standard out. * "--field-separator" is not supported because vnlog assumes whitespace-separated fields * "--zero-terminated" is not supported because vnlog assumes newline-separated records Past that, everything "sort" does is supported, so see that man page for detailed documentation. Note that all non-legend comments are stripped out, since it's not obvious where they should end up. BUGS This and the other "vnl-xxx" tools that wrap coreutils are written specifically to work with the Linux kernel and the GNU coreutils. None of these have been tested with BSD tools or with non-Linux kernels, and I'm sure things don't just work. It's probably not too effortful to get that running, but somebody needs to at least bug me for that. Or better yet, send me nice patches :) SEE ALSO sort(1)
NAME vnl-uniq - uniq a log file, preserving the legend SYNOPSIS $ cat colors.vnl # color blue yellow yellow blue yellow orange orange $ < colors.vnl | vnl-sort | vnl-uniq -c # count color 2 blue 2 orange 3 yellow DESCRIPTION Usage: vnl-uniq [options] < logfile This tool runs "uniq" on a given vnlog dataset. "vnl-uniq" is a wrapper around the GNU coreutils "uniq" tool. Since this is a wrapper, most commandline options and behaviors of the "uniq" tool are present; consult the uniq(1) manpage for detail. The differences from GNU coreutils "uniq" are * The input and output to this tool are vnlog files, complete with a legend * "--zero-terminated" is not supported because vnlog assumes newline-separated records * Only *one* input is supported (a file on the cmdline or data on standard input), and the output *always* goes to standard output. Specifying the output as a file on the commandline is not supported. * "--vnl-count NAME" can be given to name the "count" column. "-c" is still supported to add the default new column named "count", but if another name is wanted, "--vnl-count" does that. "--vnl-count" implies "-c" * In addition to the normal behavior of skipping fields at the start, "-f" and "--skip-fields" can take a negative argument to skip the *all but the last* N fields. For instance, to use only the one last field, pass "-f -1" or "--skip-fields=-1". Past that, everything "uniq" does is supported, so see that man page for detailed documentation. BUGS This and the other "vnl-xxx" tools that wrap coreutils are written specifically to work with the Linux kernel and the GNU coreutils. None of these have been tested with BSD tools or with non-Linux kernels, and I'm sure things don't just work. It's probably not too effortful to get that running, but somebody needs to at least bug me for that. Or better yet, send me nice patches :) SEE ALSO uniq(1)
NAME vnl-join - joins two log files on a particular field SYNOPSIS $ cat a.vnl # a b AA 11 bb 12 CC 13 dd 14 dd 123 $ cat b.vnl # a c aa 1 cc 3 bb 4 ee 5 - 23 Try to join unsorted data on field 'a': $ vnl-join -j a a.vnl b.vnl # a b c join: /dev/fd/5:3: is not sorted: CC 13 join: /dev/fd/6:3: is not sorted: bb 4 Sort the data, and join on 'a': $ vnl-join --vnl-sort - -j a a.vnl b.vnl | vnl-align # a b c bb 12 4 Sort the data, and join on 'a', ignoring case: $ vnl-join -i --vnl-sort - -j a a.vnl b.vnl | vnl-align # a b c AA 11 1 bb 12 4 CC 13 3 Sort the data, and join on 'a'. Also print the unmatched lines from both files: $ vnl-join -a1 -a2 --vnl-sort - -j a a.vnl b.vnl | vnl-align # a b c - - 23 AA 11 - CC 13 - aa - 1 bb 12 4 cc - 3 dd 123 - dd 14 - ee - 5 Sort the data, and join on 'a'. Print the unmatched lines from both files, Output ONLY column 'c' from the 2nd input: $ vnl-join -a1 -a2 -o 2.c --vnl-sort - -j a a.vnl b.vnl | vnl-align # c 23 - - 1 4 3 - - 5 DESCRIPTION Usage: vnl-join [join options] [--vnl-sort -|[sdfgiMhnRrV]+] [ --vnl-[pre|suf]fix[1|2] xxx | --vnl-[pre|suf]fix xxx,yyy,zzz | --vnl-autoprefix | --vnl-autosuffix ] logfile1 logfile2 This tool joins two vnlog files on a given field. "vnl-join" is a wrapper around the GNU coreutils "join" tool. Since this is a wrapper, most commandline options and behaviors of the "join" tool are present; consult the join(1) manpage for detail. The differences from GNU coreutils "join" are * The input and output to this tool are vnlog files, complete with a legend * The columns are referenced by name, not index. So instead of saying join -j1 to join on the first column, you say join -j time to join on column "time". * -1 and -2 are supported, but *must* refer to the same field. Since vnlog knows the identify of each field, it makes no sense for -1 and -2 to be different. So pass "-j" instead, it makes more sense in this context. * "-a-" is available as a shorthand for "-a1 -a2": this is a full outer join, printing unmatched records from both of the inputs. Similarly, "-v-" is available as a shorthand for "-v1 -v2": this will output *only* the unique records in both of the inputs. * "vnl-join"-specific options are available to adjust the field-naming in the output: --vnl-prefix1 --vnl-suffix1 --vnl-prefix2 --vnl-suffix2 --vnl-prefix --vnl-suffix --vnl-autoprefix --vnl-autosuffix See "Field names in the output" below for details. * A "vnl-join"-specific option "--vnl-sort" is available to sort the input and/or output. See below for details. * If no "-o" is given, we output the join field, the remaining fields in logfile1, the remaining fields in logfile2, .... This is what "-o auto" does, except we also handle empty vnlogs correctly. * "-e" is not supported because vnlog uses "-" to represent undefined fields. * "--header" is not supported because vnlog assumes a specific header structure, and "vnl-join" makes sure that this header is handled properly * "-t" is not supported because vnlog assumes whitespace-separated fields * "--zero-terminated" is not supported because vnlog assumes newline-separated records * Rather than only 2-way joins, this tool supports N-way joins for any N > 2. See below for details. Past that, everything "join" does is supported, so see that man page for detailed documentation. Note that all non-legend comments are stripped out, since it's not obvious where they should end up. Field names in the output By default, the field names in the output match those in the input. This is what you want most of the time. It is possible, however that a column name adjustment is needed. One common use case for this is if the files being joined have identically-named columns, which would produce duplicate columns in the output. Example: we fixed a bug in a program, and want to compare the results before and after the fix. The program produces an x-y trajectory as a function of time, so both the bugged and the bug-fixed programs produce a vnlog with a legend # time x y Joining this on "time" will produce a vnlog with a legend # time x y x y which is confusing, and *not* what you want. Instead, we invoke "vnl-join" as vnl-join --vnl-suffix1 _buggy --vnl-suffix2 _fixed -j time buggy.vnl fixed.vnl And in the output we get a legend # time x_buggy y_buggy x_fixed y_fixed Much better. Note that "vnl-join" provides several ways of specifying this. The above works *only* for 2-way joins. An alternate syntax is available for N-way joins, a comma-separated list. The same could be expressed like this: vnl-join -a- --vnl-suffix _buggy,_fixed -j time buggy.vnl fixed.vnl Finally, if passing in structured filenames, "vnl-join" can infer the desired syntax from the filenames. The same as above could be expressed even simpler: vnl-join --vnl-autosuffix -j time buggy.vnl fixed.vnl This works by looking at the set of passed in filenames, and stripping out the common leading and trailing strings. Sorting of input and output The GNU coreutils "join" tool expects sorted columns because it can then take only a single pass through the data. If the input isn't sorted, then we can use normal shell substitutions to sort it: $ vnl-join -j key <(vnl-sort -s -k key a.vnl) <(vnl-sort -s -k key b.vnl) For convenience "vnl-join" provides a "--vnl-sort" option. This allows the above to be equivalently expressed as $ vnl-join -j key --vnl-sort - a.vnl b.vnl The "-" after the "--vnl-sort" indicates that we want to sort the *input* only. If we also want to sort the output, pass the short codes "sort" accepts instead of the "-". For instance, to sort the input for "join" and to sort the output numerically, in reverse, do this: $ vnl-join -j key --vnl-sort rg a.vnl b.vnl The reason this shorthand exists is to work around a quirk of "join". The sort order is *assumed* by "join" to be lexicographical, without any way to change this. For "sort", this is the default sort order, but "sort" has many options to change the sort order, options which are sorely missing from "join". A real-world example affected by this is the joining of numerical data. If you have "a.vnl": # time a 8 a 9 b 10 c and "b.vnl": # time b 9 d 10 e Then you cannot use "vnl-join" directly to join the data on time: $ vnl-join -j time a.vnl b.vnl # time a b join: /dev/fd/4:3: is not sorted: 10 c join: /dev/fd/5:2: is not sorted: 10 e 9 b d 10 c e Instead you must re-sort both files lexicographically, *and* then (because you almost certainly want to) sort it back into numerical order: $ vnl-join -j time <(vnl-sort -s -k time a.vnl) <(vnl-sort -s -k time b.vnl) | vnl-sort -s -n -k time # time a b 9 b d 10 c e Yuck. The shorthand described earlier makes the interface part of this palatable: $ vnl-join -j time --vnl-sort n a.vnl b.vnl # time a b 9 b d 10 c e Note that the input sort is stable: "vnl-join" will invoke "vnl-sort -s". If you want a stable post-sort, you need to ask for it with "--vnl-sort s...". N-way joins The GNU coreutils "join" tool is inherently designed to join *exactly* two files. "vnl-join" extends this capability by chaining together a number of "join" invocations to produce a generic N-way join. This works exactly how you would expect with the following caveats: * Full outer joins are supported by passing "-a-", but no other "-a" option is supported. This is possible, but wasn't obviously worth the trouble. * "-v" is not supported. Again, this is possible, but wasn't obviously worth the trouble. * Similarly, "-o" is not supported. This is possible, but wasn't obviously worth the trouble, especially since the desired behavior can be obtained by post-processing with "vnl-filter". BUGS AND CAVEATS The underlying "sort" tool assumes lexicographic ordering, and matches fields purely based on their textual contents. This means that for the purposes of joining, 10, 10.0 and 1.0e1 are all considered different. If needed, you can normalize your keys with something like this: vnl-filter -p x='sprintf("%f",x)' This and the other "vnl-xxx" tools that wrap coreutils are written specifically to work with the Linux kernel and the GNU coreutils. None of these have been tested with BSD tools or with non-Linux kernels, and I'm sure things don't just work. It's probably not too effortful to get that running, but somebody needs to at least bug me for that. Or better yet, send me nice patches :) SEE ALSO join(1)
NAME vnl-tail - tail a log file, preserving the legend SYNOPSIS $ read_temperature | tee temp.vnl # temperature 29.5 30.4 28.3 22.1 ... continually produces data ... at the same time, in another terminal $ vnl-tail -f temp.vnl # temperature 28.3 22.1 ... outputs data as it comes in DESCRIPTION Usage: vnl-tail [options] logfile logfile logfile ... < logfile This tool runs "tail" on given vnlog files in various ways. "vnl-tail" is a wrapper around the GNU coreutils "tail" tool. Since this is a wrapper, most commandline options and behaviors of the "tail" tool are present; consult the tail(1) manpage for detail. The differences from GNU coreutils "tail" are * The input and output to this tool are vnlog files, complete with a legend * "-c" is not supported because vnlog really doesn't want to break up lines * "--zero-terminated" is not supported because vnlog assumes newline-separated records Past that, everything "tail" does is supported, so see that man page for detailed documentation. BUGS This and the other "vnl-xxx" tools that wrap coreutils are written specifically to work with the Linux kernel and the GNU coreutils. None of these have been tested with BSD tools or with non-Linux kernels, and I'm sure things don't just work. It's probably not too effortful to get that running, but somebody needs to at least bug me for that. Or better yet, send me nice patches :) SEE ALSO tail(1)
NAME vnl-ts - add a timestamp to a vnlog stream SYNOPSIS $ read_temperature # temperature 29.5 30.4 28.3 22.1 ... continually produces data at 1Hz $ read_temperature | vnl-ts -s %.s # time-rel temperature 0.013893 30.2 1.048695 28.6 2.105592 29.3 3.162873 22.0 ... DESCRIPTION Usage: vnl-ts [-i | -s] [-m] [--vnl-field t] format < pipe This tool runs "ts" on given vnlog streams. "vnl-ts" is a wrapper around the "ts" tool from Joey Hess's moreutils <https://joeyh.name/code/moreutils/> toolkit. Since this is a wrapper, most commandline options and behaviors of the "ts" tool are present; consult the ts(1) manpage for details. The differences from "ts" are * The input and output to this tool are vnlog files, complete with a legend * The format *must* be passed-in by the user; no default is assumed. * The given format *must not* contain whitespace, so that it fits a single vnlog field. * "-r" is not supported: it assumes input timestamps with whitespace, which is incompatible with vnlog * A "vnl-ts"-specific option "--vnl-field" is available to set the name of the new field. If omitted, a reasonable default will be used. Past that, everything "ts" does is supported, so see that man page for detailed documentation. BUGS This and the other "vnl-xxx" tools that wrap standard utilities are written specifically to work with the Linux kernel. None of these have been tested with BSD tools or with non-Linux kernels, and I'm sure things don't just work. It's probably not too effortful to get that running, but somebody needs to at least bug me for that. Or better yet, send me nice patches :) SEE ALSO ts(1)
https://github.com/dkogan/vnlog/
Dima Kogan ([email protected]
) wrote this toolkit for his work at the Jet
Propulsion Laboratory, and is delighted to have been able to release it
publically
Chris Venter ([email protected]
) wrote the base64 encoder
This library is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 2.1 of the License, or (at your option) any later version.
Copyright 2016-2017 California Institute of Technology
Copyright 2017-2018 Dima Kogan ([email protected]
)
b64_cencode.c
comes from cencode.c
in the libb64
project. It is written by
Chris Venter ([email protected]
) who placed it in the public domain. The
full text of the license is in that file.