Nexus is an interactive tool designed for versatile data analysis, leveraging the SPL (Search Processing Language) for expressing data manipulation tasks. Released under the GNU General Public License (GPL), Nexus offers the freedom to utilize, modify, and distribute its code.
Utilizing SPL, Nexus excels at parsing unformatted data, employing a syntax reminiscent of Unix pipelines with the pipe symbol (|). Each command within the pipeline encapsulates a specific analysis task.
The SPL syntax follows this pattern:
| <spl commands> = | <spl command> | <spl command> | ...
- Fast import
- Parsing data in arbitrary formats
- Extending data parsing capabilities with Lua plugins
- Data analysis through SPL
- Interactive user interface
Nexus takes all imported data and streams it in the form of byte streams to the extract command. The extract command parses the byte stream and transforms it into a table format, resembling a database format, based on specific processing logic. The resulting table is then passed on to the next command. Ultimately, after all commands have been executed, the results are displayed on the screen.
In addition to the extract command, Nexus includes other commands such as dedup, where, eval, sort, limit, stats, etc. These commands are designed to manipulate the data. It is important to note that, excluding the extract command, the input and output for these commands are both in table format.
make
# This command initiates an interactive interface to execute commands for data analysis
./nexus
Nexus responds to several interactive commands:
- quit/exit: Exit Nexus
- clear: Clear the screen
- Arrow keys: Navigate command history
- Supported SPL statements
File Path in nexus all use absolute paths
Nexus leverages the import statement to import data, following this syntax:
IMPORT name [, name]...
Here are two examples:
| import "file1"; -- Import file1
| import "file1", "file2"; -- Import file1 and file2
Nexus uses the clean statement to clear data, employing the following syntax:
CLEAN
Here is an example:
| clean
Data parsing in Nexus utilizes the extract statement, serving as the initial command in a query. The syntax for extract is as follows:
EXTRACT [LUA = STRING | LUA_FILE = STRING] eval_list
Consider this example:
| extract LUA_FILE="1.lua" a = 0, b = 1, c = 2 | eval a = cast(a as float) + cast(b as float) | sort 10 by a;
The extract command incorporates Lua to implement the basic logic of data parsing. Details on writing Lua scripts for various functionalities will be explained in the Lua script section.
Data deduplication in Nexus is achieved through the dedup statement, utilizing the following syntax:
dedup name [, name]
Consider these examples:
... | dedup a
... | dedup a, b
Nexus filters data using the where statement, adhering to the syntax:
where expr
Consider these simple examples:
... | where a = 1
... | where like(ip, "198.*")
Nexus projects data through the eval statement, following the syntax:
eval name = expr [, name = expr]
Consider these simple examples:
... | eval a = b + c
... | eval a = cast(b as float)
Data sorting in Nexus is achieved through the sort statement, with the syntax:
sort [int] sort-field-list
sort-field-list = name [desc|asc] [, name [desc|asc]]
Consider these simple examples:
... | sort by uid
... | sort 10 by uid, date
Limiting data in Nexus is accomplished using the limit statement, following this syntax:
limit int
Consider this example:
... | limit 10
Nexus employs the stats statement for grouping and aggregating data, with the syntax:
stats expr [as name] [, expr [as name]] [by name [, name]]
Currently supported aggregation functions for stats include:
- count
- sum
- max
- min
- avg
Consider these examples:
... | stats count()
... | stats count() as cnt by ip
... | stats sum(a), avg(a) by b, c
Functions and Operators Nexus currently supports various functions and operators:
- isnull(x): Checks if x is nil.
- isnotnull(x): Checks if x is not nil.
- cast(x as t): Converts x to type t.
- replace(x, old, new): String replacement.
- regexp_match(x, reg): Checks if x matches the regular expression reg.
- regexp_extract(x, reg, idx): Extracts data that satisfies the conditions.
- Arithmetic operators: +, -, *, /, <, <=, >=, >, =, <>, %
- Logical operators: and, or, not
Nexus supports the following data types:
- bool
- long
- double
- string
- NULL - Represents non-existence
The supported constants in Nexus are as follows:
- true, false
- 1
- 1.2
- "x", `x`
Nexus supports two types of strings. The first type is similar to "x", where the content within double quotation marks is considered the string content. It is important to note that this representation automatically handles escape characters, for example, "\n" will be treated as a newline character. The second type is `x`-style strings, where the content within backticks is considered the string content and no escape processing is applied.
Nexus utilizes a Lua script to convert byte stream data into table data. The Lua script employs io.read() to continuously read data, parsing it into table data and sending data chunks to Nexus.
The Lua script outputs a table with n rows * n columns, representing a two-dimensional array using Lua's table data structure. The specific format of the two-dimensional array is exemplified as follows:
tbl = { {"x", "y", "z"}, {"a", "b", "c"}}
The processing methodology involves using a column-by-column, one-dimensional array. Each output is transmitted to Nexus through writeResult. It is important to note that the rows in each column need to be aligned, and the number of rows must not exceed 8192, necessitating the script to output data in manageable chun
Below are several examples of Lua scripts. The first script separates the input byte stream into lines, with each line containing only one column of data:
local row = 0
local tbl = {{}}
tbl[1] = tbl[1] or {}
for line in io.stdin:lines() do
local rows = #tbl[1] or 0
if rows >= 8192 then
writeResult(tbl)
row = 0
tbl = {{}}
tbl[1] = tbl[1] or {}
end
tbl[1][row+1] = line
row = row + 1
end
local rows = #tbl[1] or 0
if rows > 0 then
writeResult(tbl)
end
writeResult(nil)
This is a simple example, and all Lua scripts are similar. They read data through io.stdin, build a two-dimensional array, and call writeResult to return results in batches, using writeResult(nil) at the end.
The next example is a more complex one designed to parse a simple CSV format. The Lua script is as follows:
-- csv.lua - parse and format csv
-- Usage: cat csv.txt | lua csv.lua
-- Input: csv.txt
-- Output: {field1, field2, field3..},{field1, field2, field3...}...
-- Author: nnsgmsone
function parseCSVLine(line)
local fields = {}
for v in string.gmatch(line, '([^,]+)') do
table.insert(fields, v)
end
return fields
end
local row = 0
local tbl = {{}}
for line in io.stdin:lines() do
local record = parseCSVLine(line)
local rows = #tbl[1] or 0
if rows >= 8192 then
writeResult(tbl)
row = 0
tbl = {{}}
end
for i, v in ipairs(record) do
tbl[i] = tbl[i] or {}
tbl[i][row+1] = v
end
row = row + 1
end
local rows = #tbl[1] or 0
if rows > 0 then
writeResult(tbl)
end
writeResult(nil)
The difference from the first Lua script is that this script outputs multiple columns. It parses CSV, divides the CSV data into units of 8192 rows, and then returns the data in table format (column storage).
The last example is a more practical Lua script that parses dump of goroutines, converting each goroutine into three columns (creator information, call stack information, goroutine survival time). The Lua script is as follows:
-- goroutine.lua - parse and format goroutine dump
-- Usage: cat goroutine.txt | lua goroutine.lua
-- Input: goroutine.txt
-- Output: {created, stack, time},{created, stack, time}...
-- Author: nnsgmsone
local row = 0
local tbl = {{}}
local time = "0"
local stack = ""
local created = ""
tbl[1] = tbl[1] or {}
tbl[2] = tbl[2] or {}
tbl[3] = tbl[3] or {}
for line in io.stdin:lines() do
local rows = #tbl[1] or 0
if rows >= 8192 then
writeResult(tbl)
row = 0
tbl = {{}}
tbl[1] = tbl[1] or {}
tbl[2] = tbl[2] or {}
tbl[3] = tbl[3] or {}
end
if string.len(stack) == 0 then
local find = string.match(line, "goroutine")
if find then
stack = line
local t = string.match(line, "(%d+)%s*minutes")
if t then
time = t
end
end
elseif string.len(created) > 0 then
local find = string.match(line, "%S+")
if find then
tbl[1][row+1] = find
else
tbl[1][row+1] = ""
end
tbl[2][row+1] = stack .. "\n" .. line
tbl[3][row+1] = time
created = ""
stack = ""
time = "0"
row = row + 1
else
local find = string.match(line, "created by")
if find then
created = find
else
stack = stack .. "\n" .. line
end
end
end
writeResult(tbl)
writeResult(nil)
These Lua scripts provide examples of reading and processing data in different formats.
This project is licensed under the GPL License.