monogon-dev / netmeta Goto Github PK

Contributions guide and dev setup.

For, say, NetFlow data it's pointless to group by a 1s interval

Just load the entire Geolite DB as an IP Trie and look at at query time.

Grafana can display RSS feeds, use it to display news about NetMeta.

-> embed https://github.com/bio-routing/bio-rd

Slightly tricky - might be hard to do via dictionary lookup, better to have a separate Kafka topic for postprocessing.

• Insert template variable autorefresh attribute that Grafana refuses to save to the JSON.
• Remove rawQuery

https://clickhouse.tech/docs/en/operations/settings/query-complexity/

https://github.com/Altinity/clickhouse-operator/blob/master/docs/quick_start.md

Prevent server from eating itself.

Add an extra annotation to the interface configs to group interfaces ("show transit ports only").

Plus an embedded node_exporter for the host itself.

Should be a LowCardinality(String) field to account for weird customers.

Viable replacements:

• microk8s
• kind
• k8os?

• Optimize for main dasboard queries.
• Low-cardinality values first for easy filtering (verify that this does what we think it does).
• Sequence Num for sampling, but how? It's as high-cardinality as it gets.
• Play with secondary idexes - are they more efficient than a carefully-considered sorting key?

We can probably pin almost every binary dependency we pull in via digests.

(is there a way to verify k3s binaries and containers?)

This does not work due to missing features in k3s/cri.

containerd/cri#1246

k3s-io/k3s#1821

k3s-io/k3s#1714

k3s-io/k3s#533

microk8s isn't any better.

This allows for declarative parameter changes in the engine spec without migrations.

The current templating engine becomes ....unwieldy:

SELECT
    $timeSeries as t,
    SrcAS,
    sum(Bytes * SamplingRate) * 8 / $interval AS Bps
FROM $table
WHERE
    $timeFilter AND FlowDirection = 0
    $conditionalTest(AND SamplerAddress = toIPv6($sampler), $sampler)
    $conditionalTest(AND SrcAddr = toIPv6('$srcIP'), $srcIP)
    $conditionalTest(AND DstAddr = toIPv6('$dstIP'), $dstIP)
    $conditionalTest(AND (SrcAddr = toIPv6('$hostIP') OR DstAddr = toIPv6('$hostIP')), $hostIP)
    $conditionalTest(AND NextHop = toIPv6('$nextHop'), $nextHop)
    $conditionalTest(AND (InIf = $interface OR OutIf = $interface), $interface)
    AND SrcAS IN (
    SELECT SrcAS
    FROM $table
    WHERE $timeFilter AND FlowDirection = 0 AND $adhoc
      $conditionalTest(AND SamplerAddress = toIPv6($sampler), $sampler)
      $conditionalTest(AND SrcAddr = toIPv6('$srcIP'), $srcIP)
      $conditionalTest(AND DstAddr = toIPv6('$dstIP'), $dstIP)
      $conditionalTest(AND (SrcAddr = toIPv6('$hostIP') OR DstAddr = toIPv6('$hostIP')), $hostIP)
      $conditionalTest(AND NextHop = toIPv6('$nextHop'), $nextHop)
      $conditionalTest(AND (InIf = $interface OR OutIf = $interface), $interface)
      $conditionalTest(AND ($extra), $extra)
    GROUP BY SrcAS
    ORDER BY count(*) DESC
    LIMIT 10)
    $conditionalTest(AND ($extra), $extra)
GROUP BY
    t,
    SrcAS
ORDER BY t, Bps

We need to fork https://github.com/Vertamedia/clickhouse-grafana and add custom templates for this.

We don't need 1-second high resolution data for long-term archival.

Use an AggreatingMergeTree, aggregate it to 5m or so and possibly keep min/max/median.

Render this waterfall:

Without melting the browser.

Depends On: #44

Once we have the AS paths, it's trivial to build a Sankey diagram for egress traffic. Might require a server-side component to feed something like this: https://github.com/kumaravel29/grafana-sankey-panel

Explain if and how to add larger customizations