Comments (6)
Hello,
// Q_Vlan
uint32 Dot1q_Vlan = 243;
uint32 Post_Dot1q_Vlan = 254;
uint32 Dot1q_Cvlan = 245;
uint32 Post_Dot1q_Cvlan = 255;
results in the following Go structures
Dot1Q_Vlan uint32 `protobuf:"varint,243,opt,name=Dot1q_Vlan,json=Dot1qVlan,proto3" json:"Dot1q_Vlan,omitempty"`
Post_Dot1Q_Vlan uint32 `protobuf:"varint,254,opt,name=Post_Dot1q_Vlan,json=PostDot1qVlan,proto3" json:"Post_Dot1q_Vlan,omitempty"`
Dot1Q_Cvlan uint32 `protobuf:"varint,245,opt,name=Dot1q_Cvlan,json=Dot1qCvlan,proto3" json:"Dot1q_Cvlan,omitempty"`
Post_Dot1Q_Cvlan uint32 `protobuf:"varint,255,opt,name=Post_Dot1q_Cvlan,json=PostDot1qCvlan,proto3" json:"Post_Dot1q_Cvlan,omitempty"`
It might be a good enhancement to use a mapping based on the name as well (semi related to #110), will think about it.
The following should work
mapping:
- field: 252
destination: InIf
- field: 253
destination: OutIf
- field: 243
- destination: Dot1q_Vlan
+ destination: Dot1Q_Vlan
- field: 254
- destination: Post_Dot1q_Vlan
+ destination: Post_Dot1Q_Vlan
- field: 245
- destination: Dot1q_Cvlan
+ destination: Dot1Q_Cvlan
- field: 255
- destination: Post_Dot1q_Cvlan
+ destination: Post_Dot1Q_Cvlan
from goflow2.
You need to make sure ClickhouseDB also has the correct proto if you are doing direct insertion.
from goflow2.
Hi Louis,
thanks a lot.. I tested it.. but without success, values are still zero.. That's my actual configuration:
flow.proto:
// Q_Vlan
uint32 Dot1Q_Vlan = 243;
uint32 Post_Dot1Q_Vlan = 254;
uint32 Dot1Q_Cvlan = 245;
uint32 Post_Dot1Q_Cvlan = 255;
mapping.yaml:
ipfix:
mapping:
- field: 252
destination: InIf
- field: 253
destination: OutIf
- field: 243
destination: Dot1Q_Vlan
- field: 254
destination: Post_Dot1Q_Vlan
- field: 245
destination: Dot1Q_Cvlan
- field: 255
destination: Post_Dot1Q_Cvlan
Clickhouse Table:
CREATE TABLE IF NOT EXISTS sisr_dev.kafka_goflow2_ipfix_proto_consumer ON CLUSTER "sisr-dev-ch-cluster"
(
SrcMac UInt64,
DstMac UInt64,
TCPFlags UInt32,
TimeReceived UInt64,
TimeFlowStartMs UInt64,
TimeFlowEndMs UInt64,
SamplerAddress FixedString(16),
SrcAddr FixedString(16),
DstAddr FixedString(16),
InIf UInt32,
OutIf UInt32,
IngressVrfID UInt32,
Dot1Q_Vlan UInt32,
Post_Dot1Q_Vlan UInt32,
Dot1Q_Cvlan UInt32,
Post_Dot1Q_Cvlan UInt32,
EType UInt32,
Proto UInt32,
SrcPort UInt32,
DstPort UInt32,
Bytes UInt64,
Packets UInt64
) ENGINE = Kafka()
SETTINGS
kafka_broker_list = 'BROKERS,
kafka_topic_list = 'goflow2_raw_ipfix_proto',
kafka_group_name = 'goflow2_proto_raw_ch_kafka',
kafka_num_consumers = 8,
kafka_thread_per_consumer = 4,
kafka_format = 'Protobuf',
kafka_schema = 'flow.proto:FlowMessage';
Do maybe have another idea ? On the other hand it is hard to debug, because the only point I can read the protobuf at the moment is in the Clickhouse DB. I can't verify if its broken at another point.
Is there a possibility to add that fields to json as well ?
I tried to start a container with the mapping above and json output but I can't find the fields in the output..
I think, you are right, mapping on the name could be very nice 👍
Thanks a lot for your help.. very appreciated :)
from goflow2.
Just to make sure: did you recompile the proto after changing it? Using make proto
.
To visualize it as JSON, until my next PR, you need to edit the fields mapped in the code (should be in the format folder).
from goflow2.
Great :).. you got it..I'm getting the values for dot1q and so on. Sorry for that dumb mistake on my side :/..
I did only a docker-compose build and forgot the "make proto"
I had a look to the changes in JSON.. but I didn't make it because I wasn't totally sure were and my go skills are limited...
I think make that easier is a good step 👍
Thanks a lot for your help :)
from goflow2.
Fixed ... :)
from goflow2.
Related Issues (20)
- Question about performance HOT 3
- ObservationDomainId is not included in NetflowV9
- level=fatal msg="Format bin not found" HOT 2
- Metrics count error for `flow_process_nf_total`
- Reception of multicast messages does not work HOT 1
- Not able to write output into KAFKA HOT 3
- Missing sflow samples in the output HOT 3
- Bad values time_flow_start_ns, time_flow_end_ns when sending Newflow v5 to collector HOT 1
- Bad values time_flow_start_ns, time_flow_end_ns when sending Newflow v9 and exporting them to Kafka HOT 4
- Support for Cisco ASA/NSEL HOT 4
- Clickhouse Compose example results in corrupt data HOT 5
- Netflow v9 templateId error HOT 1
- Discussion - Using this project as a basis for an Otel Receiver HOT 2
- NATS integration HOT 1
- Sign releases HOT 3
- time_flow_start_ns and time_flow_end_ns is incorect value HOT 7
- how scale goflow2 for high throughput HOT 5
- How to identify traffic direction in sFlow information? HOT 8
- Problem in decoding bin format HOT 4
- Add support of ExtendedMPLS for sflow HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from goflow2.