telekom / 5g-trace-visualizer Goto Github PK

Hi,

Can we use dynamic/dhcp IPs instead of fixed IP for interface IPs of servers.yaml ? In some implementations, IP allocating to a particular interface have configured using dhcp IP pool.

Thanks,
Luke.

It would be useful to be able to highlight certain messages in long diagrams

This code dont working jn Linux, because there isnt Wireshark Portable for Linux and OS path thshark is incorrect

I fix this problem by adding follow code in call_wireshark_for_one_version():

    # Add option to not use a Wireshark portable version but rather the OS-installed one
    if wireshark_version == 'OS':
        if (platform == 'Linux'):
            tshark_path = "/usr/bin/tshark"
        else:
            tshark_path = os.path.join('tshark')
    else:
        tshark_path = os.path.join(get_wireshark_portable_folder(wireshark_version), 'tshark')
    logging.debug('tshark path: {0}'.format(tshark_path))

I can create pull request for this

The current regex for multipart boundary (and content-type, content-id) does not consider '_' char while it can occur:

--oiH7UJCrm4vEWMX70dIsiqFSAYBdme6zn39WMZftu_1_M9D80Ajho1cTLSIkMVSC

mime_multipart_payload_regex should be extended by adding '_' for the boundary.

There are cases when a 5GC NF uses SCTP (for example AMF towards gNodeB), which means that one packet could contain more than one SCTP DATA chunk. These chunks contain NGAP messages.

The import_pdml function says:

    # For 5GC
    # Fix case (see free5GC trace) where there are several elements (not correct but well...)
    ngap_proto = packet.findall("proto[@name='ngap']")
    if len(ngap_proto) == 0:
        ngap_proto = None
    elif len(ngap_proto) == 1:
        ngap_proto = ngap_proto[0]

So this part (and perhaps more) would require an update to consider the fact of multiple SCTP chunks (meaning multiple NGAP messages) in one packet, which is entirely correct protocol behaviour.

Is it possible to add feature to label host label according to docker-compose's yaml file?
The parameter might similar to k8s pod data I think?

The compose file may similar like this

https://github.com/free5gc/free5gc-compose/blob/master/docker-compose.yaml

Often one has only interest in the "namespace" diagrams. Generating all of the other ones takes some time. It would be best if one coud configure what diagrams to output

In case there is IP traffic sent over another IP layer, the second one is more useful as src and dst.
I found I can fix this by replacing the following:

         ip_showname = packet.find("proto[@name='ip']").attrib['showname']

with this:

         ip_showname = packet.findall("proto[@name='ip']")[-1].attrib['showname']

Sometimes, the lines alone would be enough. It should be possible to omit the (sometimes very verbose) payloads

I am trying to run the 5g-trace-visualizer:
I have fulfilled all the requirements as stated however whenever I run the following command

python trace_visualizer.py -wireshark 3.2.2 "./doc/free5gc.pcap"

it throws this error

any help would be appreciated !

When parameters -simple_diagrams is set we don't see informatical diameter packet description:

It will be better to extend diameter packet description.
For example:

trace visualizer crashed when tried to generate svg file out of pcap

python3 trace_visualizer.py -wireshark "OS" -http2ports "1000-65535" ../../Documents/open5gs-lo.pcap

DEBUG:root:906: Internet Protocol Version 4, Src: 127.0.0.1, Dst: 127.0.0.5 Traceback (most recent call last): File "/Users/ni.bhat/Desktop/5g-trace-visualizer/trace_visualizer.py", line 1951, in <module> puml_files = import_pdml( File "/Users/ni.bhat/Desktop/5g-trace-visualizer/trace_visualizer.py", line 1387, in import_pdml nas_request = parse_nas_proto(frame_number, ngap_proto) File "/Users/ni.bhat/Desktop/5g-trace-visualizer/trace_visualizer.py", line 210, in parse_nas_proto return parse_nas_proto_el(frame_number, el, multipart_proto) File "/Users/ni.bhat/Desktop/5g-trace-visualizer/trace_visualizer.py", line 228, in parse_nas_proto_el nas_5g_dict = nas_5g_proto_to_dict(nas_5g_proto) File "/Users/ni.bhat/Desktop/5g-trace-visualizer/trace_visualizer.py", line 175, in nas_5g_proto_to_dict return xml2json(nas_5g_proto) File "/Users/ni.bhat/Desktop/5g-trace-visualizer/trace_visualizer.py", line 168, in xml2json parsed_tree = recursiv(root) File "/Users/ni.bhat/Desktop/5g-trace-visualizer/trace_visualizer.py", line 135, in recursiv data_to_append = recursiv(child_to_traverse) File "/Users/ni.bhat/Desktop/5g-trace-visualizer/trace_visualizer.py", line 135, in recursiv data_to_append = recursiv(child_to_traverse) File "/Users/ni.bhat/Desktop/5g-trace-visualizer/trace_visualizer.py", line 135, in recursiv data_to_append = recursiv(child_to_traverse) [Previous line repeated 13 more times] File "/Users/ni.bhat/Desktop/5g-trace-visualizer/trace_visualizer.py", line 127, in recursiv logging.DEBUG('Found repeated child element {0}. Renamed to {1}'.format(original_child_name, child_name)) TypeError: 'int' object is not callable

The same problem as mentioned in #18
Need to fix path for mergecap in Linux system

When a Wireshark trace is exported with name resolution enabled, the line with the IP address src and dst looks like this:

This is not detected by the regex. Workaround is to turn off Name resolution before PDML export but should be considered in the script.

It is a valid case when the NAS "5GS tracking area identity list" contains a list of TAC values and not just a single TAC value.

The xml2json function is not prepared for that, it will overwrite the TAC items and keeps the last one.

Wireshark screenshot:

PDML lines:

<field name="nas_5gs.tac" showname="TAC: 134" size="3" pos="53" show="134" value="000086"/>
<field name="nas_5gs.tac" showname="TAC: 118" size="3" pos="56" show="118" value="000076"/>

Source code:

child_name = child.attrib["name"]
...
out[child_name].append(data_to_append)

PUML "note":

        5GS tracking area identity list:
            nas_5gs.mm.elem_id: 'Element ID: 0x54'
            gsm_a.len: 'Length: 10'
            Partial tracking area list  1:
                nas_5gs.mm.tal_t_li: '.00. .... = Type of list: list of TACs belonging to one PLMN or SNPN, with non-consecutive TAC values (0)'
                nas_5gs.mm.tal_num_e: '...0 0001 = Number of elements: 2 elements (1)'
                e212.5gstai.mcc: 'Mobile Country Code (MCC): Sweden (240)'
                e212.5gstai.mnc: 'Mobile Network Code (MNC): Unknown (80)'
                nas_5gs.tac: 'TAC: 118'

Please consider updating the xml2json function for this.

hi, thanks for sharing this excellent tools. do you have plan package this tool into docker images? its a little hard to prepare the environment without knowledge of java. thanks again

Meybe it will be better to add port numbers to servers.yaml and not set its each time in command line, when we launch 5g-trace-vizualizer???

Hi ,

I tried to start with the sample command you have given in the documentation.
python3 trace_visualizer.py -wireshark "OS" ./doc/free5gc.pcap

But I got the following message:

DEBUG:root:Preparing call for Wireshark version OS
DEBUG:root:tshark path: tshark
Traceback (most recent call last):
File "trace_visualizer.py", line 1695, in
input_file = call_wireshark(args.wireshark, input_file, args.http2ports)
File "trace_visualizer.py", line 1457, in call_wireshark
output_file = call_wireshark_for_one_version(wireshark_version, input_file_str, http2ports_string)
File "trace_visualizer.py", line 1504, in call_wireshark_for_one_version
raise FileNotFoundError('Could not find tshark on path {0}'.format(tshark_path))
FileNotFoundError: Could not find tshark on path tshark

According to the message, it says that "FileNotFoundError: Could not find tshark on path tshark". But I already have installed tshark , and wirshark.

$ which tshark
/usr/bin/tshark

$ which wireshark
/usr/bin/wireshark

any suggestion please? Can I know what I have done wrong here ?
Also , can I check with you, can we use this script to draw SVG sequence diagrams using 5g http2 traces only for a specific IMSI?

Thanks
Luke.

Hello, I am a student at Kocaeli University. I am looking for open source software stack for 3G like OpenAirInterface for 5G or Osmocom for 2G for my work. (Osmocom has partial support for 3G)

I request you to direct me to the right resource that may know.

Thanks for your time and interest.

Kind regards,

I am looking for open source software stack for 3G, like OpenAirInterface for 5G or Osmocom for 2G. (Osmocom has partial support for 3G)

Kindly asking you to forward me to the right source who may know.

Thank you for your time and kind attention.

Best Regards,

Code have options -showheartbeat, but it dont work with diameter proto, because you use parse_gtpv2_proto() for parsing diameter_proto

I think it will be great, if this project can skip Diameter Common Application messages, when -showheartbeat is set.

How I create parse_diam_proto() function to parse diameter.
If you want I can create pull request

For wireshark, it provide an option to decode TLS on HTTP2 if we provide the key file.
https://github.com/free5gc/free5gc/wiki/Trouble_Shooting#6-decode-http2-packet-in-wireshark

I'm wondering if 5g-trace-visualizer is consider to support decoding encrypting SBI maybe using this feature in wireshark.

Hi,

the Readme.md states "Add in this folder the Wireshark portable installations so that they can be automatically called by the program." - am I right, assuming that this project is designed for WIN only?
Is there any chance to invoke an installed version of Wireshark on LX ?
I do have it included in the path.

(base) [tpa@devel 5g-trace-visualizer]$ tshark --v
TShark (Wireshark) 3.2.5 (Git commit ed20ddea8138)

(base) [tpa@devel 5g-trace-visualizer]$ wireshark --v
Wireshark 3.2.5 (Git commit ed20ddea8138)

Any hint welcomed. If I just try:

python trace_visualizer.py -wireshark "3.2.5" /.../5G-Attach.pcapng

I run into following FileNotFoundError, as the search seems to be done in ".../wireshark/WiresharkPortable_3.2.5/App/Wireshark/tshark'" subdir only (that does not exist in my case):

(base) [tpa@devel 5g-trace-visualizer]$ python trace_visualizer.py -wireshark "3.2.5" /home/tpa/__SNDBOX/5GC_Traces/5G-Attach.pcapng/5G-Attach.pcapng
Searching for plantuml.jar under /home/tpa/__SNDBOX/5g-trace-visualizer/plantuml.jar
Found plantuml.jar

Platform system detected: Linux
5G Wireshark trace visualizer for 5G protocol
Input file: /home/tpa/__SNDBOX/5GC_Traces/5G-Attach.pcapng/5G-Attach.pcapng
Pods YAML file: None
Maximum messages per file: 100
Debug: None
String un-escaping for HTTP requests: True
OpenStack servers file: None
HTTP/2 headers to ignore: None
Diagrams to output: ip,k8s_pod,k8s_namespace
Simple diagrams: False
Force show frames if using simple diagrams:
Show PFCP/GTPv2 heartbeat messages: False
Show HTTP/2 in spurious TCP retransmission messages: True
Ignore PFCP packet duplicates: True
Show timestamp in diagram: False
Show self-messages: False

Using Wireshark version 3.2.5
Wireshark supports nas-5gs.null_decipher option. Applying
Generating PDML files from PCAP to /home/tpa/__SNDBOX/5GC_Traces/5G-Attach.pcapng/5G-Attach_3.2.5.pdml
['/home/tpa/__SNDBOX/5g-trace-visualizer/wireshark/WiresharkPortable_3.2.5/App/Wireshark/tshark', '-r', '/home/tpa/__SNDBOX/5GC_Traces/5G-Attach.pcapng/5G-Attach.pcapng', '-2', '-d', 'tcp.port==32445,http2', '-d', 'tcp.port==5002,http2', '-d', 'tcp.port==5000,http2', '-d', 'tcp.port==32665,http2', '-d', 'tcp.port==80,http2', '-d', 'tcp.port==32077,http2', '-d', 'tcp.port==5006,http2', '-d', 'tcp.port==8080,http2', '-d', 'tcp.port==3000,http2', '-o', 'nas-5gs.null_decipher: TRUE', '-Y', '(http2 and (http2.type == 0 || http2.type == 1)) or ngap or nas-5gs or pfcp or gtpv2 or diameter or radius or gtpprime or icmp', '-T', 'pdml', '-J', 'http2 ngap pfcp gtpv2 tcp diameter radius gtpprime icmp', '-n']
Traceback (most recent call last):
File "trace_visualizer.py", line 1313, in
input_file = call_wireshark(args.wireshark, input_file, args.http2ports)
File "trace_visualizer.py", line 1137, in call_wireshark
output_file = call_wireshark_for_one_version(wireshark_version, input_file_str, http2ports_string)
File "trace_visualizer.py", line 1221, in call_wireshark_for_one_version
subprocess.run(tshark_command, stdout=outfile)
File "/home/tpa/anaconda3/lib/python3.8/subprocess.py", line 489, in run
with Popen(*popenargs, **kwargs) as process:
File "/home/tpa/anaconda3/lib/python3.8/subprocess.py", line 854, in init
self._execute_child(args, executable, preexec_fn, close_fds,
File "/home/tpa/anaconda3/lib/python3.8/subprocess.py", line 1702, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/home/tpa/__SNDBOX/5g-trace-visualizer/wireshark/WiresharkPortable_3.2.5/App/Wireshark/tshark'

Thx.

In case a HTTP/2 message has "HEADERS" only and no "DATA", the SVG is not correct since there is a missing newline char in the uml file.

For example:
"SMF" -> "AMF2": 91. HTTP/2 204 rsp.
note right of "SMF" #e6e6e6
SMF to AMF2
10.206.108.92 to 10.206.108.67 (IPs)
HTTP/2 stream: 37
:status: 204end note

In front of "end note" the newline is missing.

One possible fix is to modify parse_http_proto_stream() and to add one more IF:

if (data_ascii == '') and (http2_request != ''):
     http2_request = http2_request + '\n'
 if (data_ascii != '') and (http2_request != ''):
     http2_request = http2_request + '\n\n'

thanks for your nice project

when I try to use it,something error :

python3 trace_visualizer.py -wireshark "OS" ./doc/free5gc.pcap
DEBUG:root:Searching for plantuml.jar under 5g-trace-visualizer/plantuml.jar
DEBUG:root:Found plantuml.jar

DEBUG:root:Platform system detected: Linux
5G Wireshark trace visualizer for 5G protocol
Input file: ./doc/free5gc.pcap
Pods YAML file: None
Maximum messages per file: 100
Debug: None
String un-escaping for HTTP requests: True
OpenStack servers file: None
HTTP/2 headers to ignore: None
Diagrams to output: ip,k8s_pod,k8s_namespace
Simple diagrams: False
Force show frames if using simple diagrams:
Show PFCP/GTPv2 heartbeat messages: False
Show HTTP/2 in spurious TCP retransmission messages: True
Ignore PFCP packet duplicates: True
Show timestamp in diagram: False
Show self-messages: False

DEBUG:root:Preparing call for Wireshark version OS
DEBUG:root:Wireshark call for ./doc/free5gc.pcap. Version OS, HTTP/2 ports: 32445,5002,5000,32665,80,32077,5006,8080,3000
DEBUG:root:tshark path: /usr/bin/tshark
DEBUG:root:Received HTTP/2 port list: 32445,5002,5000,32665,80,32077,5006,8080,3000
DEBUG:root:Using Wireshark version OS
DEBUG:root:Wireshark supports nas-5gs.null_decipher option. Applying
DEBUG:root:Generating PDML files from PCAP to ./doc/free5gc_OS.pdml
DEBUG:root:['/usr/bin/tshark', '-r', './doc/free5gc.pcap', '-2', '-d', 'tcp.port==32445,http2', '-d', 'tcp.port==5002,http2', '-d', 'tcp.port==5000,http2', '-d', 'tcp.port==32665,http2', '-d', 'tcp.port==80,http2', '-d', 'tcp.port==32077,http2', '-d', 'tcp.port==5006,http2', '-d', 'tcp.port==8080,http2', '-d', 'tcp.port==3000,http2', '-o', 'nas-5gs.null_decipher: TRUE', '-Y', '(http2 and (http2.type == 0 || http2.type == 1)) or ngap or nas-5gs or pfcp or gtpv2 or diameter or radius or gtpprime or icmp or icmpv6.type == 134', '-T', 'pdml', '-J', 'http2 ngap pfcp gtpv2 tcp diameter radius gtpprime icmp icmpv6', '-n']
Running as user "root" and group "root". This could be dangerous.
tshark: -o flag "nas-5gs.null_decipher: TRUE" specifies unknown preference
DEBUG:root:Output ./doc/free5gc_OS.pdml
DEBUG:root:PDML file path(s): ['./doc/free5gc_OS.pdml']
DEBUG:root:Importing main file ./doc/free5gc_OS.pdml
DEBUG:root:Cleaning up PDML file ./doc/free5gc_OS.pdml
DEBUG:root:Finished cleaning up PDML file
DEBUG:root:Parsing PDML file ./doc/free5gc_OS.pdml
Traceback (most recent call last):
  File "trace_visualizer.py", line 1918, in <module>
    args.show_selfmessages)
  File "trace_visualizer.py", line 1177, in import_pdml
    tree = read_cleanup_and_load_pdml_file(file_path)
  File "trace_visualizer.py", line 1139, in read_cleanup_and_load_pdml_file
    parsed_et = ET.parse(file_path)
  File "/usr/lib/python3.6/xml/etree/ElementTree.py", line 1196, in parse
    tree.parse(source, parser)
  File "/usr/lib/python3.6/xml/etree/ElementTree.py", line 597, in parse
    self._root = parser._parse_whole(source)
xml.etree.ElementTree.ParseError: no element found: line 1, column 0

How to make it work fine? Thanks

I tried this on my linux but i realized that wireshark folder expect for a wireshark portable version with .exe binaries inside, which are used depending on the input (pldm, pcap).

Would it be difficult to migrate on linux environment ?
i mean, use tshark instead of tshark.exe, etc., etc.
I think that a great job is done on this python script and i wonder about this possibility because i suspect is not far away to achieve.

Thank U !

The 'platform' is not passed as a parameter thru the functions, it is basically defined globally.

Would not be better to pass it as a parameter for call_wireshark?

if __name__ == '__main__':
...
    platform = platform.system()
...
    if args.wireshark != 'none':
        input_file = call_wireshark(args.wireshark, input_file, args.http2ports)

How about:

input_file = call_wireshark(args.wireshark, **platform**, input_file, args.http2ports)
...
output_file = call_wireshark_for_one_version(wireshark_version, **platform**, input_file_str, http2ports_string, mode, check_if_exists)

Since in call_wireshark_for_one_version the platform is used few times and if the wireshark_version is properly received as an input parameter, the platform should be too.

E.g.:

        if wireshark_version == 'OS' and (platform == 'Linux'):
            mergecap_path  = "/usr/bin/mergecap"
        elif wireshark_version == 'OS' and (platform != 'Linux'):
            mergecap_path = 'mergecap'

If non-IP packets like ARP are included in the trace then looking for the IP src and dst fails. Such packets should be skipped. Workaround is to filter them out befor exporting but this could be addressed in the script

The current version does not support RADIUS packets properly.
The get_diam_description only looks for diameter which is not used in radius.

    elif 'Diameter' in protocol or 'RADIUS' in protocol or "GTP'" in protocol:
        note_color = ' {0}'.format(color_diameter_radius_gtpprime)
        protocol = get_diam_description(packet)

...

    diam_commandcode_regex = re.compile(r"diameter\.cmd\.code:\s+'Command\s+Code:\s+(.+)'")
    diam_application_regex = re.compile(r"diameter\.applicationId:\s+'ApplicationId:\s+(.+)'")
    diam_request_regex = re.compile(r"diameter\.flags\.request:\s+'(\d)")
    diam_session_regex = re.compile(r"diameter\.Session-Id:\s+'Session-Id:\s+(.+)'")

Error:

  File "trace_visualizer.py", line xxx, in packet_to_str
    protocol = get_diam_description(packet)
  File "trace_visualizer.py", line xxx, in get_diam_description
    if diam_request_regex.search(packet.msg_description).group(1) == '1':
AttributeError: 'NoneType' object has no attribute 'group'

Here is a RADIUS Access-Request example:

RADIUS Protocol:
    radius.code: 'Code: Access-Request (1)'
    radius.id: 'Packet identifier: 0x24 (36)'
    radius.length: 'Length: 97'
    radius.authenticator: 'Authenticator: 1d87536306c9628344109e29f90ba40c'
    radius.req: 'Request: True'
    radius.rspframe: The response to this request is in frame 19
    Attribute Value Pairs:
        'AVP: t=User-Name(1) l=6 val=void':
            radius.avp.type: 'Type: 1'
            radius.avp.length: 'Length: 6'
            radius.User_Name: 'User-Name: void'
        'AVP: t=User-Password(2) l=18 val=Encrypted':
            radius.avp.type: 'Type: 2'
            radius.avp.length: 'Length: 18'
            radius.User_Password_encrypted: 'User-Password (encrypted): 35853d707a452aabb06e9ece6b3f8c67'
        'AVP: t=NAS-IP-Address(4) l=6 val=10.10.10.10':
            radius.avp.type: 'Type: 4'
            radius.avp.length: 'Length: 6'
            radius.NAS_IP_Address: 'NAS-IP-Address: 10.10.10.10'
        'AVP: t=NAS-Identifier(32) l=11 val=SMF123':
            radius.avp.type: 'Type: 32'
            radius.avp.length: 'Length: 11'
            radius.NAS_Identifier: 'NAS-Identifier: SMF123'
        'AVP: t=Called-Station-Id(30) l=18 val=internet.apn':
            radius.avp.type: 'Type: 30'
            radius.avp.length: 'Length: 18'
            radius.Called_Station_Id: 'Called-Station-Id: internet.apn'
        'AVP: t=Framed-Protocol(7) l=6 val=GPRS-PDP-Context(7)':
            radius.avp.type: 'Type: 7'
            radius.avp.length: 'Length: 6'
            radius.Framed_Protocol: 'Framed-Protocol: GPRS-PDP-Context (7)'
        'AVP: t=Service-Type(6) l=6 val=Framed(2)':
            radius.avp.type: 'Type: 6'
            radius.avp.length: 'Length: 6'
            radius.Service_Type: 'Service-Type: Framed (2)'
        'AVP: t=NAS-Port-Type(61) l=6 val=Virtual(5)':
            radius.avp.type: 'Type: 61'
            radius.avp.length: 'Length: 6'
            radius.NAS_Port_Type: 'NAS-Port-Type: Virtual (5)'