openairinterface / openair-spgwu-tiny Goto Github PK

I'm testing oai-spgwu-tiny with an SMF implementation from Open5GCore (commercial product).
I got the OAI UPF register with these environment variables:

    environment:
      DNN_0: 'net6'
      ENABLE_5G_FEATURES: 'yes'
      NETWORK_UE_IP: '192.168.0.0/16'
      NRF_IPV4_ADDRESS: '255.255.255.255'
      NSSAI_SD_0: '0xFFFFFF'
      NSSAI_SST_0: '1'
      PGW_INTERFACE_NAME_FOR_SGI: 'eth3'
      REGISTER_NRF: 'no'
      SGW_INTERFACE_NAME_FOR_S1U_S12_S4_UP: 'eth1'
      SGW_INTERFACE_NAME_FOR_SX: 'eth2'
      TZ: 'Etc/UTC'
      UPF_FQDN_5G: ''

The UPF registers with SMF successfully, but PDU session establishment does not work.
packet sample on N4 interface

The error message is:

[2023-08-01 21:33:14.494] [pfcp] [error] Unknown PFCP IE type 85 (length 5)
[2023-08-01 21:33:14.494] [spgwu_sx] [info] handle_receive exception PFCP msg 50 Bad Length hdr.length 249/ sum ie 223 / check sum ie 237 Exception /openair-spgwu-tiny/src/pfcp/3gpp_29.244.hpp:528

IE type 85 refers to PFCP_IE_CREATE_BAR.
I searched the code and it appears that the relevant decoding section is commented out:

Is there any particular reason for this?

UPF build:
imageName: oai-spgwu-tiny
imageFolder: rdefosseoai
imageVersionTag: 'v1.2.0'

To deactivate the DL, the SMF modifies an existing N4 session by sending PFCP Session Modification Request and modifies the FAR rule for DL to DROP the received packets and to notify the SMF about received DL data (for MT access / Paging). This works as expected for MT access during the first CM-IDLE mode of the UE:

Upon the Session Report sent by the UPF, the SMF updates the FAR rule to FORW and includes the Forwarding parameters:

The DL traffic flows as expected.

But during the second CM-IDLE phase, once the DL FAR of the same N4 session is again modified to DROP the received packets and to notify the SMF about received DL data (for MT access / Paging), there is no Session Report sent by the UPF to the SMF anymore.

Note, the UL and DL PDRs are using QER rule with both gates (UL/DL) set to OPEN for the duration of the session. My understanding is that the QER rule should not impact the deactivation (DROP/NOCP) and activation (FORW) of the DL traffic forwarding by the means of the FAR rule, i.e. the DL gate does not have to be CLOSED for the duration of the DL flow deactivation.

Network setup:

ubuntu@domainmgr1:~$ kubectl exec -it -n free-5gc -c debug f5gc-upf-85b94f7bdd-pkg6j -- sh
/ # ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
3: eth0@if1793: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1450 qdisc noqueue state UP
    link/ether 4a:52:bb:0b:c3:97 brd ff:ff:ff:ff:ff:ff
    inet 10.233.109.178/32 brd 10.233.109.178 scope global eth0
       valid_lft forever preferred_lft forever
4: eth1@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN
    link/ether fa:16:3e:88:88:bd brd ff:ff:ff:ff:ff:ff
    inet 10.10.20.202/24 brd 10.10.20.255 scope global eth1
       valid_lft forever preferred_lft forever
5: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP qlen 500
    link/[65534]
    inet 12.1.0.1/16 scope global tun0
       valid_lft forever preferred_lft forever
/ # ip rule list
0:      from all lookup local
32765:  from 10.10.20.202 lookup 100
32766:  from all lookup main
32767:  from all lookup default
/ # ip r list table 100
default via 10.10.20.1 dev eth1
10.10.20.0/24 dev eth1 scope link  src 10.10.20.202
/ # ip r
default via 169.254.1.1 dev eth0
12.1.0.0/16 dev tun0 scope link  src 12.1.0.1
169.254.1.1 dev eth0 scope link

There is nothing suspicious seen in the UPF trace during the processing of the last session modification request:

[2022-03-30T15:18:37.451179] [spgwu] [spgwu_sx ] [info ] TIME-OUT event timer id 149
[2022-03-30T15:18:37.453721] [spgwu] [spgwu_sx ] [info ] TIME-OUT event timer id 150
[2022-03-30T15:18:37.453797] [spgwu] [spgwu_sx ] [info ] PFCP HEARTBEAT PROCEDURE hash 3457419786 starting
[2022-03-30T15:18:37.455509] [spgwu] [spgwu_sx ] [info ] handle_receive(16 bytes)
[2022-03-30T15:18:37.455592] [spgwu] [spgwu_sx ] [info ] Received SX HEARTBEAT RESPONSE
[2022-03-30T15:18:38.795860] [spgwu] [spgwu_sx ] [info ] handle_receive(33 bytes)
[2022-03-30T15:18:38.796152] [spgwu] [spgwu_app] [info ] Received SXAB_SESSION_MODIFICATION_REQUEST seid 0x1
[2022-03-30T15:18:42.454298] [spgwu] [spgwu_sx ] [info ] TIME-OUT event timer id 153
[2022-03-30T15:18:42.455789] [spgwu] [spgwu_sx ] [info ] TIME-OUT event timer id 154
[2022-03-30T15:18:42.455861] [spgwu] [spgwu_sx ] [info ] PFCP HEARTBEAT PROCEDURE hash 3457419786 starting
[2022-03-30T15:18:42.457895] [spgwu] [spgwu_sx ] [info ] handle_receive(16 bytes)
[2022-03-30T15:18:42.457998] [spgwu] [spgwu_sx ] [info ] Received SX HEARTBEAT RESPONSE

log not “Received SX HEARTBEAT RESPONSE”
[2022-02-09T09:16:15.908908] [spgwu] [spgwu_app] [start] Options parsed
[2022-02-09T09:16:16.338099] [spgwu] [spgwu_app] [info ] SettingNotFoundException : SPGW-U.ITTI_TASKS, using defaults
[2022-02-09T09:16:16.338724] [spgwu] [spgwu_app] [info ] THREAD_POOL_SIZE : 1
[2022-02-09T09:16:16.339274] [spgwu] [spgwu_app] [info ] THREAD_POOL_SIZE : 1
[2022-02-09T09:16:16.339669] [spgwu] [spgwu_app] [info ] THREAD_POOL_SIZE : 1
[2022-02-09T09:16:16.340178] [spgwu] [spgwu_app] [info ] ==== EURECOM SPGW-U vBranch: HEAD Abrev. Hash: 53a7e4b Date: Wed Dec 15 11:58:44 2021 +0100 ====
[2022-02-09T09:16:16.340451] [spgwu] [spgwu_app] [info ] Configuration:
[2022-02-09T09:16:16.340618] [spgwu] [spgwu_app] [info ] - FQDN ..................: oai-spgwu
[2022-02-09T09:16:16.340776] [spgwu] [spgwu_app] [info ] - Instance ..............: 0
[2022-02-09T09:16:16.341112] [spgwu] [spgwu_app] [info ] - PID dir ...............: /var/run
[2022-02-09T09:16:16.341380] [spgwu] [spgwu_app] [info ] - ITTI tasks:
[2022-02-09T09:16:16.341548] [spgwu] [spgwu_app] [info ] ITTI Timer task:
[2022-02-09T09:16:16.341711] [spgwu] [spgwu_app] [info ] CPU ID .........: 0
[2022-02-09T09:16:16.342075] [spgwu] [spgwu_app] [info ] sched policy....: 1
[2022-02-09T09:16:16.342357] [spgwu] [spgwu_app] [info ] sched priority..: 85
[2022-02-09T09:16:16.342529] [spgwu] [spgwu_app] [info ] SPGWU-S1U task:
[2022-02-09T09:16:16.342691] [spgwu] [spgwu_app] [info ] CPU ID .........: 0
[2022-02-09T09:16:16.342851] [spgwu] [spgwu_app] [info ] sched policy....: 1
[2022-02-09T09:16:16.343315] [spgwu] [spgwu_app] [info ] sched priority..: 84
[2022-02-09T09:16:16.343488] [spgwu] [spgwu_app] [info ] SPGWU-SX task:
[2022-02-09T09:16:16.343650] [spgwu] [spgwu_app] [info ] CPU ID .........: 0
[2022-02-09T09:16:16.343810] [spgwu] [spgwu_app] [info ] sched policy....: 1
[2022-02-09T09:16:16.344075] [spgwu] [spgwu_app] [info ] sched priority..: 84
[2022-02-09T09:16:16.344106] [spgwu] [spgwu_app] [info ] SPGWU_APP task:
[2022-02-09T09:16:16.344113] [spgwu] [spgwu_app] [info ] CPU ID .........: 0
[2022-02-09T09:16:16.344119] [spgwu] [spgwu_app] [info ] sched policy....: 1
[2022-02-09T09:16:16.344124] [spgwu] [spgwu_app] [info ] sched priority..: 84
[2022-02-09T09:16:16.344130] [spgwu] [spgwu_app] [info ] ASYNC_SHELL_CMD task:
[2022-02-09T09:16:16.344135] [spgwu] [spgwu_app] [info ] CPU ID .........: 0
[2022-02-09T09:16:16.344141] [spgwu] [spgwu_app] [info ] sched policy....: 1
[2022-02-09T09:16:16.344146] [spgwu] [spgwu_app] [info ] sched priority..: 84
[2022-02-09T09:16:16.344151] [spgwu] [spgwu_app] [info ] - S1u_S12_S4:
[2022-02-09T09:16:16.344157] [spgwu] [spgwu_app] [info ] iface ............: eth0
[2022-02-09T09:16:16.344166] [spgwu] [spgwu_app] [info ] ipv4.addr ........: 192.168.70.134
[2022-02-09T09:16:16.344174] [spgwu] [spgwu_app] [info ] ipv4.mask ........: 255.255.255.192
[2022-02-09T09:16:16.344179] [spgwu] [spgwu_app] [info ] mtu ..............: 1500
[2022-02-09T09:16:16.344185] [spgwu] [spgwu_app] [info ] port .............: 2152
[2022-02-09T09:16:16.344190] [spgwu] [spgwu_app] [info ] Reader thread:
[2022-02-09T09:16:16.344196] [spgwu] [spgwu_app] [info ] CPU ID .........: 0
[2022-02-09T09:16:16.344201] [spgwu] [spgwu_app] [info ] sched policy....: 1
[2022-02-09T09:16:16.344207] [spgwu] [spgwu_app] [info ] sched priority..: 80
[2022-02-09T09:16:16.344212] [spgwu] [spgwu_app] [info ] thread pool size: 1
[2022-02-09T09:16:16.344218] [spgwu] [spgwu_app] [info ] - SXA-SXB:
[2022-02-09T09:16:16.344223] [spgwu] [spgwu_app] [info ] iface ............: eth0
[2022-02-09T09:16:16.344230] [spgwu] [spgwu_app] [info ] ipv4.addr ........: 192.168.70.134
[2022-02-09T09:16:16.344237] [spgwu] [spgwu_app] [info ] ipv4.mask ........: 255.255.255.192
[2022-02-09T09:16:16.344242] [spgwu] [spgwu_app] [info ] mtu ..............: 1500
[2022-02-09T09:16:16.344248] [spgwu] [spgwu_app] [info ] port .............: 8805
[2022-02-09T09:16:16.344253] [spgwu] [spgwu_app] [info ] Reader thread:
[2022-02-09T09:16:16.344259] [spgwu] [spgwu_app] [info ] CPU ID .........: 0 (TODO)
[2022-02-09T09:16:16.344264] [spgwu] [spgwu_app] [info ] sched policy....: 1 (TODO)
[2022-02-09T09:16:16.344270] [spgwu] [spgwu_app] [info ] sched priority..: 81 (TODO)
[2022-02-09T09:16:16.344275] [spgwu] [spgwu_app] [info ] thread pool size: 1 (TODO)
[2022-02-09T09:16:16.344281] [spgwu] [spgwu_app] [info ] - SGi:
[2022-02-09T09:16:16.344287] [spgwu] [spgwu_app] [info ] iface ............: eth0
[2022-02-09T09:16:16.344293] [spgwu] [spgwu_app] [info ] ipv4.addr ........: 192.168.70.134
[2022-02-09T09:16:16.344300] [spgwu] [spgwu_app] [info ] ipv4.mask ........: 255.255.255.192
[2022-02-09T09:16:16.344305] [spgwu] [spgwu_app] [info ] mtu ..............: 1500
[2022-02-09T09:16:16.344311] [spgwu] [spgwu_app] [info ] gateway ..........:
[2022-02-09T09:16:16.344316] [spgwu] [spgwu_app] [info ] Reader thread:
[2022-02-09T09:16:16.344321] [spgwu] [spgwu_app] [info ] CPU ID .........: 0
[2022-02-09T09:16:16.344326] [spgwu] [spgwu_app] [info ] sched policy....: 1
[2022-02-09T09:16:16.344332] [spgwu] [spgwu_app] [info ] sched priority..: 80
[2022-02-09T09:16:16.344337] [spgwu] [spgwu_app] [info ] thread pool size: 1
[2022-02-09T09:16:16.344342] [spgwu] [spgwu_app] [info ] - PDN networks:
[2022-02-09T09:16:16.344348] [spgwu] [spgwu_app] [info ] SNAT .............: yes
[2022-02-09T09:16:16.344357] [spgwu] [spgwu_app] [info ] NW .............: 12.1.1.0/24
[2022-02-09T09:16:16.344362] [spgwu] [spgwu_app] [info ] - NON_STANDART_FEATURES:
[2022-02-09T09:16:16.344368] [spgwu] [spgwu_app] [info ] bypass_ul_pfcp_rules: no
[2022-02-09T09:16:16.344373] [spgwu] [spgwu_app] [info ] - SUPPORT_5G_FEATURES:
[2022-02-09T09:16:16.344378] [spgwu] [spgwu_app] [info ] enable_5g_features: yes
[2022-02-09T09:16:16.344396] [spgwu] [spgwu_app] [info ] register_nrf: yes
[2022-02-09T09:16:16.344402] [spgwu] [spgwu_app] [info ] use_fqdn_nrf: no
[2022-02-09T09:16:16.344407] [spgwu] [spgwu_app] [info ] NRF:
[2022-02-09T09:16:16.344414] [spgwu] [spgwu_app] [info ] IPv4 Addr .......: 192.168.70.130
[2022-02-09T09:16:16.344420] [spgwu] [spgwu_app] [info ] Port ............: 80
[2022-02-09T09:16:16.344425] [spgwu] [spgwu_app] [info ] API version .....: v1
[2022-02-09T09:16:16.344436] [spgwu] [itti ] [start] Starting...
[2022-02-09T09:16:16.344626] [spgwu] [itti ] [start] Started
[2022-02-09T09:16:16.344640] [spgwu] [async_c ] [start] Starting...
[2022-02-09T09:16:16.346037] [spgwu] [itti ] [info ] Starting timer_manager_task
[2022-02-09T09:16:16.347988] [spgwu] [async_c ] [start] Started
[2022-02-09T09:16:16.348340] [spgwu] [spgwu_app] [start] Starting...
[2022-02-09T09:16:16.349957] [spgwu] [pfcp ] [info ] pfcp_l4_stack created listening to 192.168.70.134:8805
[2022-02-09T09:16:16.350268] [spgwu] [spgwu_sx ] [start] Starting...
[2022-02-09T09:16:16.351916] [spgwu] [spgwu_sx ] [start] Started
[2022-02-09T09:16:16.352091] [spgwu] [gtpv1_u ] [info ] gtpu_l4_stack created listening to 192.168.70.134:2152
[2022-02-09T09:16:16.352285] [spgwu] [spgwu_s1u] [start] Starting...
[2022-02-09T09:16:16.353916] [spgwu] [spgwu_s1u] [start] Started
net.ipv4.conf.eth0.rp_filter = 0
net.ipv4.conf.all.forwarding = 1
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0
[2022-02-09T09:16:19.028199] [spgwu] [spgwu_app] [start] Starting...
[2022-02-09T09:16:19.030633] [spgwu] [spgwu_app] [info ] Send NF Instance Registration to NRF
[2022-02-09T09:16:19.235455] [spgwu] [spgwu_app] [warn ] Could not get response from NRF
[2022-02-09T09:16:19.235873] [spgwu] [spgwu_app] [start] Started
[2022-02-09T09:16:19.236084] [spgwu] [spgwu_app] [start] Started

Hello, I have successfully compiled spgwc and spgwu through scripts, but I don't know how to run them. Can you provide an explanation document?

When running the spgwu there is a check that no other instances are running:

https://github.com/OPENAIRINTERFACE/openair-spgwu-tiny/blob/master/src/oai_spgwu/main.cpp#L84

This creates a lot of usability problems. If you run any other process that has spgwu in the name or as part of its argument list, spgwu will fail to start.

A more robust way could be to check the /var/run/ folder for a pid.

Update FAR (for downlink teid info) is broken, when we handle both UL/DL PDRs in session establishment message.

dl_pdr_test_with_spgwu_pcap.zip

[2023-02-08T15:26:52.406765] [spgwu] [spgwu_app] [info ] Received SXAB_SESSION_ESTABLISHMENT_REQUEST seid 0x0 
[2023-02-08T15:26:52.406800] [spgwu] [spgwu_sx ] [info ] pfcp_session::add(far) seid 0x1 
[2023-02-08T15:26:52.406809] [spgwu] [spgwu_sx ] [info ] pfcp_session::add(far) seid 0x1 
[2023-02-08T15:26:52.406849] [spgwu] [spgwu_sx ] [info ] pfcp_session::add(pdr) seid 0x1 
[2023-02-08T15:26:52.406873] [spgwu] [spgwu_sx ] [info ] pfcp_session::add(pdr) seid 0x1 

+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| PFCP switch Packet Detection Rule list ordered by established sessions:                                                                                                                          |
+----------------+----+--------+--------+------------+---------------------------------------+----------------------+----------------+-------------------------------------------------------------+
|  SEID          |pdr |  far   |predence|   action   |        create outer hdr         tun id| rmv outer hdr  tun id|    UE IPv4     |                                                             |
+----------------+----+--------+--------+------------+---------------------------------------+----------------------+----------------+-------------------------------------------------------------+
|0000000000000001|0001|00000001|00000000|ACC>---->COR|none                                   |GTPU_UDP_IPV4:00000001|12.1.1.2        |
|0000000000000001|0002|00000002|00000000|COR>---X> ? |none                                   |none                  |12.1.1.2        |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

[2023-02-08T15:26:52.406980] [spgwu] [pfcp     ] [trace] Sending PFCP_SESSION_ESTABLISHMENT_RESPONSE, seq 3320474 seid 0x1 
[2023-02-08T15:26:53.402257] [spgwu] [spgwu_sx ] [info ] handle_receive(95 bytes)
[2023-02-08T15:26:53.402403] [spgwu] [spgwu_app] [info ] Received SXAB_SESSION_MODIFICATION_REQUEST seid 0x1 

+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| PFCP switch Packet Detection Rule list ordered by established sessions:                                                                                                                          |
+----------------+----+--------+--------+------------+---------------------------------------+----------------------+----------------+-------------------------------------------------------------+
|  SEID          |pdr |  far   |predence|   action   |        create outer hdr         tun id| rmv outer hdr  tun id|    UE IPv4     |                                                             |
+----------------+----+--------+--------+------------+---------------------------------------+----------------------+----------------+-------------------------------------------------------------+
|0000000000000001|0001|00000001|00000000|ACC>---->COR|none                                   |GTPU_UDP_IPV4:00000001|12.1.1.2        |
|0000000000000001|0002|00000002|00000000|COR>----|none                                   |none                  |12.1.1.2        |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

context/reproducible experiment

The issue was encountered when connecting twice a UE to the OAI network without restarting any function neither core network functions, neither the gNB. All the core network and RAN were running on the same machine.

When running the UE for the first time, the user traffic was flowing normally as shows this capture of a icmp ping request and response in the core network:

Then after stopping and restarting the UE making sure that it disconnects from the network properly and connects again with the same ip address, the user data does not flow anymore in the downlink direction:

Capture of ping from the core network to the UE were also performed and confirmed that the problem was about the downlink traffic.

Important points noticed

1. We noticed that this issue was only happening if the UE was keeping the same ip adddress from one connection to another.
2. Restarting the SPGWU alone fixes the issue for one more connection.

Idea on the issue

From what we observed, we believe that this is probably happening because some structure or object that describe the PDU Session created at first UE connection is partially but not entirely cleaned at UE disconnection so that at second connection the SPGWU does believes it already has some PDU Session ready for this UE and does not setup again properly the downlink channel for the UE. If such structure are indexed by ip addresses, it would explain why changing the ip address solves the issue for one more connection.

Details on our setup

We have bee using the core network, RAN and UE as Snap packages developed in the Mosaic5G project.
The core network and RAN were running on only one machine.
Only one UE was used and connected to the network.
The UE was modified in a way that makes sure it disconnects properly from the network so that it get the same ip address at each connection since this address was properly released.

For more info

This issue was reported by Romain Beurdouche. You can contact me by mail at [email protected] for more information on the issue.

Can we use one mme,one hss and one spgwc and two spgwu?