[Bugs] SMF panic on external (3rd party) CHF integration #562
Comments
|
Could you provide which CHF you are using? |
|
Commercial CHF from Comarch (https://www.comarch.com/telecommunications/bss-solutions/convergent-billing-system/) so not so easily available from outside (but if really needed, I could try to expose it to the external network). |
|
I just found a valid testcase, that does not require any 3rd party software :-) You just need to run Free5gc CHF on a separate host (VM). Steps to reproduce:
2024-04-30T11:08:59.151342404+02:00 [INFO][SMF][PduSess] CHF Selection for SMContext SUPI[imsi-208930000000001] PDUSessionID[2] runtime error: invalid memory address or nil pointer dereference |
Another thing that has to be configured is configuration:
chfName: CHF # the name of this CHF
sbi: # Service-based interface information
nrfUri: http://192.168.16.130:8000 # a valid URI of NRF
mongodb: # the mongodb connected by this CHF
name: free5gc # name of the mongodb
url: mongodb://192.168.16.130:27017 # a valid URL of the mongodbHowever, I am still determining if this would solve the problem, but I will get back to you ASAP. |
|
At VM2 (with CHF) I used a separate/local MongoDB (that's why there is a local IP for MongDB in my CHF config). I assumed that CHF does not need to share any data through the DB (SBI should be the only point of communication between CHF and SMF). I started webconsole on VM2 to prepare the same set of data (SIM information) as I have at VM1. This simulates quite well my real-live scenario, where CHF from different vendor uses only SBI to talk to NRF and SMF. |
|
In our Charging design, the I am checking the spec whether this procedure follows the rule. |
|
In my CHF rating groups must be directly configured. When we integrate CHF with customer network, we must agree rating group IDs that will be provided from the network and prepare configuration on our side (e.g. different charging rules per each RG). On the same way we need to provision SIM data for ech SIM card registered in customer network directly to CHF. If Free5gc CHF shares database with other network components it is a problem in real-life scenarios, where the network uses components from different vendors. So for the target solution I think there should be a place in Free5GC configuration (GUI) where you can directly define rating group configuration (or at least display RGs automaticaly configured by F5GC). In this case I'd configure the same RG IDs in CHF with dedicated charging configurations per each group). For now - is there any workaround possible? E.g. can I somehow directly insert those RGs to MongoDB? |
|
The rating group would be allocated by the PCF when the PDU Establishment procedure and the PCF would write the rating group to MongoDB. So, there is no way you want to use the rating group from MongoDB if you don't change the code in PCF. However, your point about "there should be a place to define rating group directly" is correct. |
|
Can you please decribe in more details how it works currently? I assumed that it works this way:
At this moment I don't understand, why SMF crashes before sending any data to external CHF (if only SBI should be used between those components). So can you please clarify how it is implemented currently? |
|
Sorry for the late reply 🙇 I looked deep into the log you provided from the first message; I didn't see that CHF registered to NRF. |
|
Please check the trace I attached for HTTP at port 8000. This is our misconfiguration that our CHF uses HTTP instead of HTTP/2 for NRF registration. But besides of that communication looks correct. CHF issues bootstrap message (which is not recognized by your NRF, as this is from Rel 17 (not supported on your side yet), but later it starts normal registration, that is completed with success (also we set an annoying 1s watchdog which also is correctly supported by NRF, so later we have PATCH watchdog requests each second ;-) ). See attached screen:
But as I wrote in my post (Apr 30) - the same issue can be reproduced with Free5gc CHF installed on separate VM (along with the second instance of Webconsole - to create te same SIM config in MongoDB). |
|
Could you provide the CHF config? |
|
Nothing interesting there - only SBI and NRF IP addresses changed. |
|
I didn't see where is this packet, is it from another pcap file? |
|
Those packets are filtered directly from trace.zip that was attached to my first post here. Maybe you need to change decoding manualy in Wireshark:
|
|
Thanks for the reminder. I forgot to adjust the decode setting in Wireshark. I discovered that the NFProfile from external CHF didn't contain And the SMF would use // Create Converged Charging Client for this SM Context
for _, service := range *smContext.SelectedCHFProfile.NfServices {
if service.ServiceName == models.ServiceName_NCHF_CONVERGEDCHARGING {
ConvergedChargingConf := Nchf_ConvergedCharging.NewConfiguration()
ConvergedChargingConf.SetBasePath(service.ApiPrefix)
smContext.ChargingClient = Nchf_ConvergedCharging.NewAPIClient(ConvergedChargingConf)
}
}These cause the SMF panic when connect with external CHF. |
|
I haven't tried to reproduce the use of our CHF in another VM, whether it would lead to the same question you have mentioned. |
|
Ok, I'll check this apiPrefix. But this does not explain crash with Free5gc CHF. I'll try to reproduce it once more with your CHF and attach all logs/traces. |
|
Here are logs and traces from the test with Free5gs CHF running on second VM: I see that apiPrefix is set correctly by CHF:
But there is still the same crash on SMF side (see attached logs). Tomorrow we'll try to add apiPrefix to our CHF, but it seems like this does not solve the issue... |
|
I tried to run our CHF on another VM, and everything work normally without any panic. info:
version: 1.0.3
description: CHF initial local configuration
configuration:
chfName: CHF # the name of this CHF
sbi: # Service-based interface information
scheme: http # the protocol for sbi (http or https)
registerIPv4: <VM2> # IP used to register to NRF
bindingIPv4: <VM2> # IP used to bind the service
port: 8000 # port used to bind the service
tls: # the local path of TLS key
pem: cert/chf.pem # CHF TLS Certificate
key: cert/chf.key # CHF TLS Private key
nrfUri: http://<VM1>:8000 # a valid URI of NRF
nrfCertPem: cert/nrf.pem # NRF Certificate
serviceNameList: # the SBI services provided by this CHF, refer to TS 32.291
- nchf-convergedcharging # Nchf_AMPolicyControl service
mongodb: # the mongodb connected by this CHF
name: free5gc # name of the mongodb
url: mongodb://<VM1>:27017 # a valid URL of the mongodb
quotaValidityTime: 10000
volumeLimit: 50000
volumeLimitPDU: 10000
volumeThresholdRate: 0.8
cgf:
hostIPv4: <VM1>
port: 2122
listenPort: 2121
tls:
pem: cert/chf.pem
key: cert/chf.key
cdrFilePath: /tmp
abmfDiameter:
protocol: tcp
hostIPv4: 127.0.0.113
port: 3868
tls:
pem: cert/chf.pem
key: cert/chf.key
rfDiameter:
protocol: tcp
hostIPv4: 127.0.0.113
port: 3869
tls:
pem: cert/chf.pem # CHF TLS Certificate
key: cert/chf.key # CHF TLS Private key
logger: # log output setting
enable: true # true or false
level: info # how detailed to output, value: trace, debug, info, warn, error, fatal, panic
reportCaller: false # enable the caller report or not, value: true or falseVM1 runs all NFs (without CHF) and MongoDB. |
|
Could you provide all the config files of the free5GC core? |
|
Looks like there is some progress - after we added apiPrefix to NRF registration, now we get charging request from F5gc. Now our CHF responds with Bad Request, but I don't know yet why (maybe some misconfiguration on our side). I'll notify you as soon as i'll find something. Regarding your CHF running on different VM - at first I had some crashes (as previously described) but then I dropped MongoDB and cleared/restarted everyghing and it helped. So I can confirm that CHF can run in this configuration without any issues. |
|
We've published the PCF (charging-related) document here. |
|
Thank you, I'll check. But my latest tests look very promising. I exchanged first CHF message between F5gc and Comarch CHF. There are still some errors, but it looks like configuration issues on our side, so I hope it'll be working soon. Definitely I'll share the final results with you :-) |
|
Ok, nice to tell you, that we almost have success. I managed to confgure our CHF to exchange messages with Free5GC. At first I did not add test IMSI on CHF side, so it correctly communicated USER_UNKNOWN to the F5gc. So I added IMSI to CHF and it replied with an error, that the Rating Groups are uknown. So it looks like everything is ok on communication level, but the real issue is with Rating Groups. So I need to directly configure Rating Groups - either on F5gc side, or my CHF side. I know that for this moment I cannot do this directly on F5gc side, but is there any way to check, which RG IDs will be used in communication with CHF (and what is the meaning of each RG)? I see that for online charging RGs 13 and 15 were used and for offline charging 11 and 12 - but are they generated somehow dynamically, or can I stick to those IDs (and configure it on my side?) And the second (less important) question. Currently I registered in NRF only Converged Charging Service and all communication goes through this endpoint (both for online and offline traffic). But according to the specification, there is also Offline Only Sevice for CHF. Is it also supported by Free5gc? What would happen if i'd register it also in NRF? |
|
|
I checked how Rating Groups are allocated in PCF and I see a big issue with it, when thinking of it in terms of external rating/charging configuration. By definition, Rating Group should gather a set of services that have the same cost/rating type/rating rules. So I'd expect that whenever I'll add new IMSi to the Free5gc with the same confguration as others already defined, it shoud share the same Rating Group ID. But instead, I see that each new SIM gets a new set of Rating Groups. E.g. I defined three SIMs, two with exactly the same configuration and one with an additional Flow Rule. Those SIMs do not use the same RG IDs for the same configs, but new RG IDs were created for each SIM. With this approach it is not possible to configure any rating/charging rules per RG in the external Convergent Charging System (in target system I expect to have a lot of SIMs, but only a few RGs to be configured). Would it be possible to have reusable/shared/configurable Rating Groups in next Free5gs releases? For now this is a blocker for me to move forward with any integration... :-(
|
|
Thanks for providing the information. The next release will have some bug fixes and a minor enhancement, but not contain the configurable Rating Groups. --
Could you provide more details about what the rating group should share in your cases? |
|
On my side I only need to know Rating Group ID. In real life scenarios each customer (network provider) has only a few Rating Groups configured (e.g. 2 - 10), depending on specific customer business needs. Each RG is related to some traffic classification, that need to be processed separately in rating/charging. Some Rating Group examples used by our customers:
So in Free5gc I think a good place for Rating Groups classification would be SIM configuration panel and network slice / flow rule context. E.g. I could configure 1000 SIMs, each with 3 Flow Rules (same flow definition per each SIM) and for example 4 Rating Groups (1-3 related to traffic from each flow rule, and 4 for any other traffic). In this scenario I could prepare 4 charging configurations in CHF (e.g. if it is traffic from RG 1, this means that UE connected to 8.8.8.8/32 adress range (flow rule) and this traffic should be free of charge. Whilst other RG should be extra expensive, or has some limits on amount of data that can be used. |
|
We will discuss how to adjust the charging function in free5GC for future releases. If we encounter any issues or have questions, we will post them here. Additionally, if you have any more questions, we are happy to discuss them with you. |
|
Sure, if you'll have any charging-related questions, you can put it here, write to me on LinkedIn, or we can even think of some conf-call. I'm still interested in integration between Comarch OCS/CHF and Free5gc, as I'd like to use Free5gc in our official 5G Lab (https://www.comarch.com/telecommunications/5g-lab). For now, Rating Groups configuration is a blocker, but it looks like we can solve it together. BTW, I tried to do a simple workaround and manually updated RG IDs in policyData.ues.chargingData after they were created by PCF, but I see, that with each connection attempt, those ID are dynamically overwritten with new values (I hoped that PCF creates those IDs only once, so then I could update it and use my expected values). So for now I see no valid workaround for this issue and I must wait for changes on your side... |
Describe the bug
I'm trying to configure Free5gc to use CHF from the external provider. I succesfuly registered this CHF in NRF, but when I try to connect UE, the following errors occur (full logs also attached):
SMF:
2024-04-29T16:59:34.312541475+02:00 [INFO][SMF][PduSess] CHF Selection for SMContext SUPI[imsi-208930000000001] PDUSessionID[2]
2024-04-29T16:59:34.315972245+02:00 [INFO][SMF][Charging] Handle SendConvergedChargingRequest
2024-04-29T16:59:34.316362270+02:00 [ERRO][SMF][GIN] [Debugging] panic:
POST /nsmf-pdusession/v1/sm-contexts HTTP/2.0
(...callstack...)
2024-04-29T16:59:34.316934563+02:00 [INFO][SMF][GIN] | 500 | 127.0.0.1 | POST | /nsmf-pdusession/v1/sm-contexts |
AMF:
2024-04-29T16:59:34.192410890+02:00 [DEBU][AMF][Gmm][amf_ue_ngap_id:RU:1,AU:1(3GPP)][supi:SUPI:imsi-208930000000001] Search SMF from NRF[http://192.168.16.130:8000]
2024-04-29T16:59:34.256748241+02:00 [ERRO][AMF][Gmm][amf_ue_ngap_id:RU:1,AU:1(3GPP)][supi:SUPI:imsi-208930000000001] CreateSmContextRequest Error: undefined response type
This only happens with an external CHF, when Free5gc CHF is used, everything works ok.
Except NRF registartion request and watchdog requests I see no other communication between external CHF and Free5gc, so I think the issue is not related to the external CHF response format or behaviour.
To Reproduce
Environment (please complete the following information):
Trace File
Configuration File
config_e500.zip
PCAP File
trace.zip
Log File
amf.log
smf.log
The text was updated successfully, but these errors were encountered: