Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Netmap could not allocate memory : netmap_config_obj_allocator requested objsize 140032 out of range [256, 131072] #929

Open
elite12monkey opened this issue Jul 4, 2023 · 0 comments

Comments

@elite12monkey
Copy link

env: FreeBSD 14.0 current ( just 1 week old on this day )
system has 8GB RAM with ZFS
card IXGBE 550 T2 with 8 hw queues
processor: i5 6500 , 4 cores ( this processor has no hyper threading)

i am new to github and please pardon me for any mistakes. Excellent project and thanks for putting it into open-source domain.
I have some good experience at C programming and is facing an abnormal situation. I am interested in only packet capture ( no TX required).
my app cannot guarantee that it will be be able to poll the fd as fast as possible. ( due to unavoidable reasons, latencies for POLL can be upto 10 ms/milliseconds between successive polls)

also note that i am requesting 16384 extra bufs per ring to implement ZC, in Arg3

default netmap comes with rxd size 2048,
if i change the rxds using the setting in /boot/loader.conf
dev.ix.0.iflib.override_nrxds="4096"
and start my app i get the following messages
kernel: 754.834915 [ 852] iflib_netmap_config txr 4 rxr 4 txd 2048 rxd 4096 rbufsz 2048
kernel: 754.835050 [1026] netmap_obj_malloc netmap_ring request size 65792 too large
kernel: 754.835061 [2001] netmap_mem2_rings_create Cannot allocate RX_ring

since default ring in freeBSD is around 38000

sysctl dev.netmap.ring_size=69000
and restarting application, it works fine

Till above everything is fine.
However, i have patched the ixgbe driver in freebsd to allow max nrxds from 4096 to 32768
after compiling kernel and all, i set this in /boot/loader.conf to have 8192 descriptors

dev.ix.0.iflib.override_nrxds="8192"

at this point, all networking etc ( non netmap things ) are working fine.
now when i start the my app the following kernel messages comes,

kernel: 076.863860 [ 852] iflib_netmap_config txr 4 rxr 4 txd 2048 rxd 8192 rbufsz 2048
kernel: 076.864016 [1026] netmap_obj_malloc netmap_ring request size 131328 too large
kernel: 076.864027 [2001] netmap_mem2_rings_create Cannot allocate RX_ring

at this point, netmap tells me ring size is less, just like the above case, so i set it like this
sysctl dev.netmap.ring_size=139000

however, now when i start the app, the get the following messages.
kernel: 267.695175 [1347] netmap_config_obj_allocator aligning object by 32 bytes
kernel: 267.695187 [1352] netmap_config_obj_allocator requested objsize 140032 out of range [256, 131072]

i have checked netmap source code, and figured out the following limits

.objmaxsize = 32*PAGE_SIZE,

which states that 32*4096 = 131072

after patching netmap and increasing max size, everything works great.
so my question is can it be controlled via sysctl or should i patch this into source code i maintain.

also note that i updated the max size of object in object pool.
Can you please provide a sysctl for freebsd ( or other platforms ) to control max_size of ring and max size of object in your allocator pools or whether it could be linked to existing sysctls ( this way we do not need new sysctls and existing sysctl will get the job done.)
for eg.
if dev.netmap.ring_size is greater than max size of ring pool, then automatically increase the max limit of pool.

same can be done with object pool,
if ____ is greater than max size of ring pool, then automatically increase the max object size of pool

Thank you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant