You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
After introducing BGP-EVPN in our environment, we now have hundreds of network interfaces and tens of thousands of routes on our systems. This made pyroute's "dump" operations unusably slow. Analysis revealed that a lot of CPU time is spent on parsing netlink messages.
Situation was so bad that we decided to reimplement dump operations from scratch. We used "on demand" parsing, skipping all NLAs except those that we need. This gave us a couple of orders of magnitude speedup. But as a consequence of "on demand" approach, the API to our implementation is quite different from that of pyroute (and much lower level), so it it not possible to contribute our optimisations directly to pyroute.
However, I have an idea that may allow to use similar optimisation with existing, or only slightly modified, pyroute API. The idea is to not parse the message that was received, keeping it as bytes blob. And use a custom implementation of getitem() to look up the NLA only when it is requested. (And possibly cache the result in case it will be requested again.) This would also allow to revive the possibility to fetch NLA attributes as python object attributes, by implementing getattr() in the same way, without loss of performance.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
After introducing BGP-EVPN in our environment, we now have hundreds of network interfaces and tens of thousands of routes on our systems. This made pyroute's "dump" operations unusably slow. Analysis revealed that a lot of CPU time is spent on parsing netlink messages.
Situation was so bad that we decided to reimplement dump operations from scratch. We used "on demand" parsing, skipping all NLAs except those that we need. This gave us a couple of orders of magnitude speedup. But as a consequence of "on demand" approach, the API to our implementation is quite different from that of pyroute (and much lower level), so it it not possible to contribute our optimisations directly to pyroute.
However, I have an idea that may allow to use similar optimisation with existing, or only slightly modified, pyroute API. The idea is to not parse the message that was received, keeping it as
bytes
blob. And use a custom implementation ofgetitem()
to look up the NLA only when it is requested. (And possibly cache the result in case it will be requested again.) This would also allow to revive the possibility to fetch NLA attributes as python object attributes, by implementinggetattr()
in the same way, without loss of performance.What do you think about such idea, @svinota ?
Beta Was this translation helpful? Give feedback.
All reactions