Take the Poll on Making Mesa Faster

[tpike3]

Mesa is working to make Mesa faster. To do this our developers @rht and @Tortar have built a cython version of space.py that gives variety of speed ups.

The question, should the cython version be part of standard Mesa as the default build or should it be a different install?

Costs and Benefits:

• The benefits of being part of the standard library, the speed ups will be in Mesa by default.
• The cost is that if users want to extend space.py and make their own customizations for some specific need, it will be harder to do this.
• The additional challenge would be if Mesa has space.py and cython space.py it would increase the overhead on maintaining and developing Mesa.

[jdfoote]

In what ways would this make customizations more difficult?

[rht]

I suppose writing a customization in Cython is no different from writing it in Python. What is more difficult is to ensure that the implementation is performant/in idiomatic Cython.

[Tortar]

it is actually not that true though that the customization is more difficult, if by customization we mean inheritance cython classes can be inherited by python classes (with some perf penalty but faster than python anyway), so there is not much difference

[rht]

For disclaimer: I did vote for the standard build instead of the separate install, and but I try to be unbiased.

it is actually not that true though that the customization is more difficult, if by customization we mean inheritance cython classes can be inherited by python classes (with some perf penalty but faster than python anyway), so there is not much difference

Yeah, as long as the base classes are in Cython, there should be no problem.
The constraint in https://cython.readthedocs.io/en/latest/src/tutorial/cdef_classes.html

Normal Python classes can inherit from cdef classes, but not the other way around. Cython requires to know the complete inheritance hierarchy in order to lay out their C structs, and restricts it to single inheritance. Normal Python classes, on the other hand, can inherit from any number of Python classes and extension types, both in Cython code and pure Python code.

shouldn't apply either.

I also want to add that having separate repos for space.py and time.py is going to be a maintenance problem, as every changes to the 2 files will have to be incorporated into the Cythonized version.

[Tortar]

I actually flipped my vote for the standard build now, and standard build is winning :-)

[jackiekazil]

@Tortar single handed flipped the vote. 😆

[rht]

Adding that the Cythonized version would make Mesa to be within the order of magnitude of Agents.jl's performance, instead of an order of magnitude slower. At least in a hybrid Python-Cython version, and could be further faster in a full Cython version.

[Tortar]

I actually ran the latest comparison not so much time ago: https://juliadynamics.github.io/Agents.jl/stable/comparison/ , Agents.jl is usually 2 orders of magnitude faster than Mesa, only a full cython version could arrive within 1 order of magnitude in performance

[jackiekazil]

I have never done this, but would be willing to learn to help maintain.

[rht]

Another possibility that hasn't been explored yet is to run Mesa models with Agents.jl backend. In order to be fast, it would require transpiling the model to Julia objects all at once, instead of having Python objects that constantly get converted to Julia objects.

[jackiekazil]

I both like and dislike this idea... why do this instead of just writing a python wrapper around the ABM library in julia?

[rht]

The goal is so that existing Mesa models, which use Mesa API, can be faster without extra changes in the code. But this is probably too big of a surface, because there is no telling what Python-specific expressions people do in their model.

[rht]

There is PySR, which uses https://pypi.org/project/julia/.

[wang-boyu]

How about rust? There should be some rust abm frameworks and rust/python bindings.

[rht]

There is a problem with a Julia backend, for the use case of Binder/Colab notebooks. The environment would have to contain both Julia and Python, which may take a while to set up, since it can't be saved in a wheel. As such, it is still worthwhile to implement (for a Mesa model to access features in Agents.jl not yet available in Mesa), but probably shouldn't be in the default Mesa installation.

[rht]

Opening a separate thread for Rust:

How about rust? There should be some rust abm frameworks and rust/python bindings.

Needs an example of a scientific Python library that uses Rust under the hood. One that came to my mind is rustworkx, a faster NetworkX.

[rht]

Not to mention that hash.ai is JavaScript model, with Rust backend.

[alan-isaac]

polars

[rht]

Thank you for bring up Polars! Further discussion at #2027 (reply in thread).

[Corvince]

Although I am a bit late to join the discussion, I have a question: Is Mesa truly slow? While I acknowledge that it may be slower than Agents.jl by several orders of magnitude, I wonder if this necessarily implies that it is inherently slow. Considering that Mesa as a framework does not perform extensive operations, I am genuinely curious about the significance of performance in justifying some of the approaches being discussed here. In a reasonably complex model, what percentage of time is actually spent on executing Mesa code? Additionally, to the best of my knowledge, there haven't been any open issues complaining about Mesa's performance. There may have been one or two cases where people wanted to simulate millions of agents, but this approach is not feasible even if the performance were to increase significantly. Therefore, I fail to see the need to overly complicate Mesa's code. However, I am open to hearing any concrete evidence of situations where Mesa's performance has hindered simulations.

[Tortar]

I think that generally one doesn't want to simulate millions of agents, but this doesn't mean that simulations are cheap since the scanning of parameters can be very extensive in some cases, which anyway would require a lot of time anyway. But I have to say that it is true that even if some operations were performed order of magnitudes faster, the ones performed from python would become the bottleneck, there is no complete solution for the library to this (in this case little) "two language problem"

[jackiekazil]

This is great question @Corvince. I read in a few papers it was slow, but it was the example that @Tortar lays out that is basically the same problem people try to solve.

But I think you bring up a great quantifying point, which will clarify in the next comment. :-)

[rht]

I have an example of a large model that would benefit from a faster space.py: COVID19-mesa. The model runs on a supercomputer. The get_cell_list_contents by itself accounts for 9.3% of the elapsed for the agent step. I sped it up by accessing the content directly, and it went down to 2.5% of the elapsed for the agent step. This PR, together with ncsa/COVID19-mesa#36 and ncsa/COVID19-mesa#37, sped up the simulation from 60 minutes down to 24 minutes. This definitely made it easier to experiment with ideas / parameter sweeps. In all the options that have been proposed, they can be made such that the user-facing API is intact, and it would have looked like a free lunch to the user.

[rht]

There could be self-selection in researchers who have evaluated Mesa to be slow for the scale of their model, and have decided to implement their model in Agents.jl, without filing any request for a faster Mesa.

[Corvince]

I have an example of a large model that would benefit from a faster space.py: COVID19-mesa. The model runs on a supercomputer. The get_cell_list_contents by itself accounts for 9.3% of the elapsed for the agent step. I sped it up by accessing the content directly, and it went down to 2.5% of the elapsed for the agent step. This PR, together with ncsa/COVID19-mesa#36 and ncsa/COVID19-mesa#37, sped up the simulation from 60 minutes down to 24 minutes. This definitely made it easier to experiment with ideas / parameter sweeps. In all the options that have been proposed, they can be made such that the user-facing API is intact, and it would have looked like a free lunch to the user.

If you still have the benchmark scripts, it would be great if you could run them again to provide us with the total time spent on the mesa code. This would help us gain a better understanding of the overall performance.

It's essential to differentiate between framework code (mesa) and application code in order to assess the impact. In the COVID-19 model, after applying your patch (which still uses a public mesa API, but a more suitable one), only 2.5% of the elapsed time is spent on the mesa code. While I'm unsure about the exact portion of time allocated to mesa code in total, let's assume it's around 10%. From the remaining 24 minutes, we might be able to reduce it by 2.4 minutes at most. I don't since this is significant. The drop from 60 to 24 minutes was primarily due to your other patches, which modified application code. So it could be more beneficial to offer more optimized application code as framework code (in this case a probability function) than to optimize our grid implementation.

Let's also consider the insights gained from the benchmarks. The ForestFire (large) model appears to be significantly slower in mesa, roughly 400 times slower than Julia. Analyzing the code reveals that it consists mostly of a large grid with 500x500 cells and around 225,000 agents, with minimal application-specific code. In this case, mesa seems to exhibit slower performance.

In contrast, the WolfSheep model has a greater proportion of application-specific code. Here, mesa is "only" 37 times slower than Julia, indicating that there might not be much to gain in terms of optimization. It appears that Agents.jl has a highly optimized grid implementation, although its significance diminishes as the amount of application code increases.

[jackiekazil]

Based on @Corvince's comment, what specific measurable problem are we trying to solve here?

[rht]

I think a first milestone would be to be at least faster than NetLogo in https://github.com/JuliaDynamics/ABM_Framework_Comparisons.

[Corvince]

We hit that milestone just now! (at least for a single row in the comparison) https://github.com/JuliaDynamics/ABM_Framework_Comparisons/actions/runs/5778720377/job/15660253540#step:5:50

Unfortunately the ForestFire example will now be removed.
JuliaDynamics/ABMFrameworksComparison#60

[rht]

I think, in 2024, we will instead port the slow parts of Mesa into Rust. And if possible, allow users to write fast code like they do in Polars.

I have done Rust experiments in #2042. It absolutely can do NumPy array memoryview manipulation, at a cost of slightly more verbose code. Using Rust Vec is very concise, but for the resulting Vec to be visible in Python, they have to be converted to Python list, which involves a copy operation. The only way to avoid copying from scratch in Rust is to use rust-numpy library for fast manipulation of NumPy array, but at the expense of having to wrap the operation with unsafe{ }, which is undesireable.

[OrenBochman]

The use case I have in mind is running mesa simulation in a Python shinylive app in a quarto blog which uses pyodide to convert python to webassembly in the browser. It requires a pure Python package.

Ideally, we should have an installation options mesa[core], mesa[full] etc.
where the mesa[core] is pure Python without the server and vizulization.

[jackiekazil]

Mesa IS a pure python package. To use the server and visualization, you have to be explicit.