Roadmap/active areas?

[msarahan]

I'm really impressed by the work you all have done since I was last involved. I would like to chip in, but I'm not sure where to start. I've been following @pjrule and @InnovativeInventor latest work with optimizing things, and I also saw the julia implementation and the frcw.rs implementation. Is speed something that you could use help with? Would it help to try to get frcw.rs usable in GerryChain? I have some experience with getting rust code usable as python extensions. Is there a slack channel for developers? Sometimes that helps people passively observe what the active areas are.

[InnovativeInventor]

Yes, there is a slack channel! I'm not sure how to add you (or if I have the permission to). Roughly speaking here are the things that are going on right now (just to provide some visibility):

• pcompress is now no longer a skunkworks project and is more-or-less the de-facto way we run/store/replay chains these days (this is also the way we're planning on adding interop between the various GerryChain implementations out there).
• GerryOpt is aiming to parse the AST of updaters so we can write updaters in Python but get Rust-style performance (complementary to frcw).
• GerryChain.NET another GerryChain implementation, with a focus on optimization methods (as in optimizing certain objective functions).
• frcw Insanely fast. Already has pcompress interop built-in (currently broken I believe, but we're planning to fix this soon).

For interop, we are really trying to prioritize backwards compatibility (and usability in Python). In general, we don't want to force users to pick up a new language or break their code. Personally, I've used GerryChainJulia -> GerryChain Python interop via pcompress for some performant VRA chain run analysis.

In terms of getting rust code usable with GerryChain, help would be appreciated! We're tackling this problem on several fronts. Currently, it is possible to use pcompress's streamable, compressed binary format for interop with frcw (there is a slight bug right now that makes this hard though). However, we'd like tighter (in other words, more performant) interop with Rust where Python Partition objects are instantiated in parallel and stuffed with pre-calculated updaters in Rust before being sent to Python. This is a work in progress and help is totally welcome!

Some stuff I'm personally working on (and the motivation):

• One goal for me is speeding up replays (in particular, Partition instantiation times). As this is the main bottleneck in any interop effort, the faster the replays can get, the better performance we'll see.
• I've been meaning to produce a pcompress v2 format (compressed protobuf stream maybe) that allows us to save the metadata of chain runs, etc. (I've got a few regrets about the current pcompress chain format). This isn't super-urgent as the current binary format works perfectly fine.
• GerryChain Cloud (not sure what to call this!) -- automatic storing of pcompress chain runs in a s3 bucket (along with all the metadata like timestamp, git hash, graph hash, dep versions, etc.). This is basically done (have a slight bug to work out before I push and publish). This is the current draft API implemented:

from pcompress import Record

for partition in Record(chain, "pa-run.chain", cloud="True", metadata = {whatever you want}):  # will still save to disk as well
    # normal chain stuff here

from pcompress import History

experiments = History.search(filename = "pa_run.chain")  # search by metadata
experiment = History.search_one(filename = "pa_run.chain")

for partition in Replay(graph, experiment, updaters=my_updaters):  # will error if graph hash does not line up -- this behavior can be suppressed with a user-provided flag
   # normal chain stuff here

[InnovativeInventor]

Oh, and I'm trying to add types to this codebase (with MonkeyType) here: https://github.com/mggg/GerryChain/tree/type-annotations

[msarahan]

Many thanks for the overview. Thanks also for pinning this issue. Perhaps it should be converted to a "Discussion"? That has some upvoting capability that might help with prioritizing things with community input somewhere down the line.

[InnovativeInventor]

Just did -- I've never used GitHub discussions, but it looks useful!

[InnovativeInventor]

For some public visibility, @msarahan: I have a 15-16x speedup on chain run times (backwards-compatible, experimental code is on the retworkx branch). It uses retworkx, which is a networkx-inspired library written in Rust (but usable from Python). I think there is a memory leak or stack issue with my code (occurs consistently at ~60k steps), but it is mostly complete.

Also, here is a graph of the recent performance improvements in the perf-tuning branch that I showed at lab meeting earlier this week:

The recent speedups that I wrote start at around ~27 commits ago (right when it starts to grow). The bump at around ~18 commits is from @pjrule 's __slots__ PR. The massive spike at ~16 commits ago is when I introduced a use_cut_edges flag to turn off the relatively expensive cut_edges updater (previously, it was force-enabled). All other speedups can be attributed to some refactoring, implementing caching, and the FrozenGraph datastructure.