Make axon Threading heuristics MPI aware

[siboehm]

It's better to do the threading settings in Axon, instead of setting GOMAXPROCS on infra. Allows us to change it more easily, and adapt to different model sizes.

This will get slightly more complex for cross machine (we'll have to do OMPI_COMM_TYPE_HOST comm splitting).

[siboehm]

@rcoreilly @etuleu let's discuss here instead.

I'd prefer having the code have more knowledge about it's environment, that way it's much easier to update the heuristics and we don't need to touch infra code. We want to be able to do things like overallocate Goroutines, so adjusting down GOMAXPROCS may not even always be a bad idea.

Once emer/empi#11 is implemented, the code would also become aware of SLURM environments and LVIS would continue working as is.

[rcoreilly]

I think it would be much cleaner and simpler to have one clear way of parameterizing and understanding what axon does (n mpi procs x t threads per proc), and let the infra send in the appropriate params explicitly, rather than having axon try to figure things out on its own. If we need an alternative to GOMAXPROCS then we can add that as a startup arg.