Upper bound on autoloading built-in modules

[rocky]

In the autoload-test branch I stripped down Mathics core to load only the essential modules to run Quit[] plus adding inout in case you want to try using at the stripped-down bare-bones Mathics3. (This basically provides about what mathics --full-form does. )

In sum we go from about 7 seconds down to 2.5 seconds. Of course this is going to slow down testing time a little bit since it has to ultimately load everything.

Using Python 3.10. 10. First the current load time:

$ ./time-load-test.sh 
Quit[]

real	0m7.259s
user	0m7.533s
sys	0m2.150s

and by removing Mathics3 autoload scripts, and Builtin modules except makeboxes, inout, and pattern:

$ ./time-load-test.sh 
Quit[]


real	0m2.353s
user	0m2.743s
sys	0m2.050s

[mmatera]

@rocky, the Definitions.__init__ method has a parameter add_builtin, which if is set to False, avoids loading any module. This is not very useful right now. One possibility would be to load the basic modules that you mention (inout, pattern, makeboxes, and maybe assignment) and then we would have this minimal environment using the existent API

[mmatera]

Actually, I remember that WMA had this kind of delayed load of rules: symbols that requires many complicated rules were initially attached to a rule that loads the package having the actual rules. Then, the first time the symbol is evaluated, the corresponding module is loaded.

For example, suppose that all the rules associated to Limit where stored in certain package lim.m and at the load time, Limit has just a ownvalue

Limit:= (Needs[lim`]; Limit=.)

Then, when the expression Limit[Sin[x]/x,x->0] is evaluated, the ownvalue in the head of the expression is evaluated first. If it was the first time when Limit appears in an expression, the module is loaded and the ownvalue is removed, and then, in the next evaluation round, the rules are available.

[mmatera]

In any case, the question about the overhead in the initialization is if it comes from parsing the new rules, or from building and storing the parsed rules.

[mmatera]

From the experiment I did of using pickle to store the session object, my guess is that the parsing step is not responsable of the overhead, but maybe I am missing something.

[rocky]

In any case, the question about the overhead in the initialization is if it comes from parsing the new rules, or from building and storing the parsed rules.

This is separate and different. I recall having this discussion before over separating the problem of making the interpreter run fast versus reducing the the startup time and reducing the the time to load a bunch of rules. While the later necessarily makes start up faster, GNU Emacs style autoload addresses just the startup time without speeding up interpreter time or the time to load rules.

GNU Emacs style autoload by lazy loading distributes the start-up time across the overall elapsed time spent in a session. And by doing lazy loading if there is some area that is not used like Quantum Mechanics, Calculus, or Statistics, then no time is spend in loading those modules, however fast or slow that loading is.

As stated before, testing everything is a little slower because there is overhead reading the autoload map and the initial overhead done to reissue the request after the module has been loaded.

GNU Emacs style autoloading is analogous to way virtual memory works in modern operating systems: first the system detects that the page you want isn't around, then it issues a command to load the requested page, and finally it has to reissue the instruction that was requested on that page.

As virtual memory has done for programs - allows programs to scale way beyond what they would be able to one had to be confined to actual memory - autoloading would do for the kinds of Mathics3 sessions that in fact don't use all of the Mathics3 builtin functions. This in turn makes scaling the number Builtin functions possible without paying the startup time price and memory footprint up front, which is the situation that we currently have.

[rocky]

Limit:= (Needs[lim`]; Limit=.)

I think this is the intent, but this would be done in Python. Something is populated in Definitions that marks a Symbol with the Python module it is in.

For example Binarize, ColorQuantize, etc. are located in mathics.builtin.image.colors . Then when accessing a definition like Binarize, some attribute or mark would indicate that loading the mathics.builtin.image.color module is needed. Loading that module loads not just Binarize but also ColorQuantize and the other builtin functions in that mathics.builtin.color. Then I guess the definition would either retry looking up the value, or raise an exception to have whatever was evaluated to restart.