Memory issues

[StakFallT]

I know it's been said before about the very long generation time, and I too am experiencing that. However, a bigger concern I have is the memory usage. I'm running Chatette on Hyperconverge on a VM that my Rasa bot is on. This VM is kUbuntu. Here's the uname -a output:
Linux Huginn 6.2.0-34-generic #34~22.04-1-Ubuntu SMP PREEMPT DYNAMIC Thu Sep 7 13:12:03 UTC 2 x86 64 x86 64 GNU/Linux"

and here's the cat /etc/issue output:
Ubuntu 22.04.3 LTS \n \l
_

This VM has 8 cores with HT (so 16 logical cores) and had 30GB of ram assigned to it. Initially, the script kept exiting out with the last line saying "Killed". I came to realize it must be because the Linux kernel is invoking the OOM Killer. I browsed through the GitHub issue and came across some advice that said to break up the file and also try to minimize the length of alias chains. I've done all this and it still keeps getting killed. I've even tried adding in extra code into the Chatette package so I could see the recursion depth (using editor.mergely.com here's the generated diff file contents):

24c24,27
<     def generate_train(self):
---
>     def generate_train(self, NestedLevel=0):
>         strIndentation = "";
>         for i in range(0, NestedLevel):
>              strIndentation = strIndentation +"\t"
29c32,34
<             examples = intent.generate_train()
---
>             print(strIndentation + "<" + str(NestedLevel) + ">");
>             examples = intent.generate_train();
>             print(strIndentation + "</" + str(NestedLevel) + ">");
60a66
> 

Unfortunately, it only showed:
[...]
<0>
Killed

So I thought something else must be going on, then I started to unravel that it's because the code is written extremely meta / dynamic (makes sense to do so given the context of what the program's purpose is), but the AST code looked crazy nasty and without a full understanding of what is called when, where, and why, I then decided to power the VM down and bump the VM up to 64GB of ram, and left it generating over night. I came in this morning and it still got killed. I'm not sure how much more variable-expansion I can do without starting to really rack up some serious line counts in my chatette file... So as I mentioned earlier, I've split my master file across multiple files in an attempt to generate one set at a time (since I think somewhere in an issue it was mentioned that Chatette caches all of the combinations in memory) and avoid overusing memory and to better determine where the problem lies.

Here is the directory layout of files:

Chatette_Workspace/imperative_compound_1a.chatette
Chatette_Workspace/Imperative_Compound/Aliases/aliases.chatette
Chatette_Workspace/Imperative_Compound/Slots/slots.chatette

Here are the file contents

Chatette_Workspace/imperative_compound_1a.chatette:

%[imperativeform_compound]
	// Example:
	// <Show me> a <list> of <devices> <connected to> <Access_Point-109> that also <connected to> <Access_Point-204>
	~[imperativeform_openingverb] [a] @[Request_Output] @[Request_A] [that] [also] @[Request_B]
	~[imperativeform_openingverb] [a] @[Request_Output] @[Request_A] [that] [also is] @[Request_B]
	~[imperativeform_openingverb] [a] @[Request_Output] @[Request_A] [that] [also has] @[Request_B]
	~[imperativeform_openingverb] [a] @[Request_Output] @[Request_A] [that] [also are] @[Request_B]
	~[imperativeform_openingverb] [a] @[Request_Output] @[Request_A] [that] [also have] @[Request_B]
	~[imperativeform_openingverb] [a] @[Request_Output] @[Request_A] [that] [is also] @[Request_B]
	~[imperativeform_openingverb] [a] @[Request_Output] @[Request_A] [that] [has also] @[Request_B]
	~[imperativeform_openingverb] [a] @[Request_Output] @[Request_A] [that] [are also] @[Request_B]
	~[imperativeform_openingverb] [a] @[Request_Output] @[Request_A] [that] [have also] @[Request_B]
	~[imperativeform_openingverb] [a] @[Request_Output] @[Request_A] [that] [is] @[Request_B]
	~[imperativeform_openingverb] [a] @[Request_Output] @[Request_A] [that] [has] @[Request_B]
	~[imperativeform_openingverb] [a] @[Request_Output] @[Request_A] [that] [are] @[Request_B]
	~[imperativeform_openingverb] [a] @[Request_Output] @[Request_A] [that] [have] @[Request_B]

|Imperative_Compound/Aliases/aliases.chatette
|Imperative_Compound/Slots/slots.chatette

Chatette_Workspace/Imperative_Compound/Aliases/aliases.chatette:

//================== Alias definitions =======================
~[imperativeform_opening]
	~[&imperativeform_openingverb] ~[&determiner]

~[imperativeform_openingverb]
	[give me|provide me|provide for me|show me|tell me]

// Determiners
// 	https://dictionary.cambridge.org/grammar/british-grammar/a-an-and-the
// 	https://www.thoughtco.com/a-an-and-1692639
// 	https://www.grammar-monster.com/glossary/articles.htm
// 	https://english.stackexchange.com/a/328994
// 	https://en.wikipedia.org/wiki/Syncategorematic_term
~[determiner]
	[a|the]

~[Connecting_Words]
	[that]

~[Helper_Words1]
	[also|is|has|are|have]

~[also_Helper_Words2]
	[is|has|are|have]

Chatette_Workspace/Imperative_Compound/Slots/slots.chatette:

@[Request_Output]
	@[&Requested_Format] [of?] @[&Requested_Data]

@[Requested_Format]
	[list|graph|plot]

@[Requested_Data]
	[devices|users|locations|access points|ids|serial numbers|memory|disk space used|disk space in use|free space|memory]



@[Request_A]
	@[&Verb_A] @[&Data_A]

@[Verb_A]
	[connect to|connects to|connected to|connected at]

@[Data_A]
	[Access_Point-109|Access_Point-204]



@[Filter_Verb]
	[connect to|connects to|connected to|connected at]




@[Request_B]
	@[&Verb_B] @[&Data_B]

@[Verb_B]
	[connect to|connects to|connected to|connected at]

@[Data_B]
	[Access_Point-109|Access_Point-204]




@[Logical_Operators]
	[doesn't|does not|hasn't|has not|haven't|have not|aren't|are not|wasn't|was not|isn't|is not|not|is|had|has|are|have|was]

I'm going to keep trying to minimize and variable-expand but I don't hold out a lot of hope that my efforts alone are going to solve this. It'd also be nice to have some more debugging functionality / tools to determine where it's spinning out of control (over-recursing because of some logic error in our template files)