Redesigning the event system

[kdschlosser]

I am in the process of redesigning the event system. What sparked this was the AddEvents dialog. and I had been thinking about how to populate the list with events. I didn't want to have to create the whole list of events when the dialog loaded as I know with my system that would be 10's of thousands of events. This could take a long while to populate if it had to build the list when the dialog loaded. and I started thinking. I had done some tinkering about and came up with a semi decent way.. but I knew i could make it better and more efficient. But also speed up event generation if using wild carded and also eliminate having to iterate through all of the events in the tree looking for ones that match. if you have a large tree and use wild carded events this will slow things down.

so this is what i came up with so far.

upon the load of EventGhost the first thing is that the plugins load. this would cause events to be loaded into a list if the plugin is using that mechanism. so what i thought would be best to do is this.
Instead of just adding them to a list I am splitting the event apart at the "."'s because of these "."'s an event is much like a python module. so i thought why not do something similar and and use the module outline. that way things don't get repeated that don't need to be.. but this will also make triggering an event faster due to the fact it only now has to iterate through a short list.

as an example..

Prefix1
       Midfix1-1
                      Suffix1-1-1
                      Suffix1-1-2    
       Midfix1-2
                      Suffix1-2-1
       Midfix1-3
Prefix2
        Midfix2-1
                      Suffix2-1-1
        Midfix2-2
                      Suffix2-2-1
                      Suffix2-2-2
                      Suffix2-2-3

so when an event gets triggered it would split the eventString and then search for only the prefix.
if found it would grab the event object then it would iterate the event object looking for the midfix... and so on and so forth. this makes much shorter lists to have to scroll through.

This works. You will notice a speed improvement if you have a lot of events in your tree. and it is a visible improvement.

Now if by chance there is no event object it will create on at that point. using the prefix of the event to determine if it's coming from a plugin. and put it into the correct place if it is. If the Event is dynamically created (meaning it was not added by a plugin) the information for that event as well as the plugin name will be saved into a file in the config directory called events.py. this will make it so that the AddEvents dialog can be added to as time goes on. and there is checking to see if the plugin has been removed and if so a message will ask about removing the stored events. but it will also check to see if the plugin has added the item and if it has it will be removed form the save as well. (if the plugin gets upgraded and the author has added the event)

I have also worked out a means to run each and every event in it's own thread. (if the user wants to) this can be turned on and off via a check box in the event dialog. I have maintained backward compatibility (which has been no easy task) with the existing system. so anything that uses eg.eventString, eg.result or eg.event things of that nature it will automatically go and grab the current thread (most of the time these are using in the macro which would be running inside the same thread) and with the current thread in hand it will be able to get the proper data for the running macro and pass it back. This is also nice because the actual event is storing the all of this information and the data can be held between triggers and can be accessed to find out what the last returned data was or when the last time that event was started or ended. (great for checking how fast a macro runs).

now that I have given you a basic understanding of what is going on.
I would like to know your thoughts on it.
I would also like to know any ideas or additions that should be added.
But also just putting things out there like hey did you remember to account for eg.programReturnStack which is used in managing the macros when it's running. I know there probably are some things that I have missed. or not accounted for. so please any comments or suggestions..

I know this code may or may not get used.. but it's a great proof of concept if i get it running 100% and could be used as an idea for version 1.0

So please tell me how it is.. if the idea sucks... then tell me.. and tell me why as well...

[ThomasBott]

Very nice, sounds all very logical and useful to me - I'm very curious about the "running each event in it's own thread but with backwards compatibility" thing. It sounds very good, I'm curious if it runs reliable and not leading to a lot of timing issues - if it does, holy shit! This could be the next big thing! ^^

[kdschlosser]

well this is what i did. i created objects that represent event, result, so on ans so forth. and by using the methods __get__ and __set__ this allows me to actually go and get the proper value from the proper event. and this would be dictated by the thread that goes and gets it. because again typically the things that would use attributes like eg.result and eg.event and eg.eventString would be from inside of the macro where the event was triggered. so the actions inside of the macro would be running in that thread 😄. and when an action calls for say eg.event the object i had set up would use threading.current_thread() as the key in a dictionary of running events and the value would be the actual event. and the event would house things like eventString. payload, result, and depending on which attribute you accessed it would go and get the proper value and return it. things would not be able to get all jumbled up this way.. and there would be no timing issue.

Now the only thing in which i am trying to get a handle on is with how EG is set up if you have a tree that is like so

folder
        macro1
                   event
                   action
        macro2
                   event
                   action
        folder
                   macro3
                              event
                              action
folder
         macro4
                    event
                    action

event being all the same event
I believe this is how EG works
eg would run them in this order macro1, macro2, macro3, macro4

it runs them in order down the tree. so if i moved the folder for macro4 up in the tree before the other folder. it would launch macro4 first.

I guess if someone has something that they need to have fired in a specific order. then they could disable the threading.. because they items would be added in the exact same order. I am trying to think of a simplistic way to go about doing this.. so if multiple events are in the tree it will automatically disable. It would have to because you wouldn't be able to run multiple threads for the same event.. that is when all the information would get fumbled up.

OK so that is a plan for handling that..

[ThomasBott]

I guess if someone has something that they need to have fired in a specific order. then they could disable the threading.. because they items would be added in the exact same order. I am trying to think of a simplistic way to go about doing this.. so if multiple events are in the tree it will automatically disable. It would have to because you wouldn't be able to run multiple threads for the same event..

I don't understand that part. If every event has it's own thread, it would still execute the macros that matches the eventString in serial top to bottom, wouldn't it? The only difference is that multiple events can be processed at a time, instead of queueing them, right?

[kdschlosser]

ok for example.. if you have 10 macros in your tree and that all share the same event. it would start a thread. and then execute each macro top to bottom.

it wouldn't be very hard to change out how events are stored. at the moment i am using the eventString broken apart at the "."'s. i could change over to something like GUID's instead because of the use of the GUID now for marking specific tree items, I could use that as an identifier and create event objects based on that in conjunction with EventInfo objects which would represent the actual event its self. the only crappy thing is the only way for a user to reference it after the macro has finished running would be to either set eg.event into a global for later referencing or by the guid. from inside the macro while the thread is running is not an issue because the thread id is the identifier. and have the actual tree item hold the results and other things of that nature.

if the same event was getting triggered over and over again. the macros would have to be put into a queue because you would not want one thread changing the data that another may rely on... this would allow each macro to be processed without any bumping of heads.

The biggest purpose for this would be if for some reason a macro had to wait... it wouldn't stop other events from coming in and being executed. it would stop the same event from being executed right away. Now that being said... I will probably have to put thread locks in place for eg.globals attributes so that if a marco is running and using a specific attribute it will get locked so if a thread running in parallel tries to use the same attribute it would have to wait until the current macro has finished processing

[ThomasBott]

Maybe I got it all wrong..
I didn't look at the code, but let me try to give you an example how I understood the whole thing from a high level perspective:

Let's say a plugin executes eg.TriggerEvent two times in a row.
This now triggers the event creation function two times (still in the main tread) which will create two events and handover each one to a new tread it creates.
These threads that are then doing what was happening in the main tread before, but two times in parallel.
The only difference between the two new treads is a slightly modified namespace, so everything related to the event (eg.event etc.) would be thread local and not global, that means each macro/action that is triggered by an event will get the eg.event or whatever from it's thread.

If a macro waits (so it pauses the thread of it's event) and another event get's triggered, it will just be executed, because it runs in a different thread. Even if the event triggers the same macro that currently wait's in another thread. We then have one macro that runs in two threads parallel, I don't see any issue here.

Why would you need the event object or anything connected, after the thread finished?
I mean, as it is now, eg.event is overridden if another event appears anyway..

But maybe it's not working like that or I'm just blind, do you have some examples for me? Or use cases?

Thank you!

[kdschlosser]

I got ya.. the issue is with the way i have it currently coded.. is that if you have 2 of the same events that are triggered.. they would be sharing the same data set...

so for instance if something changes and the 2nd thread gets to finish first it could set something to eg.result and then the first thread could continue after it's wait but the next step would be waiting to check the eg.result to determine what to do next.. and if the data that was set by the first thread gets changed by the second.. then the first thread would do the wrong thing..

WHEW!!!! i don't wanna have to retype that ever again! LOL. But let me try and give you a tree layout use case

macro
         event
         action
         wait if action responds with a 1
         action would run if response is a 0

event comes in and macro starts in a thread.
first action responded with a 1 so now it's waiting..
another event comes in but not the same name, but changes somehting that would cause that first action to now respond with a 0
the same event as the first one comes in and now the first action spits out a 0 while the first event is still waiting..
and the second action performs what it is supposed to do for that duplicate event.
the wait stops and now continues... but hold on now the result has changed to a 0 because of the shared data issue.. and is not going to do what it was initially set to do.

best way i think i can explain it. so i can either issue locks and if the same data objects are accessed it would cause a hold on the second thread until that macro has finished. because of the complexities of the macros even if you have the same event in 2 different macros doesn't mean that both macros will need to access something like eg.event.payload. but if the first thread that runs would put a lock on it then if it was used in the second macro it would stop the second thread until the lock was released by the first event.. and that would take place at the end of the macro run for the first event.

or we would could check and see if there is more then one tree item registered to a specific event and if there is then it would create a single thread. run each macro just like it does now.. and if a duplicate event comes in whether or not there is more then one tree item if there is a running thread for that event. it will get placed into a queue specifically for that event to finish running.

Unless i use the actual tree item to house the data instead of some other object that is based off the actual eventString. and because each treeItem is "unique" then you would have a specific data set for that specific event treeItem. but the issue there is history. i explain history below. but the only way to identify a treeItem is by it's GUID. and it would be static and never changing. but is also not user friendly and because i wanted to keep it user friendly so you could grab the last event history by using something like eg.EventManager['some.event.string'] and it would return an object for that string and the object would be just like using eg.event and would house all the information from the last time it ran

History:
and I know that the current setup is eg.Result gets overridden and eg.event.. so on and so forth.. but it would be nice to be able to look at a history without the need to specifically set items into eg.globals.. to just have them there whenever i want.. or if i want access to all of the information in one place just drop the tree item or eg.event into a single global then you would have the last payload, start time stop time result and whatever else might be useful. because this is not a single use item.. it would get used over and over again each time that event gets triggered and the data would populate if there was a match in the tree for that event.

[ThomasBott]

I see, so eg.result is the problem?
Why not making it thread local then too? Just set it to None when the thread is crated... I mean, I don't think you will need the value of eg.result in another thread, so it doesn't need to be global, right?
It would also be easy to use the last value for your history then..

Each event (each thread) will need it's own data set, everything else doesn't make much sense, I think.

btw. History: That's also something new isn't it?
To be honest, it sounds like carrying a lot of unnecessary ballast, don't you think? I'm not in favor of carrying more history around than what you see in the Eventlog...

[kdschlosser]

well I know you are one for memory consumption and performance. things of that nature.. so the history would be something the user could turn on and off. I was using eg.result as an example. eg.result. eg.event.payload eg.eventString.. they all would be located in the event object..

and what i was saying is if there is only one event object and there are 10 event tree items. in 10 different macros. then the 10 macros would have to share the data set from the one event item.

it just makes it a little more difficult to keep track of what's going on when how and why when you throw yet another object into the mix.

because for backwards compatability eg.EventGhostEvent has to remain. the treeItems have to remain. i have to add a new object called EventInfo which would represent the actual event it's self. and eg.EventManager which would manage registering of the event tree items, and the creation of the EventInfo objects as well as populating the AddEventsDialog, it would house the dictionary for the running threads as well as a dictionary or the prefix level EventInfo objects it would take care of the saving of events that are dynamically created as well as figuring out if it belongs to a plugin or not. whether it's an event tree item that the user made or from an actual event that was triggered.

the purpose to registering the event tree items is so that if a user uses wildcards it will only have to do the fnmatch when the event tree item is created and if a new event gets dynamically created and is housed in the same event structure as the wildcarded one.. which the latter would simmer down after EG has run for a while and has populated the saved events. so it would get to a point that the only time it would be used is when a tree Item got created.. and that happens when the users adds an event and when a saved xml is loaded.this in it's self is a big deal because fnmatch is expensive on the resources. and not something you want doing every time you have an event triggered which is the way it is set up now. so with the way i have started to make it will only slow down the load of EventGhost if you have a lot of them. not slow down the whole event process.

and history would be a yes but maybe No because I have not tested to see if eg resets the data to None. but also there is plugin.lastEvent or something like that. so the event is stored already. this would be very similiar but not just the last event every event would be stored. this is going to lent to slightly higher memory consumption if history is turned off. but the speed benifits would outweigh the memory consumption... once i get a running prototype i will check speeds. and post a link to the branch i create for it so you can give it a try.. it will be a first effort and not evenrything will be functional. like the AddEventDialog which will be last to integrate. and the save for the dynamic events will also be on the back burner. I am almost done coding up the threading bits. so if you can think of anything i need to focus on or may have forgotten. specifically in the area of attributes that are specifically used in macros. so i can make sure i have the event specific ones.. but this is what i have thus far

eg.result
eg.eventString
eg.event
eg.programCounter
eg.programReturnStack

[ThomasBott]

the history would be something the user could turn on and off

That's fine for me then

I was using eg.result as an example. eg.result. eg.event.payload eg.eventString.. they all would be located in the event object..
and what i was saying is if there is only one event object and there are 10 event tree items. in 10 different macros. then the 10 macros would have to share the data set from the one event item.
it just makes it a little more difficult to keep track of what's going on when how and why when you throw yet another object into the mix.
because for backwards compatability eg.EventGhostEvent has to remain. the treeItems have to remain...

That's basically what I wrote, right? No thread locks, no additional queues or threads or even data sets for single macros.
I don't know if it will work out 100% for backwards compatibility, but it sounds possible and we will only find if out if we test it.

the purpose to registering the event tree items...

Yes, I did get that part from your initial post and I like it :)
Please keep in mind that you also need to catch if an event tree item is modified, not only when it is created.

You will need eg.indent as well, otherwise the log will look very strange ;-)
Not sure if something else is missing, but I guess that's some good base to start with

[kdschlosser]

eg.indent..

well now there is in an interesting bit...

see because you will have multiple threads possibly running all over the place for various events.. things will be getting printed to the log all over the place. and not make for something easily followed..

so my plan is.. to use UltimateListCtrl. The reason for this is because you can add a wxWindow item to a list item.. and it also allows for the use of pydata so the secondary log queue can go bye bye.
It will be set up so the listctrls are nested. meaning.. you have the primary log... and when an event comes in it will show the event. and if there is a macro triggered by that event it will then show the macro name just under it but not as a different entry just new line thing..
but then you will have the ability to double click it and it will expand and inside with be a secondary log.. now the cool thing with this is the actual log for the event i am going to store so it can be used multiple times. nice thing about this is it will group all actions that have run for a specific event and macro all together. and will allow for proper indenting and what have you
and it will not have to create a new main log entry each time it will move it to the bottom of the list. and maybe put a pointer at the old position in the main log. and for the rest of the information it will append to the information inside of it. but i also wanted to have it add a pointer to the event log file if a triggerevent was called from inside of the running macro to point to the new event that was created in the main log.

and I am going to give an option to change whether you want to have it automatically expanded when it runs. and automatically collapse when finished.

this is a rough draft idea i am still working on the main log bits of it now. not to far off from finishing the conversion to UltimateListCtrl

I have also put some thought into using something like foldpanelbar to use for the main log and then you could expand and collapse the different entries that way. but I am not so sure it would look like a "log" so I am going to attempt the listctrl route.

and I also to a try/except when running the event because i set the threads up as daemons because it would be hard to try and stop them if someone has something running in a loop inside of it. so what i did was by using the daemons they will get closed with the main program but when you close EG as it destroys specific things during that process it will cause the running event to most likley raise an exception that gets caught because of me setting the base level catching and it will allow the thread to end all nice nice.. and the really cool thing is also if a user uses a thread inside of the event thread which you can do.. the thread they create by default will become a daemon thread unless they specifically specify something different. which will terminate when EG closes if it didn't throw a traceback for something.

in theory the threading setup should work.. gonna have to see. there are just so many little bits to this.. and the backwards compatibility thing for the moment is not any kind of a big issue. as i have already tested that piece of it and it works so i will not have to change any of the default variable names. the only big thing i am going to change as far as the existing "API" is how the logCtrl gets appointed to eg.Log at the present moment it does it it's self.. but instead of having duplicate code i feel it would be wiser to move it out of the mainframe folder and into Classes because of needing to use it more then the one time.. I do not want to have that much repeat code..

i think with some really thought out planning on this end of it. it could end up being something really nice and not to hard to follow. but will also allow for proper indications of what happened and when and it what order.

this is something i am really going to need a lot of ideas/advise on to make it the best that it can be..

this would be a HUGE upgrade to eg if i can get it to run 100%.. the threading bits i did a test and eg ran.. i did have things that needed to be sorted out but it showed that EG will do it. and a lot of the core bits and pieces do function in a manner that is good. just have to hammer out the log and the little pieces that we didn't think about. But if we can keep all of the various variable names the same then having it backwards compatible so plugins that rely on those bits will not have to be changed is a major time saving thing for everyone.

[kdschlosser]

and threading locks will have to be used if the same event comes in while another is running for the same event. the lock is what will act like the queue because it will stack threads requesting the lock and allow them to go in the order in which they tried to obtain the lock. this is a first effort setup and i am sure will be improved upon. I do have several ideas but i want to get the thing working and then change out things that could be made better.

[ThomasBott]

the listctrl thing sounds very interesting, I like it! :)

i set the threads up as daemons because it would be hard to try and stop them if someone has something running in a loop inside of it. so what i did was by using the daemons they will get closed with the main program but when you close EG as it destroys specific things during that process it will cause the running event to most likley raise an exception

Well, but if they don't? I classical thing like "while True" would never stop and continues running forever and the user may wonder what that high CPU load is - it's not EventGhost, I closed it! :)
No, I don't think it's a good idea to make all threads daemons by default.

and threading locks will have to be used if the same event comes in while another is running for the same event.

Why? What is the problem with running the same macro two times in parallel? I still don't get it

The whole thing will be a huge update indeed.

[kdschlosser]

as far as the daemons, i have to because there is no way to close them when they are running.

the only thing a daemon thread does is that when the process for eg terminates so does all the threads associated with it.. that is all it does. because there is no way to have the thread close if i am using a non daemon thread. so if the user does does "while True" in a loop unless it is a daemon thread it won't close properly when EG goes to exit. this is the solution to that. it's just how to deal with

[kdschlosser]

and I do have good news about the listCtrl thing... it works! well nesting them works.. i had to change out the default log control to the ultimate list ctrl and i made it so that the control doesn't run anything specific to being the main log. the only thing that i will have to do when i want to use it for the main log which actually is a good thing because i can still keep LogCtrl.py in the main frame folder is i have to override the OnItemActivate so for the main log double clicking an event will cause it to expand and show the secondary log.. and if you double click the event again it will close it and in the secondary log i am thinking about setting a header for it to the name of the event and if you double click the header it will go to that item in the tree logging is i will have to make a class and override one thing.

the really cool thing is the main log will ascetically look the same except it will only show events and tracebacks that happen outside of an event.

what happens when it gets expanded is it will fill the whole line item with an event log replacing the main log entry text. that is why i need to add the header so there is something to double click but also if you have a bunch of them open there is a way to identify it

[ThomasBott]

Oh yeah, I kinda mixed deamon up with something else.. My apologies, I agree then.

and I do have good news about the listCtrl thing... it works!

Excellent!

Sounds awesome / very promising!

[kdschlosser]

I do that all the time as well. But I am also checking with ya and making sure that ideas are pretty sound. But it also makes me think of new ways to handle things. There is one issue I am trying to tackle and that is I am unable to set the size of the event log. Meaning if I use a parameter to expand the thing. It will only become so tall. Width works perfect. But the height stinks. And it gives no concession to doing a resize on it either. But the ultimate list ctrl does give the ability to set custom cell renderers which is the way I think I will have to go with this. I will also have to use a custom header renderer because the icon doesn't show in the stock renderer. So there will be some work to do on that end of it. If it's something you want to tackle I can pass you over what I have so far. And because the ultimate list ctrl is not written in c it's easy to get the source to browse through that is what I have been doing.

I can key up an outline of what my brain is thinking . And if interested you can key of the GUI end of things and I can go back to hammering away at the core event code.

[ThomasBott]

It's a very interesting topic and I'd love to help with the coding, but unfortunately I’m working on another huge project that’s eating up all my time, so I will not get to do anything for this one the next months.

[kdschlosser]

this is going to take a long while i think to get done up. it's a very complicated idea... I am going to make a branch on my github.. tho the code is not working it will show the idea and a kind of direction. and you can chime in from time to time if an idea comes to mind on how to do something. or if you do get some available time and want to throw some code at it...

[kdschlosser]

ok maybe not as long as i thought... I have almost all of the logging sorted out.. I have a few minor bits left.. but it works.. YAY

and the threading portion is all done.. and works as well. the only crappy thing is how long it takes for the ultimatelistctrl to draw the screen.. but i am also running at 5760 x 1080 resolution with a million windows open. don't think that in any way would have a positive impact on it. but I really don't even need the ultimatelistctrl and I think i am going to remove the thing and go back to the original listctrl. it has also had a sizeable impact on memory. like 3 meg and that's without a fully populated log.

I have yet to add the pieces for the eventList the plugins can supply. and retrofit the AddEventsDialog But it seems to be working with no errors.

[kdschlosser]

ok so i hit a road block. and i need help on this one... maybe someone can think of a way to go about doing this. but here is the problem.

in order to maintain backwards compatibility I am trying to replace the current attributes that deal with the various event data with a class that would return the correct data from the thread that is actually running that specific event.

the means of doing this is pretty simple in design because the data is almost always accessed by the same thread. but is accessed from all kinds of different spots in code. even from of a plugin it's self.

as an example. eg.programCounter is used to house the next macro elements to be run. this can be changed at any time during the running of a macro. the key is we want to populate it correctly. so we would use threading.currentThread() to identify which set of data to change or return

now that all works fine and dandy. but in order for this to work I have to declare the attribute programCounter at the class level of eg. but because of eg's DynamicModule which is really "eg" this should be a non issue. but because the DynamicModule when it is constructed overwrites it's own dict with the one from the module it is contained in (init) it deletes the instances of the declared attributes. and there is no way to set these attributes in any other manner then to do it at the class level that I have found. otherwise it just doesn't work. I am not sure of the reasoning for setting the instances dict as the modules dict I am thinking it has something to do with the import mechanism and how it obtains imports from the normal hierarchy.

[ThomasBott]

Hmm, not sure if this will work in this case, but can you maybe use vars() to create a new eg object at class level?

[kdschlosser]

the problem is the DynamicModule "eg" changes it's dict out to the module level one. and once that is done it wipes the existence of the descriptor class and you can't add it after the fact. it has to be done when the module is first constructed. But I have found a way... i really dislike it but it works.

here is some of the code for init but you will see what has to be done when the DynamicModule gets constructed

from Classes.Events.EventProgramReturnStack import EventProgramReturnStack
from Classes.Events.EventProgramCounter import EventProgramCounter
from Classes.Events.EventString import EventString
from Classes.Events.EventResult import EventResult
from Classes.Events.EventIndent import EventIndent
from Classes.Events.EventObject import EventObject


class DynamicModule(object):
    programCounter = EventProgramCounter()
    programReturnStack = EventProgramReturnStack()
    event = EventObject()
    result = EventResult()
    eventString = EventString()
    indent = EventIndent()

    def __init__(self):
        mod = sys.modules[__name__]
        self.__dict__['__path__'] = mod.__path__
        self.__dict__['__package__'] = mod.__package__
        self.__dict__['__file__'] = mod.__file__
        self.__dict__['__name__'] = mod.__name__
        self.__dict__['__original_module__'] = mod
        sys.modules['eg.Utils'].eg = self

        sys.modules['eg.Classes.Events.EventProgramReturnStack'].eg = self
        sys.modules['eg.Classes.Events.EventProgramCounter'].eg = self
        sys.modules['eg.Classes.Events.EventString'].eg = self
        sys.modules['eg.Classes.Events.EventResult'].eg = self
        sys.modules['eg.Classes.Events.EventIndent'].eg = self
        sys.modules['eg.Classes.Events.EventObject'].eg = self

        sys.modules[__name__] = self

        import __builtin__
        __builtin__.eg = self

    def __getattr__(self, name):
        if name in self.__dict__:
            return self.__dict__[name]

        try:
            mod = __import__("eg.Classes." + name, None, None, [name], 0)
            self.__dict__[name] = attr = getattr(mod, name)
            return attr
        except (ImportError, AttributeError) as err:
            if hasattr(self.__dict__['__original_module__'], name):
                return getattr(self.__dict__['__original_module__'], name)
            raise err

by doing the above I am able to set the descriptor classes into place at the class level of DynamicModule and grab the needed entries from module.dict and set them into DynamicModule.dict and this keeps the lazy import mechanism working.

Its funny tho because all the docs state is that the descriptor class has to be in DynamicModule.dict but that is not 100% true, because if i initialize the class at the module level and then let DynamicModule do it's thing and overwrite DynamicModule.dict with module.dict like it originally does that brings the initialized descriptors into the class but they don't work if I do that. I believe it's a bug. I am not sure tho.

[kdschlosser]

well I finally have done it. I got past this headache of a problem i was having with dynamically passing specific attributes based on what thread was asking for it.

This is sweet. I have some more tinkering with the log to do. but other then that it runs without error so far. and the API is 100% in tact. So unless some plugin does something really goofy with overriding some core piece of the event system this should work just fine with the plugins.

Funny tho. the solution was staring right at me and ended up being 1/2 the amount of code.

[soraxas]

Hi! The idea of using separate threads to handle events is very intriguing and would enhance the responsiveness of running tasks (especially if there are wait command within the actions).
Are there progress on this? I tried to install the latest from the master branch but couldn't see anything similar to that.

[kdschlosser]

I have a running version for the multiple threads. I am still working on how to display the logging information in a way that can be understood. This part is the complex bit because there can be n threads running at the same time. Each of those threads is going to log information. The trick is keeping them from getting all jumbled up. n threads is the number of threads the system is able to run based on how much ram it has. Windows allocates at least 1 megabyte of ram per thread. With Eventghost and what it is doing it will typically not use more then that. So on a system where you have 16GB of ram and say 8GB of that is free then there is a maximum of 8000 ish threads tht can all run at the same time...

Imagine that happening and 8000 threads writing logging information. It is easy to get it all out of sorts.

The other challenge is not breaking the API in EG. I personally do not want to have to go and update almost 400 plugins. This would really suck.

The other thing is performance. just because there are more then one thread processing information does not make it faster then havng a single thread do the same work. This is because of shared data access and thread locks. You cannot have 2 threads writing information or a thread reading and another writing to the same data container at the same time. You will end up with data corruption. so a lock needs to be put into place. The lock will stop all other threads from accessing the container while one thread is writing information to it. Locks are expensive to use and the more of them you have the slower and slower the thread becomes. Until the point where you are actually going backwards in terms of performance. This is an interesting problem because in order to keep the API there is going to have to be a performance penalty for doing it.

Everything has to be weighed and problems MUST be sorted out. The way EventGhost is designed it is not a single thread that handles an event... You have the "event" thread that manages event creation and location of the macros that need to run. Then you have the "action" thread which is what runs the actions in a macro. The event thread sits and waits until the action thread is done running an action. There is a very explicit purpose to the action thread that I am not going to get into, it's very long winded. e. Then last but not least the "main" thread, anything that changes in the GUI must be done from the main thread. There is no way round this as it is the design of the GUI framework that is used. I have never looked into why it is done this way, I am pretty sure that it is not a limitation in Window. The GUI framework is cross platform so it could be because one of the other OS's the framework supports. This is a huge bottleneck when dealing with anything GUI related.

My point is, it is a complex thing to do and to have run properly while keeping the code understandable. I have written the thing over probably 10 times now, each time it gets easier to understand and also requires less code changes. I have not gotten to the point where I am happy with the code and how it is done. I have an idea and a plan. I execute it and bug test. by time I am done correcting the bugs the code ends up hard to follow.

[ThomasBott]

Hey Kevin, I have an idea when it comes to performance and logging:
Performance: How about a parameter in the EG options to define a thread limit? Just an integer that defaults to 1 (which means single threaded, so as it is at the moment). A user MUST specify a limit, I wouldn't allow the use of unlimited threads.
Logging: It has been a while I looked into it, but I think the log is an array filled with objects for events, actions, macros etc. like: [{event},{macro},{action},{event}]
In a multithreaded scenario we could add another level to this array; the first element of that array is always an event and the following elements are macros and actions that belong to that event. Everything that's displayed in the log but is not related to an event (like notifications from threads in plugins) would still go to the main array. like: [[{event},{macro},{action}],{info},[{event}],[{event},{macro},{action},{macro},{action}]].
To do that, the thread that is created needs to know the index of the array it should append it's messages to. In a nutshell that mean, events are displayed always in the order they were created, but the actions are displayed in the order they are created relative to the events. That would of course mean your log not only grows at the end, but also in the middle as actions could be still appended to "old" events. However, the timestamp will show when they were created and as the user has to enable multithreading in the first place for it to happen, this should be fine.

The thing I'm not sure with in this design is the recycling of the log, as "long term" tasks may want to append to an recycled item :/
Also the index of the array will change when elements are removed from it, that needs to be covered.
As for plugins: The only plugin that, I guess, would need to be fixed is the log redirect plugin

What do you think?

[kdschlosser]

@ThomasBott

you know this stuff.. Think about it. there is 0 difference if you have 10 threads running or 8000 the same locks would still need to be in place so the actual number of threads is not causing the performance impact. It is the thread locks and accessing the data that do.

and the logging is not a huge deal handling the storage of the data.. the actual event it's self can hold that information. The issue is when they run for a long while so instead of having a single log queue you either end up with a multidimensional array or you end up with multiple arrays. It's the same thing. More memory consumption.

My issue is how to properly display the logging in EG for the user to see. that is the tricky part.

Here is what I am going to do. I will give it a go and write the damned thing again. Each time it gets better and better. I have the bulk of it down to about 300 -400 lines of code without getting into the intricate logging BS.

The single largest issue is we are going to have to deep six the action thread. no point in processing multiple events at the same time to just have them dump the actions into the action thread to run. the action thread is going to get backed up and it is going to cause slow downs.

This is the really hard part and it's going to require having to recode quite a few of the plugins. any time you deal with say a COM object or with the windows notification system. This all has to be done from the same thread. so this is what takes place..

EG starts.. the action thread starts.. the event thread starts..
then the autostart group is told to run (the autostart group is actually a macro) all of the items contained within it get loaded into the action thread to run. this includes calling the plugins start method. this is when a plugin will typically set up COM objects or register for notifications crap like that. This is OK to do because normally because when those things get accessed it gets done from... An action in the plugin. and that action gets run from the exact same thread that started the plugin and hence you end up with no issues with COM objects and the like.

So if i do away with the action thread, which is what needs to be done in order to make it all worth it then plugins are going to get started in a thread that is different from the thread an action gets run in and all kinds of phantom issues are going to arise because the developers at Microsoft felt that there should be no need to raise some kind of an error when creating and accessing from different threads. Maybe thy thought it would be amusing to see a user go bananas trying to figure out what in the hell is happening because things are going bananas that really seem to not be related to the cause of the problem.

The logging issue is I would have to write a custom wxPython widget to handle the displaying of the log. I would need it so that the "main" log only shows the events that have transpired and by clicking on an arrow icon or a plus icon like a tree control then the log for that event would expand. It gets really complicated because you are dealing with multiple sets of logs that have to scroll properly when the user scrolls them but also when things get put into the log. so imagine a line in the current log being expanded and a new event comes in and also an event that matches the log item you have expanded. Bot like ould have to scroll at the same time. the main log would have to move the expanded log up one notch while the expanded log was scrolling... Have you ever watched the credits at the end of an Animation movie. like Shrek.. they have the CGI outtakes. I can imagine something like that kind of stuff taking place until the bugs get sorted out if it.

I am not saying it can't be done. It is going to be some hair pulling to get it to work right. The next issue is the rendering speed, because it is going to have to be custom drawn there is going to be quite a bit of cost for doing that. Python is not a fast language. it is some 50 times slower then C code. so what would take 1 second to run in C it is going to take 50 seconds to accomplish the same task in Python. GUI's require some pretty heavy lifting in the math department and thus the operations take some time. There is a way for me to be able to get python code to run as fast as C code. That is going to require me to rewrite a substantial part of the build system to do it. The resulting file(s) for the new log control would be a DLL and would not be editable without having to recompile EG.

@ThomasBott you are the one that made me very aware of resource use when you objected to the deletion of the code that destroys the EG GUI when you minimize it to the tray. while the impact is small in memory consumption on todays PC's when EG was written a single meg of memory was important when you only had 256 megs in the machine. We have to also keep that thought going with the threads. creating new threads all the time slows things down as well but leaving the thread idle consume resources when they are not being used. so how do we solve this problem? I came up with a system that creates a thread when an event comes in (if the thread is not already running) it does it's thing and then waits 3 seconds before terminating. I do this because of rapid incoming repeat events from say a remote of some kind, or the system volume event. This is what I came up with to get the best of both.

I have combed over this many times. It works. I can make it run. either we are going to have to break API and change plugins in order to make it "blazing saddles" fast and have a small footprint in the resources department. or we keep as much of the API in tact as possible but the cost of doing so is going to be not so optimal performance and higher resource footprint. Both ways are going to run a heap faster then what EG does currently. But I simply do not have the answer as to which was we should go about doing it.

[ThomasBott]

Thinking about my last post... Events that are still processed could be highlighted in the log, so you can easily see what is still "interesting" to look at. Also a count somewhere how many threads are currently active would be nice.
As for the problems in my last post: A dict could be created containing all the IDs and indices of the running event threads and whenever a thread finishes, the dict entry is deleted.
For the log recycling: Just don't throw away events in the log that are still running :D
And whenever the log gets recycled, the dict that is holding the indices of the running events in the log array is just recreated.

@kdschlosser:
Less threads means less potential thread locks. Also having 8000 threads running the same time there is clearly something wrong with your setup.. A thread limit could prevent "out of memory" issues, as creating a thread consumers memory. So if there are already 100 threads stuck, why creating the 101ths? That's what I mean by limiting the threads ;)
If a thread limit would exist, I'd maybe set it to 4 in my environment.. Being able to set it to 1 (or disable multithreading) will give users the same experience as they have now, as many doesn't seem to need multiple threads anyways.

As the the data storage: yeah, I know, but that's actually what I mean; making the log a multi dimensional array and display it the same way as the array structure is in the log window as it is now... (So just one list) Maybe my explanations are not very good here, sorry about that ^^°
This would be the KISS approach I guess, no need for foldable lists that need to scroll in sync and it should work with the current UI framework.

I'd keep the API and also the UI as close as possible to the one we have right now. I'm not entirely sure what happens and how the woodoo in the action thread works too, so when you say this has to be looked into and redesigned or it doesn't make any sense, well, I believe you, but as you said, would be better to keep it and not break things :)
(3 years ago you said you managed to do it without breaking the API at all ;) )

Anyways, I know you already put lots of thoughts into this topic and I'm looking forward to that feature. I'm excited what you will come up with :)

[kdschlosser]

if you go to my personal eventghost repository and sift through the branches I am sure you will be able to locate a copy of the threaded events. There is probably more then a single branch as I have coded it on more then a single occasion.

[ThomasBott]

Ok, I'll check it out - but you did get my idea about the log now right? (I just edited my post again) Just one list, keep it simple mate :)

[kdschlosser]

Yeah I understand what you are saying.. But think about it this way. iterations are expensive in python. the less we have to do the better off we are. Even doing something as simple as

blah = {'key': 'value'}
if 'key' in blah:
   etc...

causes an iteration over the dict to take place... I'll be it this does happen in c code but it is still an iteration none the less.

Now.. what does EG's whole design operate around? Events.

why not make the events stick around? we can do this. It actually has a plethora of benefits. The first being history.. another is the number of times it runs. This is a biggie.. you know that dialog that opens up when you add an event and is always empty??? well it wouldn't be empty anymore.

Because you do not want to have a clashing from the possibility of running the same macro at the same time from the same event we can only have a single thread per event. any duplicate events that come in will add 1 to the loop counter if the event already has a thread running. when this loop counter gets to 0 then the thread waits 3 seconds and will exit so long as another duplicate event does not come in.

The event class ends up being the thing that everything else is going to work around. It is going to store the results (eg.results) the payload the string etc.... why not use a simple list to stash the log data for that event in the actual event? no need to monkey about with dictionaries or nested lists or anything like that. That same event can also be in charge of displaying it's own log information and it can also be used to keep track of it's position in the "main" log.

There is another benefit to going this route. EG knows what events are most important to the user.. It's the events they have added to their tree. The event class can get constructed when EG starts and also when a user adds an event to their tree. This is going to aide in keeping the speed up. if an event comes in that has not be assigned to a macro it will construct the class instance real time, the little smidge of extra time it is going to take to construct the class we are not concerned with because e really do not care about performance for events that are not added to a macro.

This is also going to help out. the event thread in it's entirety is going to go bye bye. There will be no need to have it. the plugin when triggering an event is going to handle the creation of the event class instance if one does not exist or the starting of the thread if one does exist or incrementing the loop counter if one exists and it is already running. This deligates that task across however many threads are running for the plugins.

Technically speaking the action thread is no longer needed. I think it should be re tasked. I can perform some hocus pocus with a plugin and it's actions by use of a decorator that will add a metaclass but that metaclass is going to rework how the __start__ and __call__ gets called. It will push the running of them into the action thread.

what about an option that when set will destroy a non macro'd event after it is done running. instead of caching it which would consume a small amount of resources.

I just had a wild idea. I m going to have to test it out and see if I can do it. I need to be able to subclass threading.Thread and also be able to restart the bugger.

[kdschlosser]

OK so that is pimp daddy!!..
what I just tested is subclassing threading.Thread and being able to restart the thread after it terminates. technically speaking I do not restart it I recreate it in a manner that allows me to keep the current class instance in place.

Let me explain the purpose to this.

In EG you have variables like eg.result and eg.lastFoundWindows plus a bunch more. These variables are event specific meaning the information within them can change based on what is being carried out. so how do we keep these variable returning the data that only applies to a specific event and how to do it without using any locks or lookup tables to store the data..

This is how this is going to work.

import threading
import time


class Test(threading.Thread):

    def run_test(self):
        print 'Thread function called'

    def run(self):
        do()


def do():
    t = threading.current_thread()
    t.run_test()


test = Test()
test.start()

if you run the above code you will understand what I am getting at.

the output is going to be the method run_test getting called when do get called form the run method in the Test class. This is because the class instance is what gets returned when threading.current_thread gets called from inside of the do function.

When we try to call do from the main thread a traceback ensues because the thread instance returned by threading.current_thread() does not have the method run_test

so for those variables like eg.result i change them to a get/set property

class EG(object):

    @property
    def result(self):
        thread = threading.current_thread()
        return thread.result

    @result.setter
    def result(self, value):
        thread = threading.current_thread()
        thread.result = value

NO LOCKS REQUIRED!!!!!!! and the reason why no locks are required is because these variables can only be accessed from the event (thread) that holds them. because a single thread cannot be 2 places at once it is impossible to modify the data and also collect it at the same time. because we are doing away with the action thread and the actions would be running inside of the same thread as the event an action/macro is only going to be able to collect the data for the event that the action is being run from.

This still leaves the issue with the COM objects and crap like that but it really shouldn't be that much of an issue because the action thread is only going to be able to run one action at a time I queue the thread that made the call to add the action to be processed. the action thread would have properties that match the variables that the event has and when an action makes a call to get the data from eg, it will get redirected to the action thread, and the action thread will then redirect it to the thread that queued the action to be run.

There is the solution for that problem. and the decorator for the plugin class which will cause the plugin to startup in the action thread and any actions that get called to be run in the action thread. This solves the COM problems with a minimial code footprint.

This is to easy. I have to be missing something. If this works without having to use thread locks or worry about data corruption I will be really happy. This is the solution I have waiting for my brain to come up with. Now I am going to start questioning if it would work because of how easy it is. This should literally take me < hour to get coded up, that is without handling the logging aspect.

[kdschlosser]

OK I am not fully done yet. But this is the workhorse portion of the threaded events code.
kdschlosser@9cafdf5

I still have to eliminate the event thread and re-task the action thread. but for the most part the code that is needed to handle the actual delegation of the threads is all in that commit. This is the smallest version I have written thus far coming in at 505 lines of code. And the API is still 100% in tact!. I could make the code footprint even smaller but it would not be easily read, code completion and intellisence in an IDE would not work either. I didn't have to have all of the properties explicitly typed out like you see being done. I can actually remove some of them and go directly to the container I made to hold the event data instead of bouncing off of the EventGhostEvent class. There is also a large amount of sanity checks that probably don't need to be there really.

[ThomasBott]

Sounds awesome! :)
I was actually only thinking about multithreading but your improvement goes way beyond that, I like it! Storing the data thread-local instead of class-local really makes sense if you re-use the classes instead of throwing them - I hope that solves the problems you had with the locks and event processing once and for all :)

About the eventlog.. when you store the log information with the event this is indeed more complicated to display than a multidimensional array. I see your point with history (even though history is a memory eater and not so useful unless it's persistent) but I still think a multidimensional array wouldn't be too bad for the log. Yes, there would be iterations, but you can keep them to a minimum, you don't need to interate over the whole list all the time. And keep in mind that lots of object cells are also expensive ;)
You know, I just think a multidimensional array will be more easy for the GUI and as you know, GUI what shows things down the most...

[kdschlosser]

what I was getting at with the logs is that I can use the same mechanism that I am using for the event attributes in EG to "stash" the log information for a specific event in the event class. because the event class is the thread class and because I managed to make that class persist even tho I am restarting the thread storing the log data in the event class of the running event actually is going to make the logging easier to deal with. I can add some kind of an expandable/collapsible view for the event logs..

So in the main log all you would see are the events and nothing more. if there is no entry in the log for an event it will make one. If there is an entry in the log it will move it to the bottom of the main log view. if you expand the log entry for an event it will then show all of the logging information that has taken place for that event. this would be complex to do because of trying to squeeze that much information into that single logging window. maybe the thing to do would be to use a tree control in combination with the main log window. The main log window will only show the events coming in and not the actions or macros. Those things could be selected from the tree control if they are wanting to be viewed. The viewing would take place in a different pane or optionally there could be a pane for each of the events that you want to view the log for.
That adds complexity for the user. I am trying to keep it simple for the user.

I can also stash all of the logging information that happens while an event is running and spit it all out in one go. This would keep the order in tact. But if you have a bunch of events all running at the same time and ending about the same time the log is going to fly at warp speed.

I have to explain to you at some point the purpose to the InstanceSingleton class and what it does, how it works and the possible uses for it.

[kdschlosser]

I also wanted you to take a look at the MCE Vista plugin and what I am doing with it.

https://github.com/kdschlosser/EventGhost/tree/mce_ir_vista/plugins/MceRemote_Vista

[ThomasBott]

@kdschlosser Yeah, I understood your plan with the eventlog and from a technical point of view it's quite good - object orientated, preventing redundantcies and so on, but from the usability this is not exactly ideal.. A multidimensional array type of log would be more easy to implement and be the way to make the least UI changes, so EG veterans would understand it instantly, but it's also not ideal of course and maybe it's really time for change here.

Let me too a little into details: Your proposed solution displays only events in the main log - ok, you can open these events somehow to see what they triggered - also ok (just make this open in a new window or something, not within the main log window, that would really be nightmare when it comes to creating it and also UI performance. Also your events in the main thread should somehow indicate that there are actions executed by this events without the need to open the log for that event). Now here comes the thing I see a problem with: You plan to shift the most recently triggered event to the bottom of the list. This is really a bad idea, because the log should show what happened in what order - also in past, so we really need these "redundant" event entries in the log. But even if you say now "Ok, let's make a new reference at the bottom of the list and keep the old one where it was" there is still the problem that you don't see what was executed in which order without opening hundreds of event logs and comparing the timestamps of the actions. That's very inconvenient compared to how it was before.

I have a new idea: How about let's split the log into two. So we have one log with all the events in order of their appearance and another one with all the actions in order of their execution time (just actions, no macros here). You could still open your events, as you planned it, to see the macros and actions that have been executed by the event. In addition there is the action log showing executed actions independently of the events or macros that triggered it. The actions in that log you could right-click to "show originating event" or "show originating macro" which would make the view in the appropriate pane jump to the desired item.

How about that? :)

About the IR plugin, I'll check out, I'm very curious about it! But I'll need some days for that, as I actually moved away from the MCE remote (because of the crashing of the service, to be honest) and I need to find it again (don't know where I put it ^^°)
If the plugin is now as awesome as I hope it is, I think I will go back to it instead of using LIRC (what I do right now)