No activity history reported from clients

[tm6907]

From switching to v7 where I had the same issue. Any history timeline or history health data isn't being recorded anymore on Grafana or in the PostGres DB. Everything should be connected networking wise, so maybe its the application or something I've missed in the config.

The client is routinely checking in for updates and the dashboard displays this but doesn't show any of the activity the client is performing. Below is all the relevant images that might be of help. It seems data is being sent but not collected or stored.

^ Client running ghosts.exe in debug mode ^ ^ Client app.log ^ ^ clientupdates.log ^ ^ Grafana dash showing the client is checking in ^ ^ Dash showing the client is still polling but no health checks ^ ^ Postgres container showing no history data ^ ^ Postgres container logs ^ ^ Api container logs ^ ^ Grafana container logs ^

[capstonewazuhelk]

I have same issue. This issue because of the Date Time.

I try to change the date manually, and it works, the activity number increase.

It cant handle the first row Timeline Date 23/04/2024

I hope it can be solve soon #323 . Thank you @sei-dupdyke.

[tm6907]

Ok I did what you suggested, I waited until clientupdates.log appeared quickly changed the date format and that one bit of activity got through and showed on the grafana dashboard.

What I don't get is if there's always been a difference in formatting why was it working once before ? and the same issue occurs on v7 and v8. If this is the cause of the issue than surely no one can get Ghosts working right now.

[sei-dupdyke]

I highly doubt the difference in dates is this non-reporting issue, I haven't touched the formats of the dates for ages (ever?) and it has always been the way that it is.

I will look into it as soon as I can, I am currently at 100% load.

[capstonewazuhelk]

Previously, my system was functioning correctly, when the day/dd is not more than 12 . I believe the issue arises when the day (denoted as ‘dd’) exceeds 12, such as 13, 14, and so on.

For the first row with the date ‘23/04/2024’, the system can’t convert it to datetime because there is no 23rd month. For the fourth row with the date ‘12/04/2024’, it can be added to the PostgreSQL database, but with 12 being interpreted as the month and 04 as the day. Maybe, because the format date is MM/dd/YYYY, thats why there is an error like below picture "22/04/2024 13:27:59 was not recognized as a valid DateTime".

I’m not sure why others aren’t experiencing the same problem as we are.

[sei-dupdyke]

Would you mind sharing some details on your postgres install? Did you use the default with the docker-compose file or something else? Is it US/EN or some other locale by chance?

[capstonewazuhelk]

Yes, I use default docker-compose file and default configuration, from the web https://cmu-sei.github.io/GHOSTS/quickstart/#windows-client, I install it in ubuntu 22.04. How can I check US/EN ?

[sei-dupdyke]

I just posted an update to the windows executables v8.0.10.50 that I confirmed does post to the API without issue. There was a bug of rather stupid origins mixing delay before and after, so while you're waiting for activities to commence, its mixing delay after as something to wait for before execution.

Sorry for the mix-up.

[sei-dupdyke]

Note I did not change dates in the two files. I am still not convinced we need to change one to match the other. The app log doesn't ever show up in the api (it stays entirely local)... 

HOWEVER, I'm wondering if it is beneficial to think about sending those logs to the server as well, because it may make it easier to manage large installations?

[tm6907]

I will try out this update now, will let you know if anything goes wrong. Re: your point about sending over the app.log as well, I don't see why it would hurt, could contribute to a separate graph about machine health info which would be good.

[tm6907]

Ok gave the new version a go. Completely removed that last version and reinstalled the new one but doesn't seem to have helped. Its checking into the API fine and polling but not reporting the Firefox activity its doing. I double checked the version too

[sei-dupdyke]

Ok, let's work through this.

First, the datetime in the clientresults.log is not directly converted by postgres, it is first converted by C# using Convert.ToDateTime, and then inserted into postgres appropriately.

Here is my current postgres verison:

Seems close enough (16.2 vs 16.0) that there would not be some compatibility issue lurking?

Second...

When you start the API, you should see the following. First, app starts up:

Ghosts.Api/bin/Debug/net6.0/ghosts.api.dll
             ('-. .-.               .-')    .-') _     .-')    
            ( OO )  /              ( OO ). (  OO) )   ( OO ).  
  ,----.    ,--. ,--. .-'),-----. (_)---\_)/     '._ (_)---\_) 
 '  .-./-') |  | |  |( OO'  .-.  '/    _ | |'--...__)/    _ |  
 |  |_( O- )|   .|  |/   |  | |  |\  :` `. '--.  .--'\  :` `.  
 |  | .--, \|       |\_) |  |\|  | '..`''.)   |  |    '..`''.) 
(|  | '. (_/|  .-.  |  \ |  | |  |.-._)   \   |  |   .-._)   \ 
 |  '--'  | |  | |  |   `'  '-'  '\       /   |  |   \       / 
  `------'  `--' `--'     `-----'  `-----'    `--'    `-----'  
GHOSTS API 8.0.0.0 (8.0.10.0) coming online...
2024-04-25 07:31:17.2825|INFO|Ghosts.Api.Program|GHOSTS API 8.0.0.0 (8.0.10.0) coming online...
Now listening on: http://0.0.0.0:5000
Application started. Press Ctrl+C to shut down.

Now we should see clients checking in:

2024-04-25 07:31:26.9834|TRACE|Ghosts.Api.Hubs.ClientHub|dev-box f542eddf-a032-4133-b476-24764f1c1223 (RMAjIYITqaUTlIAj9lucGw) - ReceiveId

then they post results:

2024-04-25 07:34:52.0442|TRACE|Ghosts.Api.Controllers.ClientUpdatesController|Request by f542eddf-a032-4133-b476-24764f1c1223
2024-04-25 07:35:28.8269|TRACE|Ghosts.Api.Controllers.ClientResultsController|payload received: TIMELINE|4/25/2024 11:29:59 AM|{"Handler":"Excel","Command":"create","CommandArg":"%homedrive%%homepath%\\Documents"}
TIMELINE|4/25/2024 11:29:59 AM|{"Handler":"Word","Command":"create","CommandArg":"%homedrive%%homepath%\\Documents"}
TIMELINE|4/25/2024 11:30:00 AM|{"Handler":"Word","Command":"create","CommandArg":"pdf"}
TIMELINE|4/25/2024 11:30:00 AM|{"Handler":"Excel","Command":"create","CommandArg":"pdf"}
TIMELINE|4/25/2024 11:30:03 AM|{"Handler":"PowerPoint","Command":"create","CommandArg":"%homedrive%%homepath%\\Documents"}
TIMELINE|4/25/2024 11:30:04 AM|{"Handler":"PowerPoint","Command":"create","CommandArg":"pdf"}
TIMELINE|4/25/2024 11:30:17 AM|{"Handler":"BrowserFirefox","Command":"random","CommandArg":"{\"Uri\":\"http://www.mayoclinic.org/\",\"Category\":null,\"Method\":\"GET\",\"Headers\":null,\"FormValues\":null,\"Body\":null}","Result":""}

Now you have a client that has reported into a queue — the server then processes that queue asynchronously:

2024-04-25 07:36:39.4591|TRACE|ghosts.api.Infrastructure.Services.QueueSyncService|Queue Processed — Machines: 1, Histories: 1, Timeline: 0, Health: 0, Trackables: 0, Webhooks: 0
2024-04-25 07:38:43.9003|TRACE|ghosts.api.Infrastructure.Services.QueueSyncService|TIMELINE|4/25/2024 11:29:59 AM|{"Handler":"Excel","Command":"create","CommandArg":"%homedrive%%homepath%\\Documents"}
TIMELINE|4/25/2024 11:29:59 AM|{"Handler":"Word","Command":"create","CommandArg":"%homedrive%%homepath%\\Documents"}
TIMELINE|4/25/2024 11:30:00 AM|{"Handler":"Word","Command":"create","CommandArg":"pdf"}
TIMELINE|4/25/2024 11:30:00 AM|{"Handler":"Excel","Command":"create","CommandArg":"pdf"}
TIMELINE|4/25/2024 11:30:03 AM|{"Handler":"PowerPoint","Command":"create","CommandArg":"%homedrive%%homepath%\\Documents"}
TIMELINE|4/25/2024 11:30:04 AM|{"Handler":"PowerPoint","Command":"create","CommandArg":"pdf"}
TIMELINE|4/25/2024 11:30:17 AM|{"Handler":"BrowserFirefox","Command":"random","CommandArg":"{\"Uri\":\"http://www.mayoclinic.org/\",\"Category\":null,\"Method\":\"GET\",\"Headers\":null,\"FormValues\":null,\"Body\":null}","Result":""}
2024-04-25 07:38:44.0537|TRACE|ghosts.api.Infrastructure.Services.QueueSyncService|Queue Processed — Machines: 1, Histories: 1, Timeline: 8, Health: 0, Trackables: 0, Webhooks: 0

In all of the screenshots I see posted in this thread, none have this flow indicated in the server logs, but it could be the case that just wasn't posted up here. In the server logs are any of you seeing this flow?

[sei-dupdyke]

I see the issue now...

The client is logging:

TIMELINE|25/04/2024...

written via:

_timelineLog.Info($"TIMELINE|{DateTime.UtcNow}|{o}");

But this writes in the local system format, not always US, and the server assumes always US. Let me see how to correct this!

[tm6907]

Ah yeah Im in the UK, might be the issue. Weird that it wasn't a problem before

[tm6907]

Ok changed the region to US and the datas started flowing in now finally. Doesn't look pretty but maybe if I leave it running

[sei-dupdyke]

I just posted up v8.0.11.5000 (as the default client download) to write the date in US format for now. I'll probably need to look into this further to deal with internationalization properly unless you all want to switch to inches and pounds too?

[tm6907]

Cool thanks for the help, think i'll do a clean setup on the new version

[sei-dupdyke]

Should just need to swap out the client. In fact, if you're recent, probably just update the .exe and it will work. Please LMK how it works out.

[capstonewazuhelk]

It works now, only change the .exe. Thank you so much @sei-dupdyke.