Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

100% CPU Usage 😢 #588

Open
rahulraj80 opened this issue Jul 22, 2023 · 8 comments
Open

100% CPU Usage 😢 #588

rahulraj80 opened this issue Jul 22, 2023 · 8 comments

Comments

@rahulraj80
Copy link

I am getting 100% CPU usage because of ET : Can someone help and guide?

I had installed et v6.2.4 on an embedded debian setup (Storage device) :

root@wdmc:~# et --version
et version 6.2.4
root@wdmc:~# uname -a
Linux wdmc 4.14.4 #14 SMP Wed Dec 20 01:09:34 +05 2017 armv7l GNU/Linux

It works well and keeps the connection etc, for the reverse tunnel that I had setup to monitor the server using another on a remote, but accessible machine. After some time - the CPU usage goes up to 100% and stays there forever. This happens within 5 minutes of the connection being triggered.

image
etclientLogs.tar.gz
etserverLogs.tar.gz

Line in crontab which triggers the ET connection - I have started it from terminal and also without the -x option, and that gives me the same results:
#@reboot 0 * * * * /usr/bin/et -x xxxxxx@yyyyyyyy:pppp -r 8xxx:10xxxx &

While retrieving the logs:
This commands fails as I don't have these log files:
tar -cvzPhf /tmp/etclientLogs.tar.gz /tmp/etclient_err /tmp/etclient.INFO

But I had these which I have attached:
tar -cvzPhf /tmp/etclientLogs.tar.gz /tmp/etclient-2023-07-22_14-20-02.log /tmp/etclient-stderr-2023-07-22_14-20-02.log

Similarly this failed for the server as I did not have these files:
tar -cvzPhf /tmp/etserverLogs.tar.gz /tmp/etserver_err /tmp/etserver.INFO

But I had these - attached:
sudo tar -cvzPhf /tmp/etserverLogs.tar.gz /tmp/etserver-2023-07-20_11-33-10_2532693.log /tmp/etserver-stderr-2023-07-20_11-33-10_2532693.log

The etclient log file is flooded with "Got data from stdin" - Not sure if it is a bug.

Tail of the log file without "Got data" lines

root@wdmc:~# cat /tmp/etclient-2023-07-22_14-20-02.log | grep -v "Got data" | tail
[V4 2023-07-22 14:51:10,259 client-main UnixSocketHandler.cpp:23] socket 10 has data
[V4 2023-07-22 14:51:10,259 client-main UnixSocketHandler.cpp:45] Unixsocket handler read from fd: 10
[V2 2023-07-22 14:51:10,259 client-main BackedReader.cpp:68] Reading message of length: 95
[V2 2023-07-22 14:51:10,259 client-main BackedReader.cpp:71] bytes remaining: 95
[V4 2023-07-22 14:51:10,259 client-main UnixSocketHandler.cpp:23] socket 10 has data
[V4 2023-07-22 14:51:10,259 client-main UnixSocketHandler.cpp:45] Unixsocket handler read from fd: 10
[V3 2023-07-22 14:51:10,259 client-main TerminalClient.cpp:280] Got terminal buffer
[INFO 2023-07-22 14:51:10,260 client-main TerminalClient.cpp:326] Window size changed: row: 60638 column: 25787 width: 45510 height: 3856
[V2 2023-07-22 14:51:10,260 client-main BackedWriter.cpp:50] Message length with header: 37
[V4 2023-07-22 14:51:10,260 client-main UnixSocketHandler.cpp:61] Unixsocket handler write to fd: 10

The log file has 128k lines in total:

root@wdmc:~# cat /tmp/etclient-2023-07-22_14-20-02.log | wc
 128004 1159042 10840264

Of these, 118k lines were the "Got data from stdin lines" - and 9.5k other lines:

root@wdmc:~# cat /tmp/etclient-2023-07-22_14-20-02.log | grep -v "Got data" | wc
   9543   92893  889540

The 'Got data line looks like this:

root@wdmc:~# tail -2 /tmp/etclient-2023-07-22_14-20-02.log
[V4 2023-07-22 14:51:10,261 client-main TerminalClient.cpp:210] Got data from stdin
[V4 2023-07-22 14:51:10,261 client-main TerminalClient.cpp:210] Got data from stdin
@MisterTea
Copy link
Owner

MisterTea commented Jul 24, 2023 via email

@rahulraj80
Copy link
Author

The conf file was not changed.

; et.cfg : Config file for Eternal Terminal
;

[Networking]
port = 2022
# bind_ip = 0.0.0.0

[Debug]
verbose = 0
silent = 0
logsize = 20971520
telemetry = true
logdirectory = /tmp

@rahulraj80
Copy link
Author

I found a ubuntu/focal based firmware instead of the jessie/stretch one. Let me see if that was the reason for my issue. Will upgrade OS and report.

@rahulraj80
Copy link
Author

Sorry for the delay: I ran into a ton of issues with my embedded system due to a wrong flash. Fixed it after a ton of reading and effort.

Update:
On the new OS as well, I have the same issue. The et executable starts using up 100% of the CPU a few minutes after being activated, and then stays at 100% until killed. It also overheats the device enclosure and I can hear the fan working at a higher speed.

I see no errors in the error logs, but if I change verbosity to 1 in et.cfg, I see these two lines repeated continuously.

[INFO 2023-08-03 17:26:38,830 client-main TerminalClient.cpp:326] Window size changed: row: 28811 column: 46730 width: 14152 height: 46849
[INFO 2023-08-03 17:26:38,830 client-main TerminalClient.cpp:326] Window size changed: row: 28811 column: 46730 width: 50832 height: 46848
[INFO 2023-08-03 17:26:38,899 client-main TerminalClient.cpp:326] Window size changed: row: 28811 column: 46730 width: 14152 height: 46849
[INFO 2023-08-03 17:26:38,900 client-main TerminalClient.cpp:326] Window size changed: row: 28811 column: 46730 width: 50832 height: 46848

What can I do to fix it please?

@MisterTea
Copy link
Owner

MisterTea commented Aug 4, 2023 •
edited

Those prints are saying that your window size is changing. The window size is measured in CHARACTERS, so a window size of 50k by 46k is bonkers and will definitely blow up your CPU. You need to find out why the window size is so large.

@rahulraj80
Copy link
Author

Those prints are saying that your window size is changing. The window size is measured in CHARACTERS, so a window size of 50k by 46k is bonkers and will definitely blow up your CPU. You need to find out why the window size is so large.

So - this is running from a cron job, and what it does is setup a reverse tunnel to forward a port (say) A from the device to a remote server on port B. It is a non-standard port, so unless I visit the https://server:B there should be no activity.

Where does the window thing come (and its size)? Also, the size is alternating for some reason. Are there two sessions running simultaneously? I am using the -x option which I understood from the docs that it will kill the other sessions, so there should be only one active at any time.

@MisterTea
Copy link
Owner

How often does the job run? The window size is based on the size of the shell that called et. If it's done non-interactively, maybe the size is arbitrary?

@rahulraj80
Copy link
Author

It is run from cron. This is the line in my crontab :

#@reboot 0 * * * * /usr/bin/et -x xxxxxx@yyyyyyyy:pppp -r 8xxx:10xxxx &
0 * * * * /usr/bin/et -x xxxxxx@yyyyyyyy:pppp -r 8xxx:10xxxx &

I am not sure how window size is defined when a job is run by the cron daemon.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@MisterTea @rahulraj80