Skip to content

Lifailon/shell-metrics

BranchesTags

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

65 Commits

screen

screen

 
 

scripts

scripts

 
 

service

service

 
 

README.md

README.md

 
 

logrotate_metrics.conf

logrotate_metrics.conf

 
 

Repository files navigation

Content:

dir-monitor-log

Directory monitoring for get metric to file count, size and modify. When the number of files is changed, a diff of changes is logging.

Service path: /etc/systemd/system/dir-monitor-log.service
Script path: /root/dir-monitor-log.sh
Script variables:
path_mon - target directory for monitoring (example: /var/lib/jenkins)
path_log - path to log file (example: /var/log/metrics/dir-monitor.log)

Example:

root@devops-01:~# systemctl daemon-reload
root@devops-01:~# systemctl enable dir-monitor-log.service
root@devops-01:~# systemctl start dir-monitor-log.service
root@devops-01:~# touch /var/lib/jenkins/test /var/lib/jenkins/test2
root@devops-01:~# sleep 10
root@devops-01:~# rm /var/lib/jenkins/test /var/lib/jenkins/test2
root@devops-01:~# sleep 10
root@devops-01:~# cat "/var/log/dir-monitor.log" | tail -n 15
4 Sep 01:40:41  Size: 333M  File Count: 12107  Modify: 4 Sep 13:35
4 Sep 01:40:52  Size: 333M  File Count: 12107  Modify: 4 Sep 13:35
4 Sep 01:41:02  Size: 333M  File Count: 12107  Modify: 4 Sep 13:35
4 Sep 01:41:12  Size: 333M  File Count: 12107  Modify: 4 Sep 13:35
4 Sep 01:41:22  Size: 333M  File Count: 12107  Modify: 4 Sep 13:35
4 Sep 01:41:32  Size: 333M  File Count: 12107  Modify: 4 Sep 13:35
4 Sep 01:41:43  Size: 333M  File Count: 12107  Modify: 4 Sep 13:35
4 Sep 01:41:53  Size: 333M  File Count: 12109  Modify: 4 Sep 13:41
+ 0     /var/lib/jenkins/test2
+ 0     /var/lib/jenkins/test
4 Sep 01:42:03  Size: 333M  File Count: 12107  Modify: 4 Sep 13:42
- 0     /var/lib/jenkins/test2
- 0     /var/lib/jenkins/test
4 Sep 01:42:13  Size: 333M  File Count: 12107  Modify: 4 Sep 13:42
4 Sep 01:42:24  Size: 333M  File Count: 12107  Modify: 4 Sep 13:42

top-metrics-log

Collecting metrics from software top for write to the log file and logging of high-load process.

Script path: /root/top-metrics-log.sh
Script variables:
path - path to log file (example: /var/log/metrics/top-metrics.log)
trigger - CPU load percentage (example: 20 %) for logging for logging high-load process

Initialization unit to systemd:
Creat service to path: /etc/systemd/system/top-metrics-log.service
systemctl daemon-reload
systemctl enable top-metrics-log.service
systemctl start top-metrics-log

root@devops-01:~# systemctl status top-metrics-log.service
● top-metrics-log.service - Metrics from software top for write to the log file
     Loaded: loaded (/etc/systemd/system/top-metrics-log.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2023-09-04 09:45:32 MSK; 3h 31min ago
   Main PID: 2023242 (bash)
      Tasks: 2 (limit: 4515)
     Memory: 7.0M
        CPU: 6min 20.318s
     CGroup: /system.slice/top-metrics-log.service
             ├─2023242 /bin/bash /root/top-metrics-log.sh
             └─2404579 sleep 2

Sep 04 13:16:34 devops-01 bash[2404195]: 4 Sep 01:16:34  CPU(user): 14.3 %  CPU(system): 7.1 %  Proc: 1/239  Users: 2  MEM: 1598.5/3876.4 MB>
Sep 04 13:16:34 devops-01 bash[2404219]: Warning:  4 Sep 01:16:34  Top Process: mysqld  Time: 86:30.06  CPU: 12.5 %  MEM: 18.3 MB  User: mys>
Sep 04 13:16:36 devops-01 bash[2404278]: 4 Sep 01:16:36  CPU(user): 7.1 %  CPU(system): 7.1 %  Proc: 1/239  Users: 2  MEM: 1598.7/3876.4 MB >
Sep 04 13:16:39 devops-01 bash[2404334]: 4 Sep 01:16:39  CPU(user): 52.9 %  CPU(system): 5.9 %  Proc: 2/239  Users: 2  MEM: 1598.9/3876.4 MB>
Sep 04 13:16:39 devops-01 bash[2404358]: Warning:  4 Sep 01:16:39  Top Process: java  Time: 5:10.34  CPU: 50.0 %  MEM: 13.3 MB  User: jenkins
Sep 04 13:16:41 devops-01 bash[2404416]: 4 Sep 01:16:41  CPU(user): 58.8 %  CPU(system): 0.0 %  Proc: 2/239  Users: 2  MEM: 1598.9/3876.4 MB>
Sep 04 13:16:41 devops-01 bash[2404440]: Warning:  4 Sep 01:16:41  Top Process: java  Time: 5:11.27  CPU: 56.2 %  MEM: 13.4 MB  User: jenkins
~
~
~
root@devops-01:~# cat /var/log/top-metrics.log
4 Sep 01:16:34  CPU(user): 14.3 %  CPU(system): 7.1 %  Proc: 1/239  Users: 2  MEM: 1598.5/3876.4 MB  SWAP: 0.8/3889.0 MB
Warning:  4 Sep 01:16:34  Top Process: mysqld  Time: 86:30.06  CPU: 12.5 %  MEM: 18.3 MB  User: mysql
4 Sep 01:16:36  CPU(user): 7.1 %  CPU(system): 7.1 %  Proc: 1/239  Users: 2  MEM: 1598.7/3876.4 MB  SWAP: 0.8/3889.0 MB
4 Sep 01:16:39  CPU(user): 52.9 %  CPU(system): 5.9 %  Proc: 2/239  Users: 2  MEM: 1598.9/3876.4 MB  SWAP: 0.8/3889.0 MB
Warning:  4 Sep 01:16:39  Top Process: java  Time: 5:10.34  CPU: 50.0 %  MEM: 13.3 MB  User: jenkins
4 Sep 01:16:41  CPU(user): 58.8 %  CPU(system): 0.0 %  Proc: 2/239  Users: 2  MEM: 1598.9/3876.4 MB  SWAP: 0.8/3889.0 MB
Warning:  4 Sep 01:16:41  Top Process: java  Time: 5:11.27  CPU: 56.2 %  MEM: 13.4 MB  User: jenkins
4 Sep 01:16:43  CPU(user): 60.0 %  CPU(system): 6.7 %  Proc: 1/239  Users: 2  MEM: 1599.3/3876.4 MB  SWAP: 0.8/3889.0 MB
Warning:  4 Sep 01:16:43  Top Process: java  Time: 5:12.71  CPU: 95.0 %  MEM: 13.6 MB  User: jenkins
4 Sep 01:16:46  CPU(user): 6.2 %  CPU(system): 6.2 %  Proc: 1/238  Users: 2  MEM: 1599.3/3876.4 MB  SWAP: 0.8/3889.0 MB
4 Sep 01:16:48  CPU(user): 6.7 %  CPU(system): 6.7 %  Proc: 1/242  Users: 2  MEM: 1599.3/3876.4 MB  SWAP: 0.8/3889.0 MB

iostat-metrics-log

Collection metrics iostat from the set sysstat for write log file

Service path: /etc/systemd/system/iostat-metrics-log.service
Script path: /root/iostat-metrics-log.sh
Script variables:
path - path to log file (example: /var/log/metrics/iostat-metrics.log)

root@devops-01:~# systemctl daemon-reload
root@devops-01:~# systemctl enable iostat-metrics-log.service
root@devops-01:~# systemctl start iostat-metrics-log.service
root@devops-01:~# systemctl status iostat-metrics-log.service
● iostat-metrics-log.service - Directory monitoring for get metric to file count, size and modify
     Loaded: loaded (/etc/systemd/system/iostat-metrics-log.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2023-09-05 12:44:33 MSK; 1min 20s ago
   Main PID: 724914 (bash)
      Tasks: 3 (limit: 4515)
     Memory: 14.7M
        CPU: 977ms
     CGroup: /system.slice/iostat-metrics-log.service
             ├─ 724914 /bin/bash /root/iostat-metrics-log.sh
             ├─ 729062 sleep 5
             └─2644388 iostat -yh /dev/sda 1

Sep 05 12:45:35 devops-01 bash[728116]: 5 Sep 12:45:34  tps = 0  read/s = 0.0k  write/s = 0.0k
Sep 05 12:45:41 devops-01 bash[728450]: 5 Sep 12:45:40  tps = 164  read/s = 0.0k  write/s = 1.4M
Sep 05 12:45:41 devops-01 bash[728465]: 5 Sep 12:45:40  tps = 0  read/s = 0.0k  write/s = 0.0k
Sep 05 12:45:41 devops-01 bash[728480]: 5 Sep 12:45:40  tps = 0  read/s = 0.0k  write/s = 0.0k
Sep 05 12:45:47 devops-01 bash[728776]: 5 Sep 12:45:46  tps = 123  read/s = 0.0k  write/s = 1.3M
Sep 05 12:45:47 devops-01 bash[728791]: 5 Sep 12:45:46  tps = 0  read/s = 0.0k  write/s = 0.0k
Sep 05 12:45:47 devops-01 bash[728806]: 5 Sep 12:45:46  tps = 0  read/s = 0.0k  write/s = 0.0k
Sep 05 12:45:54 devops-01 bash[729031]: 5 Sep 12:45:52  tps = 99  read/s = 0.0k  write/s = 1.2M
Sep 05 12:45:54 devops-01 bash[729046]: 5 Sep 12:45:52  tps = 0  read/s = 0.0k  write/s = 0.0k
Sep 05 12:45:54 devops-01 bash[729061]: 5 Sep 12:45:52  tps = 0  read/s = 0.0k  write/s = 0.0k

logrotate

Automated log file rotation for all metrics: /etc/logrotate.d/logrotate_metrics.conf

Debug:

root@devops-01:~# logrotate -d /etc/logrotate.d/logrotate_metrics.conf
...
rotating pattern: /var/log/metrics/dir-monitor.log  10485760 bytes (2 rotations)
empty log files are rotated, old logs are removed
considering log /var/log/metrics/dir-monitor.log
Creating new state
  Now: 2023-09-05 21:31
  Last rotated at 2023-09-05 21:00
  log does not need rotating (log size is below the 'size' threshold)

rotating pattern: /var/log/metrics/top-metrics.log  10485760 bytes (2 rotations)
empty log files are rotated, old logs are removed
considering log /var/log/metrics/top-metrics.log
Creating new state
  Now: 2023-09-05 21:31
  Last rotated at 2023-09-05 21:00
  log does not need rotating (log size is below the 'size' threshold)

rotating pattern: /var/log/metrics/iostat-metrics.log  10485760 bytes (2 rotations)
empty log files are rotated, old logs are removed
considering log /var/log/metrics/iostat-metrics.log
Creating new state
  Now: 2023-09-05 21:31
  Last rotated at 2023-09-05 21:00
  log does not need rotating (log size is below the 'size' threshold)

iostat-to-influxdb

Service path: /etc/systemd/system/iostat-to-influxdb.service
Script path: /root/iostat-to-influxdb.sh
Script variables:
ip - ip-address server InfluxDB
db - Database name
table - Measurement/Table name

root@devops-01:~# systemctl daemon-reload
root@devops-01:~# systemctl enable iostat-to-influxdb.service
root@devops-01:~# systemctl start iostat-to-influxdb.service
root@devops-01:~# systemctl status iostat-to-influxdb.service
● iostat-to-influxdb.service - Collection metrics iostat from the set sysstat for send to influxdb
     Loaded: loaded (/etc/systemd/system/iostat-to-influxdb.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2023-09-05 12:43:10 MSK; 1h 44min ago
   Main PID: 720656 (bash)
      Tasks: 4 (limit: 4515)
     Memory: 1.0M
        CPU: 1min 22.648s
     CGroup: /system.slice/iostat-to-influxdb.service
             ├─ 720656 /bin/bash /root/iostat-to-influxdb.sh
             ├─1039830 /bin/bash /root/iostat-to-influxdb.sh
             ├─1039831 iostat -ky /dev/sda /dev/sda1 /dev/sda2 /dev/sda3 /dev/sdb /dev/sdb1 /dev/sdc /dev/sdc1 1 1
             └─1039832 grep -w sd.

Sep 05 14:27:30 devops-01 bash[1039591]:                                  Dload  Upload   Total   Spent    Left  Speed
Sep 05 14:27:30 devops-01 bash[1039591]: [158B blob data]
Sep 05 14:27:30 devops-01 bash[1039591]: HTTP/1.1 204 No Content
Sep 05 14:27:30 devops-01 bash[1039591]: Content-Type: application/json
Sep 05 14:27:30 devops-01 bash[1039591]: Request-Id: 33045ca9-4bdf-11ee-bc29-000c294f9f2b
Sep 05 14:27:30 devops-01 bash[1039591]: X-Influxdb-Build: OSS
Sep 05 14:27:30 devops-01 bash[1039591]: X-Influxdb-Version: 1.7.11
Sep 05 14:27:30 devops-01 bash[1039591]: X-Request-Id: 33045ca9-4bdf-11ee-bc29-000c294f9f2b
Sep 05 14:27:30 devops-01 bash[1039591]: Date: Tue, 05 Sep 2023 11:27:30 GMT
Sep 05 14:27:30 devops-01 bash[1039591]:

proc-stats-to-influxdb

Creating metrics for monitoring consumption memory and CPU time process.
Script variables:
process_name - value for Tag process via space (example: "mysqld" "jenkins")

root@devops-01:~# systemctl status proc-stats-to-influxdb
● proc-stats-to-influxdb.service - Process statisctics for send to influxdb
     Loaded: loaded (/etc/systemd/system/proc-stats-to-influxdb.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2023-09-20 11:43:40 MSK; 3min 17s ago
   Main PID: 285471 (bash)
      Tasks: 2 (limit: 4515)
     Memory: 804.0K
        CPU: 8.171s
     CGroup: /system.slice/proc-stats-to-influxdb.service
             ├─285471 /bin/bash /root/proc-stats-to-influxdb.sh
             └─296048 sleep 5

Sep 20 11:46:34 devops-01 bash[285471]: process=mysqld cpu_proc=1.3,mem_proc=9.9,sz_mb=327,rss_mb=387,vsz_mb=1310,threads=37,core=0
Sep 20 11:46:34 devops-01 bash[285471]: process=jenkins cpu_proc=0.3,mem_proc=12.4,sz_mb=785,rss_mb=481,vsz_mb=3140,threads=61,core=0
Sep 20 11:46:40 devops-01 bash[285471]: process=mysqld cpu_proc=1.3,mem_proc=9.9,sz_mb=327,rss_mb=387,vsz_mb=1310,threads=37,core=0
Sep 20 11:46:40 devops-01 bash[285471]: process=jenkins cpu_proc=0.3,mem_proc=12.4,sz_mb=785,rss_mb=481,vsz_mb=3140,threads=61,core=0
Sep 20 11:46:45 devops-01 bash[285471]: process=mysqld cpu_proc=1.3,mem_proc=9.9,sz_mb=327,rss_mb=387,vsz_mb=1310,threads=37,core=0
Sep 20 11:46:45 devops-01 bash[285471]: process=jenkins cpu_proc=0.3,mem_proc=12.4,sz_mb=785,rss_mb=481,vsz_mb=3140,threads=61,core=0
Sep 20 11:46:50 devops-01 bash[285471]: process=mysqld cpu_proc=1.3,mem_proc=9.9,sz_mb=327,rss_mb=387,vsz_mb=1310,threads=37,core=0
Sep 20 11:46:50 devops-01 bash[285471]: process=jenkins cpu_proc=0.3,mem_proc=12.4,sz_mb=785,rss_mb=481,vsz_mb=3140,threads=61,core=0
Sep 20 11:46:56 devops-01 bash[285471]: process=mysqld cpu_proc=1.3,mem_proc=9.9,sz_mb=327,rss_mb=387,vsz_mb=1310,threads=37,core=0
Sep 20 11:46:56 devops-01 bash[285471]: process=jenkins cpu_proc=0.3,mem_proc=12.4,sz_mb=785,rss_mb=481,vsz_mb=3140,threads=61,core=0

InfluxDB

Get data iostat

SELECT * FROM "iostat_metrics_table" WHERE time > now() - 5m

Image alt

Key by name selection:

SELECT * FROM iostat_metrics_table WHERE disk = 'sda' and time > now() - 10m

Image alt

Founction for output of the maximum tps value:

SELECT read_kb_sec,write_kb_sec,MAX(tps) FROM "iostat_metrics_table" GROUP BY instance

Image alt

Get process

Process stats Jenkins when running the pipeline

Image alt

About

Select metrics for outputs to log file and influxdb

Topics

linux shell bash ios devops monitoring influxdb log curl metrics logging scripts units process-monitor cpu-monitoring influx file-monitoring iostat process-monitoring directory-monitoring

Resources

Readme
Activity

Stars

0 stars

Watchers

2 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

No packages published

Languages