Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

log-vm: VM to secure and centralize logs #565

Draft
wants to merge 6 commits into
base: main
Choose a base branch
from
Draft

log-vm: VM to secure and centralize logs #565

wants to merge 6 commits into from

Conversation

vilvo
Copy link
Contributor

@vilvo vilvo commented Apr 16, 2024 •
edited

Concepting to support the design:

  • log-vm scaffolding based on copy of net-vm
    • cloud-hypervisor instead of qemu
  • sets up systemd-journal-remote on default port 19532
  • logs from host and other VMs (net-vm and gui-vm) redirected to a share (/var/log/journal/remote)
  • log-vm start-up order right after host
  • boot-logs to log-vm
  • binary files to log-vm
  • external producer (e.g. RT microcontroller) logs redirection to log-vm?
  • log extraction - to ELK-stack in cloud (outside ghaf)

Description of changes

Checklist for things done

  • Summary of the proposed changes in the PR description
  • More detailed description in the commit message(s)
  • Commits are squashed into relevant entities - avoid a lot of minimal dev time commits in the PR
  • Contribution guidelines followed
  • Ghaf documentation updated with the commit - https://tiiuae.github.io/ghaf/
  • PR linked to architecture documentation and requirement(s) (ticket id)
  • Test procedure described (or includes tests). Select one or more:
    • Tested on Lenovo X1 x86_64
    • Tested on Jetson Orin NX or AGX aarch64
    • Tested on Polarfire riscv64
  • Author has run nix flake check --accept-flake-config and it passes
  • All automatic Github Action checks pass - see actions
  • Author has added reviewers and removed PR draft status

Testing

List loggers

[ghaf@ghaf-host:~]$ ls -la /var/log/journal/remote/
total 131100
drwxr-xr-x+ 2 rtkit systemd-journal     4096 Apr 29 10:52 .
drwxr-sr-x+ 4 root  systemd-journal     4096 Apr 29 10:01 ..
-rw-r-----+ 1 rtkit systemd-journal  8388608 Apr 29 10:57 remote-192.168.101.1.journal
-rw-r-----+ 1 rtkit systemd-journal 75497472 Apr 29 10:52 remote-192.168.101.2.journal
-rw-r-----+ 1 rtkit systemd-journal  8388608 Apr 29 10:52 remote-192.168.101.3.journal

Follow logs

journalctl -f --file=/var/log/journal/remote/remote-<ip_address>.journal

@vilvo vilvo temporarily deployed to internal-build-workflow April 16, 2024 07:22 — with GitHub Actions Inactive
@vilvo vilvo marked this pull request as draft April 16, 2024 07:22
@vilvo vilvo temporarily deployed to internal-build-workflow April 19, 2024 14:12 — with GitHub Actions Inactive
@vunnyso
Copy link
Contributor

vunnyso commented Apr 23, 2024

Hi @vilvo with following patch log-vm can accept logs from ghaf-host vunnyso@e723fc2

@vilvo
Copy link
Contributor Author

vilvo commented Apr 24, 2024

Hi @vilvo with following patch log-vm can accept logs from ghaf-host vunnyso@e723fc2

Thanks, please push directly to the PR branch.

@vunnyso vunnyso temporarily deployed to internal-build-workflow April 24, 2024 11:17 — with GitHub Actions Inactive
@vunnyso vunnyso temporarily deployed to internal-build-workflow April 24, 2024 11:24 — with GitHub Actions Inactive
@vunnyso vunnyso temporarily deployed to internal-build-workflow April 24, 2024 11:25 — with GitHub Actions Inactive
@vunnyso vunnyso temporarily deployed to internal-build-workflow April 24, 2024 13:58 — with GitHub Actions Inactive
@vunnyso vunnyso temporarily deployed to internal-build-workflow April 25, 2024 08:50 — with GitHub Actions Inactive
@vunnyso vunnyso temporarily deployed to internal-build-workflow April 25, 2024 13:54 — with GitHub Actions Inactive
@vilvo vilvo temporarily deployed to internal-build-workflow April 29, 2024 06:55 — with GitHub Actions Inactive
This was referenced Apr 29, 2024
Open
Closed
vilvo and others added 6 commits May 17, 2024 16:02
@vilvo @vunnyso
log-vm: VM to secure and centralize logs
7d0194c 
* Design https://ssrc.atlassian.net/wiki/x/GYA4Pw
* log-vm scaffolding based on copy of net-vm
  - sets up systemd-journal-remote on default port 19532

Signed-off-by: Ville Ilvonen <ville.ilvonen@unikie.com>
@vilvo @vunnyso
log-vm: module for systemd-journal-upload service
0f30790 
* run service to upload journal to log-vm
  - tested only to start without errors
* formatting (nix fmt)

Signed-off-by: Ville Ilvonen <ville.ilvonen@unikie.com>
@vunnyso
log-vm: starting systemd-journal-remote service in log-vm
f98f70b 
Without this patch if ghaf-host tries to do `systemd-journal-upload`
there will be following error "Failed to connect to 192.168.101.66
port 19532 after 0 ms: Couldn't connect to server".
As there will be no listener running in log-vm to accept logs.

With this patch we add listener in log-vm on 19532 port to accept logs.
Logs will be stored at "/var/log/journal/remote" in log-vm.
`ghaf-host` logs are getting stored in log-vm successfully.

Signed-off-by: Vunny Sodhi <vunny.sodhi@unikie.com>
@vunnyso
log-vm: Extending upload.service and creating persistent storage
74e25fd 
This patch will extend support of logging of different vms such as
gui-vm, net-vm and app-vms.
Central logs will be stored `/var/log/journal/remote` on ghaf-host
Same logs can be viewed in log-vm as well.
App-vms logs will be saved as remote-192.168.101.1.journal.

Journal logs will be persistent even after reboot.

Signed-off-by: Vunny Sodhi <vunny.sodhi@unikie.com>
@vilvo @vunnyso
rebase: sshkeys imports
8464d31 
Signed-off-by: Ville Ilvonen <ville.ilvonen@unikie.com>
Signed-off-by: Vunny Sodhi <vunny.sodhi@unikie.com>
@vunnyso
fix: change service name
decf1a1 
Signed-off-by: Vunny Sodhi <vunny.sodhi@unikie.com>
@vunnyso vunnyso temporarily deployed to internal-build-workflow May 17, 2024 14:38 — with GitHub Actions Inactive
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@vilvo @vunnyso