Osquerybeat: Strip osqueryd binary for linux #38952

aleksmaus · 2024-04-15T18:31:41Z

Proposed commit message

Strip osqueryd binary for linux that is unstripped in the official osquery tar.gz distro.
Size reduction
ARM: 273,418,504 -> 78,956,480 bytes
x86: 270,282,072 -> 86,097,240 bytes

Changed dev-tools code for linux crossbuilds.
Noticed that the image docker.elastic.co/beats-dev/golang-crossbuild:1.21.9-arm currently exists for both ARM and x86 archs and is being used randomly, which causes a problem for strip tool when arch of binaries are mismatched with the arch of the tool. This affects only linux builds.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

Ensure that everything builds as before. The size of osqueryd is 3x smaller than before on linux OS, currently is under 90M.

Related issues

Closes Package stripped osqueryd executable with osquerybeat on all platforms #38890

Screenshots

Before:

After:

elasticmachine · 2024-04-15T18:31:43Z

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

botelastic · 2024-04-15T18:31:48Z

This pull request doesn't have a Team:<team> label.

mergify · 2024-04-15T18:32:19Z

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @aleksmaus? 🙏.
For such, you'll need to label your PR with:

The upcoming major version of the Elastic Stack
The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

cmacknz

Thanks!

elasticmachine · 2024-04-15T19:52:22Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Duration: 140 min 12 sec

❕ Flaky test report

No test was executed to be analysed.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

blakerouse

I ask that this be changed to support agentbeat as well as osquerybeat. Agentbeat must also ship with osqueryd or the osquerybeat subcommand will not work.

See this PR for the changes to make it work with agentbeat - #38951

aleksmaus · 2024-04-16T18:31:10Z

Updated PR in order to address agentbeat build as well.

Couple of things worth mentioning:

Needed to have the "strip" called from within the container the arch of the binary and the "strip" tool have to match otherwise it fails complaining on the binary architecture, so kept the stripping in GolangCrossBuild still
Didn't see a good way to detect if the crossbuild was initiated from osquerybeat or agentbeat. The build/install directory where osqueryd is unpacked is different. So the script now searched in two locations for osqueryd binary: osquerybeat and if not found agentbeat.

mergify · 2024-04-17T02:06:28Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b feature/strip_linux_oquerybeat upstream/feature/strip_linux_oquerybeat
git merge upstream/main
git push upstream feature/strip_linux_oquerybeat

blakerouse · 2024-04-17T20:10:18Z

With agentbeat merged, lets get the conflicts resolved and then I can take a better look.

…_oquerybeat

aleksmaus · 2024-04-17T22:15:20Z

Picked up the latest main, resolved conflicts, improved code slightly

cmacknz · 2024-04-18T13:24:27Z

Adding the backport-v8.14 label, IMO not stripping osqueryd is a bug.

aleksmaus · 2024-04-18T13:55:53Z

Adding the backport-v8.14 label, IMO not stripping osqueryd is a bug.

Or a feature upstream ;-)

blakerouse · 2024-04-18T15:22:09Z

x-pack/osquerybeat/magefile.go

+	}
+
+	return nil
+}


Why is this still here? The code has changed to not need this specifically in osquerybeat.

This a mistake on conflict resolution (context switching), the function is not used. Removed.

blakerouse · 2024-04-18T15:22:49Z

x-pack/osquerybeat/magefile.go

+	if err := stripLinuxOsqueryd(); err != nil {
+		return err
+	}
+


How does this work for agentbeat? Agentbeat doesn't import the magefile from osquerybeat.

This works because agent beat calls out to osquerybeat to build the extension

beats/x-pack/agentbeat/magefile.go

Line 112 in 5e16f25

return callForBeat("crossBuildExt", "osquerybeat")

This path is also followed for stanalone osquerybeat crossbuild.

blakerouse

Okay. I forgot I did it that way. ;-)

pkoutsovasilis

As stripping the osqueryd binary goes this LGTM! but I am not familiar with AgentBeat and I can't guarantee that nothing erupts there but I trust @aleksmaus comment here

* Osquerybeat: Strip osqueryd binary for linux * Improve code comment, fix a typo * Address code review request to make it compatible with agentbeat build * Remove unused function (cherry picked from commit ddddf80)

amitkanfer · 2024-04-22T21:52:33Z

Sweet!!

PR elastic#38952 surfaced a bug with elastic#38366 where the parameters weren't enclosed in doublet quotes. https://buildkite.com/elastic/beats-xpack-osquerybeat/builds/1973#018f06dd-19fd-40e0-ba21-960d195f178e/131-316

* Osquerybeat: Strip osqueryd binary for linux * Improve code comment, fix a typo * Address code review request to make it compatible with agentbeat build * Remove unused function (cherry picked from commit ddddf80) Co-authored-by: Aleksandr Maus <aleksandr.maus@elastic.co>

Osquerybeat: Strip osqueryd binary for linux

eeb6b0e

aleksmaus added enhancement Team:Elastic-Agent Label for the Agent team >enhancement Team:Security-Deployment and Devices Deployment and Devices Team in Security Solution labels Apr 15, 2024

aleksmaus requested review from a team as code owners April 15, 2024 18:31

aleksmaus requested review from belimawr and fearful-symmetry April 15, 2024 18:31

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Apr 15, 2024

aleksmaus requested review from cmacknz and blakerouse April 15, 2024 18:32

mergify bot assigned aleksmaus Apr 15, 2024

cmacknz approved these changes Apr 15, 2024

View reviewed changes

aleksmaus added 2 commits April 15, 2024 14:38

Improve code comment, fix a typo

571834c

Merge branch 'main' into feature/strip_linux_oquerybeat

f176027

blakerouse requested changes Apr 16, 2024

View reviewed changes

Address code review request to make it compatible with agentbeat build

3880667

aleksmaus requested a review from blakerouse April 16, 2024 18:50

Merge remote-tracking branch 'upstream/main' into feature/strip_linux…

4f8c44e

…_oquerybeat

Merge branch 'main' into feature/strip_linux_oquerybeat

cbf2753

cmacknz added the backport-v8.14.0 Automated backport with mergify label Apr 18, 2024

blakerouse requested changes Apr 18, 2024

View reviewed changes

aleksmaus added 2 commits April 18, 2024 14:43

Remove unused function

d1bc83d

Merge branch 'main' into feature/strip_linux_oquerybeat

c073832

aleksmaus requested a review from blakerouse April 18, 2024 18:47

aleksmaus added 2 commits April 22, 2024 10:18

Merge branch 'main' into feature/strip_linux_oquerybeat

ed2008e

Merge branch 'main' into feature/strip_linux_oquerybeat

77ba933

blakerouse approved these changes Apr 22, 2024

View reviewed changes

pkoutsovasilis approved these changes Apr 22, 2024

View reviewed changes

cmacknz merged commit ddddf80 into elastic:main Apr 22, 2024
171 of 188 checks passed

mergify bot mentioned this pull request Apr 22, 2024

[8.14](backport #38952) Osquerybeat: Strip osqueryd binary for linux #39149

Merged

6 tasks

dliappis mentioned this pull request Apr 23, 2024

Test fix for BK packaging failure #39155

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Osquerybeat: Strip osqueryd binary for linux #38952

Osquerybeat: Strip osqueryd binary for linux #38952

aleksmaus commented Apr 15, 2024 •

edited

elasticmachine commented Apr 15, 2024

botelastic bot commented Apr 15, 2024

mergify bot commented Apr 15, 2024

cmacknz left a comment

elasticmachine commented Apr 15, 2024 •

edited

Build stats

blakerouse left a comment

aleksmaus commented Apr 16, 2024

mergify bot commented Apr 17, 2024

blakerouse commented Apr 17, 2024

aleksmaus commented Apr 17, 2024

cmacknz commented Apr 18, 2024

aleksmaus commented Apr 18, 2024

blakerouse Apr 18, 2024

aleksmaus Apr 18, 2024

blakerouse Apr 18, 2024

aleksmaus Apr 18, 2024

blakerouse left a comment

pkoutsovasilis left a comment

amitkanfer commented Apr 22, 2024

Osquerybeat: Strip osqueryd binary for linux #38952

Osquerybeat: Strip osqueryd binary for linux #38952

Conversation

aleksmaus commented Apr 15, 2024 • edited

Proposed commit message

Checklist

How to test this PR locally

Related issues

Screenshots

elasticmachine commented Apr 15, 2024

botelastic bot commented Apr 15, 2024

mergify bot commented Apr 15, 2024

cmacknz left a comment

Choose a reason for hiding this comment

elasticmachine commented Apr 15, 2024 • edited

💚 Build Succeeded

Build stats

❕ Flaky test report

🤖 GitHub comments

blakerouse left a comment

Choose a reason for hiding this comment

aleksmaus commented Apr 16, 2024

mergify bot commented Apr 17, 2024

blakerouse commented Apr 17, 2024

aleksmaus commented Apr 17, 2024

cmacknz commented Apr 18, 2024

aleksmaus commented Apr 18, 2024

blakerouse Apr 18, 2024

Choose a reason for hiding this comment

aleksmaus Apr 18, 2024

Choose a reason for hiding this comment

blakerouse Apr 18, 2024

Choose a reason for hiding this comment

aleksmaus Apr 18, 2024

Choose a reason for hiding this comment

blakerouse left a comment

Choose a reason for hiding this comment

pkoutsovasilis left a comment

Choose a reason for hiding this comment

amitkanfer commented Apr 22, 2024

aleksmaus commented Apr 15, 2024 •

edited

elasticmachine commented Apr 15, 2024 •

edited