Optimizations to reduce MySQL writer DB load #18880
Conversation
getvictor
force-pushed
the
victor/18838-master-db-read-reduction
branch
from
5b9d1c3
to
dc3d4d8
Compare
| // The calculation must match the one in softwareChecksumComputedColumn | ||
| func computeRawChecksum(sw fleet.Software) ([]byte, error) { | ||
| h := md5.New() //nolint:gosec // This hash is used as a DB optimization for software row lookup, not security | ||
| cols := []string{sw.Name, sw.Version, sw.Source, sw.BundleIdentifier, sw.Release, sw.Arch, sw.Vendor, sw.Browser, sw.ExtensionID} |
There was a problem hiding this comment.
is the intention here to generate the same hash as the current implementation, because the columns are in a different order here.
There was a problem hiding this comment.
I don't follow. They look like the same order to me:
CONCAT_WS(CHAR(0),
%sname,
%[1]sversion,
%[1]ssource,
COALESCE(%[1]sbundle_identifier, ''),
`+"%[1]s`release`"+`,
%[1]sarch,
%[1]svendor,
%[1]sbrowser,
%[1]sextension_id
)| level.Debug(ds.logger).Log( | ||
| "msg", "software item not found or created", "name", software.Name, "version", software.Version, "source", software.Source, | ||
| "bundle_identifier", software.BundleIdentifier, "checksum", uuidString, | ||
| ) |
There was a problem hiding this comment.
might also want to consider batching inserts below to account for a lot of new rows during host enrollment
There was a problem hiding this comment.
Yes, let's leave this optimization to after the release. We should also batch the UPDATE statements. I added a TODO to the issue #18838
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #18880 +/- ##
==========================================
- Coverage 66.75% 66.71% -0.04%
==========================================
Files 887 887
Lines 108471 108523 +52
==========================================
- Hits 72409 72406 -3
- Misses 30181 30219 +38
- Partials 5881 5898 +17
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
| ) | ||
| for checksum, software := range softwareChecksums { | ||
| uuidString := "" | ||
| checksumAsUUID, err := uuid.FromBytes([]byte(checksum)) |
There was a problem hiding this comment.
Curious about why we need to convert to UUID (non printable chars?)
There was a problem hiding this comment.
This is just for debugging. In MySQL, we can use BIN_TO_UUID to compare to the printed value.
There was a problem hiding this comment.
I don't expect these print statements to be hit.
#18838 and #18986
Optimized master DB accesses during host software ingestion.
Checklist for submitter
If some of the following don't apply, delete the relevant line.
changes/,orbit/changes/oree/fleetd-chrome/changes.See Changes files for more information.