Maintain high capture rate for long batch captures #46
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
By default, the ChipWhisperer API starts every new trace segment (in the project file, not related to the segment/batch captured in one shot) with the size of 1 trace. Upon appending new traces to the project, the segment is resized in steps of 25 traces. This results in frequent array resizing and decreasing capture rate. With this commit, the allocation size is frequently checked. If it's lower than the known segment size, it's corrected directly to this value. The next resize operation will allocate sufficient memory for the whole segment array. Signed-off-by: Pirmin Vogel <vogelpi@lowrisc.org>
moidx
approved these changes
Jun 16, 2021
| @@ -97,17 +98,34 @@ def run_batch_capture(capture_cfg, ot, ktp, scope): | |||
| f"actual: {actual_last_ciphertext}\n" | |||
| f"expected: {expected_last_ciphertext}" | |||
| ) | |||
| # Make sure to allocate sufficient memory for the storage segment array during the | |||
There was a problem hiding this comment.
Would it make sense to add a reference to the CW issue here? newaetech/chipwhisperer#344
alphan
approved these changes
Jun 16, 2021
By default, the ChipWhisperer performs a range check `_updateRanges()` on all previously captured traces whenever adding a new trace. As a result, the complexity of the `append()` function scales linearly with the number of previously captured traces which significantly slows down the capture rate. With this commit, all except for the latest two trace storage segments are disabled. The `_updateRange()` function then behaves like a sliding window and only looks at the latest two instead of all segments. This allows to maintain high capture rates even when capturing millions of traces. Signed-off-by: Pirmin Vogel <vogelpi@lowrisc.org>
Signed-off-by: Pirmin Vogel <vogelpi@lowrisc.org>
vogelpi
force-pushed
the
avoid-frequent-array-resize
branch
from
601e67e
to
7ead6a0
Compare
|
Thanks for the quick review guys! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR contains two commits in order to increase the capture rate for long batch captures.
disableall except for the latest two storage segments to keep the run time of theappend()function approximately constant. By default, it increases linearly with the number of traces captured.With these changes it is possible to maintain high capture rates up to millions of traces. Further changes are probably needed to avoid memory limitations. Currently everything is kept in memory during a capture and traces are written to disk only at the very end. I've opened an issue newaetech/chipwhisperer#344 in the hope of getting some further advice.