Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ability to simply, fully verify file checksum #76

Open
nh2 opened this issue Sep 7, 2021 · 2 comments
Open

Add ability to simply, fully verify file checksum #76

nh2 opened this issue Sep 7, 2021 · 2 comments
Labels
enhancement New feature or request

Comments

@nh2
Copy link

nh2 commented Sep 7, 2021

For accepting E57 data in automated processing, it would be awesome if libE57Format could offer a simple way to verify (the checsums of) an entire file with a simple function that can be called ahead of time, vs noticing E57_ERROR_BAD_CHECKSUM during (potentially more time consuming) stream processing.

If we wanted to add a .checkAll() function, how should it best be designed to generically traverse all data while still being as fast as possible?

Are there some slower approaches that would do the trick until then?

Finally, after a file has been verified this way, what would be the best way to disable checksumming during subsequent stream processing?

@nh2 nh2 changed the title Ability to fully verify file checksum Add ability to simply, fully verify file checksum Sep 7, 2021
@nh2
Copy link
Author

nh2 commented Sep 7, 2021

As a related aside:

I am getting a bit confused by CHECKSUM_POLICY_ALL docs saying:

Only verify 50% of the checksums.

and CHECKSUM_POLICY_HALF saying

Only verify 25% of the checksums.

Are those copy-paste errors?

asmaloney added a commit that referenced this issue Sep 7, 2021
@asmaloney
Copy link
Owner

Yes - the checksums are very expensive which is why I added the checksum policy.

Offhand I don't think there's an easy way to skip processing of the data - i.e. just read the bytes and do the checksum. The format is very.... intricate. The checksum is calculated on each 1020 bytes of the file, so that alone makes reading these thing a mess. Speed-wise it might be fine for a 5MB files, but with 1.5GB it becomes unmanageable.

I haven't looked at this code in a very long time, but if I can see a way to do what you're asking I'll let you know.

Are those copy-paste errors?

Nope - just my mishandling of the doxygen comments. I've fixed it - thanks!

@asmaloney asmaloney added the enhancement New feature or request label Sep 8, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants