Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make tests fully deterministic #177

Open
tomato42 opened this issue Jan 9, 2018 · 4 comments
Open

Make tests fully deterministic #177

tomato42 opened this issue Jan 9, 2018 · 4 comments
Labels
bug unintented behaviour in tlsfuzzer code enhancement new feature to be implemented help wanted
Projects

Comments

@tomato42
Copy link
Member

tomato42 commented Jan 9, 2018

While client_random already is set to all-zero (though it creates different problems: #139 ) and RSA premaster_secret is handled the same, stuff like client (EC)DH share is not.

Runner should be extended to pick a seed value that is then used for specific test runs, so that a combination of static seed and a particular name of conversation will generate the same conversation (provided the server is doing the same).

Then update the premaster_secret generation and client_random selection to use that seed for their generation.

@tomato42 tomato42 added enhancement new feature to be implemented help wanted labels Jan 9, 2018
@nmav
Copy link
Collaborator

nmav commented Jan 10, 2018

What is the issue you trying to address with this change? Going to a fully deterministic mode for all tests will most likely cause other problems as it will limit the code paths exercised by the IUT.

@tomato42
Copy link
Member Author

@nmav By that I mean "fully deterministic for a given seed", if you don't specify the seed the test case will be just as random (and I want to make them more random) as they are now.

The aim is so that in case of failure, you want to be able to take the log of the run and reproduce it exactly on your development machine - now when the actual fuzzing is non-existant to minimal it's not much of a problem, but it will get only worse as time progresses.

@tomato42
Copy link
Member Author

tomato42 commented Feb 4, 2019

This is necessary to make the workflow with wireshark much nicer (currently only one connection at a time can be collected and easily decrypted)

@tomato42 tomato42 pinned this issue Feb 4, 2019
@tomato42 tomato42 added the bug unintented behaviour in tlsfuzzer code label Feb 4, 2019
@tomato42 tomato42 added this to To do in Usability via automation Nov 9, 2019
@tomato42
Copy link
Member Author

tomato42 commented Jun 5, 2020

There are actually two RNGs we need to worry about, one is the PYTHONHASHSEED that influences in older pythons the order in which the objects are returned from a dictionary (important for testing
extensions)

then we have the situation when we are running tests one by one and want to reproduce a single test, at such point the RNG may have advanced arbitrary amount of steps, so when actually running the test, we should capture the state of the RNG before executing the test and then print it on failure (+some mechanism to set it when just single test is executed)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug unintented behaviour in tlsfuzzer code enhancement new feature to be implemented help wanted
Projects
Usability
  
To do
Development

No branches or pull requests

2 participants