Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Generate test case data for better test coverage #640

Open
Rycieos opened this issue Dec 21, 2020 · 3 comments
Open

Generate test case data for better test coverage #640

Rycieos opened this issue Dec 21, 2020 · 3 comments
Assignees
Labels
help wanted An issue where ideas or patches are welcome tests Related to automated testing

Comments

@Rycieos
Copy link
Collaborator

Rycieos commented Dec 21, 2020

Lately, making changes to Liquidprompt has been tough without accidental breaking something for some users, or adding a regression for a fix. I do not have access to many types of environments (especially not MacOS), so manually testing Liquidprompt is simply impossible.

I have been working on improving test coverage, but testing how a data function interacts with the environment's data source is impossible without knowing what that data source looks like.

To solve this, I built the tools/external-tool-tester.sh script, which will run the external tools currently used by Liquidprompt to collect data and save the output to a file where they can be added to the testing suite as test cases.

If you want to help Liquidprompt be more stable, or want to make sure that your unique environment stays compatible in the future, please submit a report from your machine!

IMPORTANT: you must upload the file, do not copy+paste the file contents. Invisible characters are in the file, and will be lost with copy+paste.

You can upload the file by dragging it into the comment box on this issue thread, which uploads it to GitHub.

The file may contain unique and identifying information, like your local username and hostname. We will censor identifying information before adding your outputs as test cases, but if you wish to censor them before uploading, you may do so. Only replace alpha characters with alpha characters, and number characters with numbers. Special Unicode characters (like ä) and whitespace leave as they are to improve test coverage. For example:

- my.unique3.hostname.net: jän smith
+ my.others1.domainss.net: äsr names

Use a text editor that does not change control characters or line endings. There is a line near the top of the file that you can use to check if they have been modified that should look like one of these lines (depending on how your editor displays them):

Special character check: ^G^H<tab>^A^M
Special character check: ^G^H	^A^M
Special character check: \a\b\t\001\r
@Rycieos Rycieos added help wanted An issue where ideas or patches are welcome tests Related to automated testing labels Dec 21, 2020
@Rycieos Rycieos self-assigned this Dec 21, 2020
@Rycieos Rycieos pinned this issue Dec 21, 2020
@augmentedfourth
Copy link
Contributor

These are my three main environments:
augmentedfourth_macos10.14.txt
augmentedfourth_macos10.15.txt
augmentedfourth_ubuntu20.04.1.txt

One of these days I'm sure I'll get one of my machines up to macOS 11, but it's still too new for me to wade into those waters yet.

@jaesivsm
Copy link
Contributor

Here are my ubuntu 20.04 dell laptop and my soon to be upgraded debian stretch server and various buster server. Not unique at all around here but you never now :)

@Rycieos
Copy link
Collaborator Author

Rycieos commented Dec 28, 2020

Thanks @augmentedfourth, I added test cases based on your machines in 37db052.

And thanks @jaesivsm, I added test cases based on your machines in cef9cb1.

You both had Debian machines (where the default shell is Dash), which made me discover that my tester script had a bug, fixed in 4b7fd88.

And @jaesivsm, one of your machines had acpi, which I didn't have any test cases for before, so I added a tester for that which helped me catch two related bugs, fixed in cf8bf97 and eb30942.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help wanted An issue where ideas or patches are welcome tests Related to automated testing
Projects
None yet
Development

No branches or pull requests

3 participants