Tests to verify the sample code in the readme functions

[m8ttyB]

I'm not sure if this is possible but wanted to capture the intention after seeing issue #149 filed by @mudgil. Can a process, like those run in TravisCI, verify when a pull request is made that sample code embedded in the Readme continues to function?

I recognize this might not be possible, or that the effort involved may be too great. Let's do a brief investigation into the viability of this.

[kimberlythegeek]

@m8ttyB I was wondering the same thing. updating the readme and examples is something that I often forget when making changes...

[kimberlythegeek]

@davehunt Do you know of any tools that can lint or test example code in docs?

[davehunt]

There's doctest but that's based on Python's docstrings. I'm not aware of anything that would be able to test example code from a README though. Another option might be to build the README from a snippet that's tested independently.

[kimberlythegeek]

Another option might be to build the README from a snippet that's tested independently.

That sounds promising. After doing a quick search, there may be tools that exist already to test code in readme files. Will follow up on this.

[terrameijar]

@kimberlythegeek @m8ttyB I second what @davehunt said about testing a code snippet independently. It is possible to write a script that reads the README, extracts the python code from it and then runs it through a linter like flake8 or pycodestyle. I'd be happy to give this a shot.

[kimberlythegeek]

thanks @terrameijar I always welcome contributions :)

[m8ttyB]

+1 Thanks @terrameijar. Sure, let us know what you come up with.

[terrameijar]

Before the code in the file can be checked for errors, we first need to extract or find it in the document.

There are two options I can think of to solve this:

1. To convert the README from .rst to .md format. It is relatively simple to extract code out of markdown files. Converting from one format to another will require using a third party tool for converting documents like pandoc.
2. Second option is to work on the README directly to extract the python code from it using regular expressions.

I prefer option 2. I am working on figuring out how to write the regex pattern that'll work in this case.

[terrameijar]

@m8ttyB @kimberlythegeek
Update: I have written a script that extracts Python code from a README document and runs checks on it using flake8. I put up this gist .

I am not entirely sure what the best way to invoke tests from the script is, but this proves at least that linting code in READMEs can be done.

[terrameijar]

This might be worth a look at, it is a tool for linting Python code blocks in markdown files:
https://github.com/johnfraney/flake8-markdown,