Add runtime metrics into Tide service

[hellofromtonya]

Description

Error-free runtime is another tenant of code quality. I propose that we include it into our metrics to make Tide more robust and useful.

The Problem

The problem is:

When a theme or plugin has a runtime issue, Tide may be perceived as less informative and useful.

Imagine that a user picks a plugin or theme based upon its Tide report. Then s/he installs it only to discover that the plugin/theme throws errors or the logs have notices or warnings. Whoopsie, the perception of Tide is then diminished.

Sniffing for adherence to coding standards and best practices and measuring PHP compatibility are valuable parts of measuring code quality. But these techniques do not catch all runtime issues.

For example, consider:

• A piece of code relies on a component that is not yet loaded into memory. A runtime error will occur.
• A typo in the code may produce an error in the best case or unexpected behavior in the worst case.
• A nonexistent key, property, or method would produce an error when the code does not protect for that condition.
• Passing the wrong data type can trigger an error.

These types of problems reveal themselves when exercising the code's control flow through where the problem lives inside of the theme's or plugin's codebase.

Proposal

I propose that we exercise each theme and plugin through a set of pre-defined, diverse types of content/web pages (i.e. to exercise different parts and paths through the code), and then analyze the runtime logs (back-end) or console (front-end). Better yet, let's build a test suite to exercise and record the results.

What to Measure?

We could start small and measure error severity and occurrences. On the back-end, that's measurable through the error logs. On the front-end, that's measurable through the console.

How

It will require us to run the theme or plugin in order to exercise the code. We can leverage the Lighthouse server.

1. Create different types of web pages to represent different types of content, i.e. to exercise different parts of the code. This step would give us a more diverse population of data points.
2. Turn on WP_DEBUG, first proposed by Mike Schinkel.
3. Consider other debug runtime tools.
4. For the back-end, build a PHP exception handler to manage what happens and add it to the analysis stack.
5. Build a headless test suite to exercise the web pages and capture the front-end issues.

[jrfnl]

As a sister-issue, I believe it would be good to scan for parse errors too in all PHP versions supported by the plugin/theme.
Just running php -l over all files in the plugin/theme (possibly excluding a vendor directory) on all supported PHP versions should be able to take care of that.

[rheinardkorf]

@jrfnl That would be great as a seperate issue so that it does not get lost. I think that is a neat idea. (cc @jeffpaul )

@hellofromtonya I agree this would make Tide better if we could get actual runtime audits, but lets not focus on the score. There is no score (since WCUS).

[hellofromtonya]

Thank you, @rheinardkorf. "score" terminology has been removed.

[mikeschinkel]

+1 for scanning for catching plugins that generate errors when WP_DEBUG is true.

[jeffpaul]

Per bugscrub today, we're moving this to Future Release.

[paaljoachim]

Btw having a plugin similar to https://wordpress.org/plugins/p3-profiler/ to measure load times of various resources such as themes and plugin, and check for clean code etc would be great! It would make it real easy for people to check their own web sites and how it is performing.

[jeffpaul]

@hellofromtonya any chance you'd want to work up a PR to start adding this sort of info to Tide?

[hellofromtonya]

Hey @jeffpaul, Sorry for the delay in my response. For the next several months, I don't have extra capacity to work on the PR. I'd love for someone to take the lead on this one before then to get it rolling.