Run PHPStan programmatically and get analysis results

[maks-rafalko]

Hello.

My question/request is quite specific because we need to run PHPStan under the hood of @infection - Mutation Testing library - to check if modified files still pass SA.

attention: understanding of Mutation Testing is needed to continue reading, so please read at least this Introduction

Why we need optional Static Analysis for killing Mutants is well explained here: https://github.com/Roave/infection-static-analysis-plugin?tab=readme-ov-file#background - this is a "brutal" implementation of Psalm integration with Infection as a standalone library.

What I want to achieve is basically the same, but with PHPStan, and not as a hacky solution but natively in Infection.

So, keeping Infection's implementation/integration out of this topic, here is what we really need:

• we have project files
• Infection mutates (changes) one single operator on one line of code at a time, e.g. changes $a + b to $a - $b in src/Calculator.php - and creates a tempory file with this change, e.g. /tmp/infection/Calculator.s0m3h4sh.php and runs tests as a separate process (in parallel with other "Mutants")
• we need to run PHPStan programmatically inside Infection execution so that it checks if /tmp/infection/Calculator.s0m3h4sh.php is still valid / passes PHPStan checks.

If new code (/tmp/infection/Calculator.s0m3h4sh.php) does not pass PHPStan check, we need to know result error messages (like Parameter $date of method HelloWorld::sayHello() has invalid typehint type DateTimeImutable.)

This is how it's currently implemented in Roave's infection static analysis plugin for Psalm, where Psalm is executed programmatically from the php code:

https://github.com/Roave/infection-static-analysis-plugin/blob/1.36.x/src/Roave/InfectionStaticAnalysis/Psalm/RunStaticAnalysisAgainstMutant.php#L28-L58

Questions that I have before playing with this:

• is something like that already supported in PHPStan?
• should we run PHPStan against only 1 (changed) file or againts all the original project's files with the only one file replaced? (like src/Calculator.php is replaced with /tmp/infection/Calculator.s0m3h4sh.php? I'm pretty sure that we need to run PHPStan against all the files, because there can be such mutations (changes) that produce error only when modifed Calculator class is used outside in other classes (like changing public to private for methods)

For example, if we do the following mutation:

class Calculator {
-    public function add(int $a, int $b): int 
+    protected function add(int $a, int $b): int 

then PHPStan will not find an issue if we run it against only this file. But it will find an issue if we run PHPStan against all the files in the project.

[ondrejmirtes]

I really like this idea, it'd make mutation testing a lot more useful! As for how to do this, you could certainly run PHPStan programatically inside other PHP application, but you'd lose a lot of benefits. If you simply run PHPStan on the command line as an external program, you get a lot for free and it's certainly easier. You get parallelism out of the box, and also the result cache (which is very important for this).

So in order to achieve your goal, I'd add more CLI options to the main analyse command for your needs. In fact, PHPStan used to contain these options for PHPStan Pro needs (it also modified a single file temporarily and saw what errors it fixed or added). So I could carefully revert and review this commit (phpstan/phpstan-src@f595e35) and this part would be done.

We'd introduce two new options: --tmp-file and --instead-of.

againts all the original project's files with the only one file replaced

Yes, you need to analyse the whole project, otherwise you miss out on errors. Fortunately the result cache can help a lot with the performance.

As for other considerations, I think you need to make sure that PHPStan passes with 0 errors before trying out temporary changes in the mutated file. I think this can be simply a prerequisite for Infection to work, similarly to the test suite being green before you run Infection. The baseline would certainly make that easy.

Also: you need to ask the user how to run PHPStan. So these things would have to be added to Infection config I think:

• Path to PHPStan (could be autodetected from Composer)
• Path to config file (if not specified, phpstan.neon and phpstan.neon.dist are auto-discovered in current working directory)
• Current working directory (usually project root but people do weird things)
• Analysed paths (they can be present in the config but the user can also pass them as arguments when running PHPStan)
• Other CLI options like --level, --autoload-file, --memory-limit

One more important question: Is Infection interested in actual error messages reported by PHPStan, or not? If not, we could also add an option --stop-on-first-error that would speed up the process. If Infection is interested in the actual error messages, we could add a new error formatter (https://phpstan.org/user-guide/output-format) to suit your needs.

Let me know if I'm right and all of this would be sufficient to adopt PHPStan in Infection.

[Wirone]

and only then when using by infection for a mutation, it doesn't need parallelism (since it's a change within a single file)

As it was pointed out already, change in a single file can lead to invalidating more than one file as a result cache. But in general I agree - Infection should have its own version of PHPStan's result cache, warmed up at the beginning of the process.

[ondrejmirtes]

This needs experiment of several approaches, and measure what works best.

[maks-rafalko]

@ondrejmirtes just to double check, did I get it write that

1. you are ok with getting back --tmp-file and --instead-of into the core for Infection purpuses
2. and you will revert that commit yourself

?

or should/can I help somehow?

[ondrejmirtes]

I will do this myself at some point after release of PHPStan 1.11. And then we should collaborate on when should actually be PHPStan executed by Infection.

[maks-rafalko]

sounds good, thanks

[romm]

Just commenting to let you know that I'm interested on giving the solution a try, whenever it is ready for early testing. 😊

Thank you for making PHP better everyday! ❤️