Handling annotation processors (Error: Annotation processors must be explicitly declared now)

[andrewleech]

Let me just start with wow, thanks, wow.
I've been maintaining & building on a app/android platform from a defunct company (navdy) for some time now, gradually decompiling bits and compiling them in a project against a the original apk (turned into jar) and it's always been such a fight to avoid bugs, as you clearly well know.
Don't know why it took me so long to find this project, but thank you!

This error report is low priority and can be easily worked around (for now at least).
I'm also just getting started with your tools and quite probably doing something wrong at my end too.

The original apk uses a few annotation processors in its build process (dagger, mortar etc) and I suspect they (in the automatically extracted dex/classes used by your plugin) are triggering the warning below.

Execution failed for task ':javaPreCompileDebug'.
> Annotation processors must be explicitly declared now.  The following dependencies on the compile classpath are found to contain annotation processor.  Please add them to the annotationProcessor configuration.
    - classes.jar (< App Components > :app-components:unspecified)
  Alternatively, set android.defaultConfig.javaCompileOptions.annotationProcessorOptions.includeCompileClasspath = true to continue with previous behavior.  Note that this option is deprecated and will be removed in the future.
  See https://developer.android.com/r/tools/annotation-processor-error-message.html for more details.

The override command suggested works, but as they say it might be removed at any time.

If you ever want to see it yourself, a copy of the original apk is available at https://gitlab.com/alelec/navdy/display-rom/blob/master/system/priv-app/Hud/Hud.apk

[Lanchon]

hey Andrew,

thanks! i developed dexpatcher for my own use, as i needed to patch some app myself, and it helped enormously. with the time cost of developing dexpatcher factored in i can't say i ended up saving time, but at least it made the whole project much more enjoyable. there's also a D.R.Y. principle behind it, as i can reuse the effort i put in dxp in later projects ...if they ever existed :). i hoped the community would find uses for it, but in reality it seems very few people ever tried it. so i'm always happy to find out i'm helping someone. so thanks!

re: "Annotation processors must be explicitly declared now."

the gradle team has put huge effort into turning their nice slow tool into the top notch performance contender it is today. gradle is getting very savvy at optimizing builds. and it is well known that the most optimized way of handling a task is avoiding having to do it. recent gradle versions implement a feature they call 'compilation avoidance', and the story goes like this...

imagine you have a huge project with components A through Z. imagine component A produces a jar library artifact consumed by projects B through Z. they noticed somebody made a spelling mistake in a print statement in A, so they corrected it and launched a whole project build.

just by looking at A's code, it was obvious for anybody with basic understating of the JVM that A's change wouldn't impact artifacts of projects B through Z. but nonetheless a new A artifact was produced having a new hash, and the build system dutifully noticed the change so it needlessly recompiled everything under the sun.

build systems all know what a file dependency is and how to detect its change. but some smarter systems know what a library jar is and how it its used during compilation. they know, for instance, that dates of class files inside a jar are don't cares for compilation, and they fingerprint each class file inside the jar instead of working on the jar as a monolithic dependency. so if a consumed jar is rebuilt in the future but producing unchanged code, dependencies don't need to be recompiled just because some ignored metadata in the jar (dates/times) have changed. this is all great, but doesn't help in the case we imagined above, where a printf() string argument changed, producing an actual class file change.

enter gradle's compilation avoidance. gradle knows jars but also knows much more about javac and the JVM. it fingerprints each class of each jar, not as a monolithic entity, but instead dives into its structure and extracts all aspects of the class file that can potentially impact a compilation that includes it in its classpath. gradle calls this the 'ABI', the application binary interface, and as long as it doesn't change, non-ABI changes in a dependency do not trigger recompilation of its dependents.

this is an amazing feat, and my hats goes off to the gradle team for it. but it has issues. or rather, javac has issues.

java's designers made a big mistake when defining annotation processors (APs): instead of feeding APs to javac through a well-defined interface, they chose to have javac receive a single promiscuous mix of classpath classes and APs, and have it detect APs within it.

one of the effects if this ill-informed decision is: some classes in javac's classpath cause javac to behave in a very unexpected way. normally, non-ABI content of javac's classpath (say, actual code inside a classpath method) does not influence javac's output. (this is why, for instance, the android SDK can get away with compiling android apps against a pre-processed, vacated, empty shell of android's api, instead of the actual api with implementation code produced by javac that runs on a device.)

but in reality, code of some methods in javac's classpath can influence javac's output: some methods can execute during compilation if they are called from an AP, and there is no way of knowing which methods are those.

so what happens to compilation avoidance? gradle did the right thing: being efficient today, correct, backwards compatible, and more efficient in the future. gradle looks for APs in the compile classpath and disables compilation avoidance if it finds any (to be efficient, correct, and backwards compatible). but they want to be more efficient in the future by deprecating APs in the compile classpath. so you receive a warning: "please move your APs to a different configuration; we'll detect any and all changes there to disable compilation avoidance just for the next build. (changes in APs are extremely rare for regular projects; so don't worry, we got your build performance covered.) if you don't move your APs we'll continue to build your project for now, but with permanently disabled compilation avoidance, so you'll pay a price on each build. and we don't want to be called 'slow' because you are lazy, so we'll stop accepting this in the future."

what to do about it:

• move your APs to the annotationProcessor configuration.
• but even if you are not using APs, some badly-made libraries leak their APs (which are implementation details) into their output artifacts and thus your project. and you were executing foreign untrusted code as your user each time you javac'ed with them in the classpath and you didn't even know (!!!), and hopefully these APs were kind enough to leave your code alone, but who knows. and it was causing slow compiles and disabling compilation avoidance. now with gradle you know this is happening, and you can choose to deactivate and ignore these leaked APs (and gradle's warning) with options.compilerArgs << '-proc:none'.

note that if you put anything at all in annotationProcessor, you are declaring a new-style build, and thus implicitly disabling classpath APs (and the warning). default javac behavior (and by extension old default gradle behavior) is dangerous; i'm glad gradle stepped in and did something about it.

so what about DexPatcher and your project?

gradle did the right thing so dxp doesn't have to do anything at all.

APs typically contain two parts: the AP proper (generates code), plus a runtime lib (invoked by generated code). the runtime lib must be included in the final build just like any lib, but the AP proper should never. so APs should have been partitioned in 2 jars; always, from the start. but the confusion caused by javac's ill-informed choice caused some AP implementers to produce just one jar; after all, it all sort of goes in the classpath, right?

some APs are so down this rabbit whole that they don't even want to hear about whats wrong with it. they blame gradle, but the problem was always javac:

This warns/errors when using lombok. Main contributor of project lombok here: As far as we can tell this is a misunderstanding by team gradle; we've been asked to split up our jar to 'solve' this problem, but if we do that, we get 50 different bug reports because, well, that'd break a lot of other things; as such we are disinclined to fix this problem." (gradle/gradle#5056)

so what happened here? someone made a mistake when building the original source app you are working on, and they bundled some AP proper into the APK when they should only have included its runtime lib (mabye because the AP author didn't care to split them?).

first thing is, you need to stop that untrusted code from running: either use the annotationProcessor configuration to include some APs you want to use, or else do something like:

tasks.withType(JavaCompile) {
    options.compilerArgs << '-proc:none'
}

or you can upgrade to gradle 5.0+.

next, if you find out that the offending APs are split between proper and runtime lib, you may choose to remove the AP proper implementation to eliminate dead code and tidy things up. (see my dxp min-faq to see how to remove packages and package trees.) but note that you must not remove the runtime libs, if any, or you'll break the app.

pheeeeewww... this was a long post! sorry!

everything clear now?

[andrewleech]

That's incredible, thanks for such a detailed response, it's quite fascinating.

Don't take this the wrong way, but I'm not that surprised you haven't had a huge number of users. Even within the software development industry there's very few people willing to step into the reverse engineering area, it's got such a "shady" sort of vibe and there's so much fear about getting sued for copyright etc.
It's also generally just really hard. You can spend days/weeks pulling apart a bit of code to tweak just one little barely noticeable feature, not many people are willing to make the effort.

Geez tell you what though, you should be really proud of DexPatcher, I've never come across a tool in the reveng space that makes such a complex task look so easy.

With my project, that info is really useful, cheers. The original apk's were from ~2017 using early versions of dagger etc so yeah, they probably are including versions that have these issues.

I'll try out your suggestions and see how I go cleaning up the libs used in the project to avoid the issue. The suggestion my build times might improve if I sort it out is always tempting, though to be honest I'm blown away at how quick it is already considering what your plugin is doing behind the scenes!

[andrewleech]

I've been trying to remove any packages I think might be the problem by creating package-info.java files eg

@DexRemove(recursive = true)
package javax.annotation.processing;

Without much luck. Going the other way, in the error message as I originally posted it calls out
- classes.jar (< App Components > :app-components:unspecified)
So I went looking for that, the only one I find is build/intermediates/dexpatcher/extra-resources/classes.jar

Opened that up to take a look.
I know my APK has some extra resource data for fabric and a google phonenumber library which I see in this jar, but it also has META-INF/services/javax.annotation.processing.Processor which definitely looks like the culprit.

Can you suggest a way to exclude that?

[Lanchon]

ok... so what goes into this (inner dexpatcher implementation detail) 'classes.jar' is odd stuff thats thrown around the source APK. it definitely doesn't include classes (dex code) (unless the dex files in question are thrown around the APK and expected to be loaded by some weird non-standard mechanism), it's put in that 'classes.jar' to convince the android build system to scatter the contents around the APK when rebuilding. but neither the android build system nor dxp know what sort of dead cats might be in there. so there is no way to affect the contents. (dxp can patch dex code, manifests, and the various resources, but not stuff it doesn't know the first thing about.)

so DexRemove won't do a thing, because it's not dex code that you need to remove.

apparently, (because i didnt know) AFAICT from your screenshot, annotation processors are declared in the META-INF section of the jar, possibly as a text file. you could intercept the apklib and have that removed, but really, there is no need. just tell gradle to ignore the APs with options.compilerArgs << '-proc:none' and be happy. once you do that, compilation avoidance is restored.

not that it matter, because no separately compiled code will ever consume your patch code. but hey, at least now you know what stuff is doing what where :) and you'll have to forgive my lack of rigor in this post, as it might be the case that i might be tripping at this moment. but you'll never know... :-p

[andrewleech]

hehe, no need to ask forgiveness, you're giving me time and knowledge for free here ;-)

For certainty, I deleted that services folder out of the jar and the warning disappeared, so that was the trigger.

I added the suggested setting to build.gradle but that doesn't get rid of the warning (though I believe it will make the compilation itself safer.

tasks.withType(JavaCompile) {
    options.compilerArgs << '-proc:none'
}

I also needed javaCompileOptions.annotationProcessorOptions.includeCompileClasspath = false setting to allow the build to complete. I changed the error's suggestion from true to false, presuming that'll probably achieve basically the same goal as the options.compilerArgs above.

Thanks again!