Compilation time issue when having a lot of requests in a project

[guizmaii]

Hi everyone,

We have a project that contains a lot of SQL queries written with Quill, and we see our compilation time increase significantly.
Is there any tip on optimising the compilation time in such cases?

I personally don't know much about Scala2 (nor Scala3) macros but how can we help to provide you with information so that you could maybe find optimisation in your code?

Version: latest
Module: quill-jdbc-zio
Database: PG

@getquill/maintainers

[lachezar]

I can second this complaint. We have invested quite a lot of time to migrate from Doobie to pure Quill queries and now the compilation times are really slow. It feels like for some reason the work is done in sequence in a single thread instead of in parallel, but I don't know quite why.

Any kind of suggestion how to speed up the compilation time will be highly appreciated.

Or at least some explanation why is there such slowdown...

[guizmaii]

FYI @lachezar, I discovered this PR documenting a tip to speed up compilation time: #2532

Can you give it a try maybe?

[lachezar]

Thanks for the pointer @guizmaii , however the compilation time did not improve significantly and I started getting runtime errors from Quill 😬 , so this solution did not work in my case. It might be due to the fact that we use quill-doobie and not a pure quill implementation 🤷

[jilen]

@lachezar

Add implicit QueryMeta instance should not affect runtime behavior, it only affect compile time implicit resolution.
Also there's

 -Yprofile-trace FILE 

in scalac, this can help find the slow part of compilation.

[lachezar]

Thanks for the suggestions, @jilen

When trying to run with -Yprofile-trace I get java.lang.ExceptionInInitializerError and a very long stack trace. I use Java 11.0.19 and sbt 1.8.3 and scala 2.13.10.

Regarding the runtime error: I've detected the error when running the unit/integration tests of the project and it is due to java.lang.NoSuchMethodError

Error sample:

java.lang.NoSuchMethodError: 'java.lang.Object <redacted>.repositories.OfferRepositoryQueries$.$anonfun$offerQueryMeta$1$adapted(java.lang.String)'
	at <redacted>.repositories.OfferRepositoryQueries$$anon$1.$anonfun$extract$1(OfferRepository.scala:451) ~[classes/:na]
	at io.getquill.doobie.DoobieContextBase.$anonfun$extractorToRead$1(DoobieContextBase.scala:210) ~[quill-doobie_2.13-4.4.1.jar:4.4.1]
	at io.getquill.doobie.DoobieContextBase.$anonfun$extractorToRead$1$adapted(DoobieContextBase.scala:210) ~[quill-doobie_2.13-4.4.1.jar:4.4.1]
	at doobie.hi.resultset$.$anonfun$build$1(resultset.scala:90) ~[doobie-core_2.13-1.0.0-RC2.jar:1.0.0-RC2]
	at doobie.free.KleisliInterpreter.$anonfun$primitive$2(kleisliinterpreter.scala:113) ~[doobie-free_2.13-1.0.0-RC2.jar:1.0.0-RC2]
	at zio.ZIOCompanionVersionSpecific.$anonfun$attempt$1(ZIOCompanionVersionSpecific.scala:100) ~[zio_2.13-2.0.6.jar:2.0.6]
	at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:1111) ~[zio_2.13-2.0.6.jar:2.0.6]
	at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:1068) ~[zio_2.13-2.0.6.jar:2.0.6]
	at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:1011) ~[zio_2.13-2.0.6.jar:2.0.6]
	at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:934) ~[zio_2.13-2.0.6.jar:2.0.6]
	at zio.internal.FiberRuntime.evaluateEffect(FiberRuntime.scala:389) ~[zio_2.13-2.0.6.jar:2.0.6]
	at zio.internal.FiberRuntime.evaluateMessageWhileSuspended(FiberRuntime.scala:512) ~[zio_2.13-2.0.6.jar:2.0.6]
	at zio.internal.FiberRuntime.drainQueueOnCurrentThread(FiberRuntime.scala:229) ~[zio_2.13-2.0.6.jar:2.0.6]
	at zio.internal.FiberRuntime.run(FiberRuntime.scala:138) ~[zio_2.13-2.0.6.jar:2.0.6]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
	at java.base/java.lang.Thread.run(Thread.java:829) ~[na:na]

If I remove the manual creation of the QueryMeta instance and rely on the macro, then everything works normally.

[jilen]

@lachezar You may need things like --add-exports or add-open , the full stacktrace should show the java9+ module errors.

Or just try java8 instead.

[guizmaii]

@jilen I ran the compilation of my "db" sbt module (containing all my Quill code) with the -Yprofile-trace flag.
What takes the most time to compile are my tests, which are containing a lot of queries (aka ctx.run(myQuery)), and apparently especially one file, TestUtils.scala, which contains all the instances of my case classes representing my DB data (Org_A, Org_B, User_0, etc. are case classes instances) is taking the most time to compile with two "apply" calls. Not sure to understand why there are 2 "apply" calls 🤔