Skip to content

Migrating from Leiningen to tools.deps

Jump to bottom
Alexander Solovyov edited this page Oct 11, 2023 · 26 revisions

PR https://github.com/metabase/metabase/pull/16749 migrates from Leiningen as our Clojure deps/build tooling to tools.deps/tools.build/Depstar. This dramatically improves REPL launch time as well as driver and uberjar build time. This page details differences in dev workflow once the PR lands.

Note for FE people who only run stuff in ./bin (e.g. ./bin/build):

You only need to upgrade your CLI version. The scripts in ./bin should take care of everything else for you automatically. You can read the next section and skip the rest of this document.

Note for people who usually just used lein run:

You need to upgrade your CLI version and run the prep steps to get things ready. After that, you can do clojure -M:run (see Running Commands for more information).

Installing the Latest Clojure CLI Version

You no longer need to install Leiningen to hack on the Metabase backend codebase or build the uberjar. Instead, you must install a newer version of the Clojure CLI. At the time of this writing, 1.10.3.933 is the latest version; you can check what version you have locally by running

clojure --help | grep Version

Most of our scripts in ./bin/ (e.g. ./bin/build) will check this for you and fail if your version isn't current enough. If you don't have at least 1.10.3.905, you should install the latest version by following the instructions at https://clojure.org/guides/getting_started.

Preparing Dependencies

The Clojure CLI cannot currently automatically run :java-source-paths or :aot compilation steps for you the way Leiningen can; however the current version can run custom "prep" steps for you with clojure -X:deps prep.

Our lone Java depenendency needs to be compiled this way, and two Spark SQL :gen-class namespaces need to be AOT compiled; you can do this by running

clojure -X:deps prep 
cd modules/drivers
clojure -X:deps prep
cd ../..

You must do this before running any other clojure commands in the repository, but you only need to do this once. In the future, it should not be necessary to run clojure -X:deps prep in two different folders (Alex Miller has said he is working on a fix to let you prepare dependencies for aliases automatically). I will update these docs when that lands.

Various scripts in ./bin/ (e.g. ./bin/build) will do this for you automatically.

Aliases

Clojure CLI aliases are the equivalent of Leiningen profiles. The biggest difference between the two is that deps.edn aliases cannot be composed declaratively, e.g. the :test alias cannot automatically include the :dev alias. This is an open issue in ask.clojure.org; please upvote https://ask.clojure.org/index.php/10564/specify-an-alias-that-is-a-set-of-other-aliases?show=10564.

This table lists the various aliases mean to be composed with other aliases.

Alias Purpose
:dev Includes dependencies used in test code and linters, and OSS test source paths. For REPL-based development.
:ci Adds some CI-specific JVM flags.
:ee Adds the EE namespaces to the classpath (excluding test namespaces).
:ee-dev Adds the EE test namespaces to the classpath.
:drivers Adds sources for drivers in modules/drivers to classpath (excluding tests). For local dev without building drivers.
:drivers-dev Adds sources for driver tests in modules/drivers to classpath. For running tests against drivers.

Running Commands

Various lein commands are replaced with clojure equivalents. Refer to the aliases table above to see why things are composed the way they are.

Here is a table of common lein commands and Clojure CLI equivalents:

Description lein command clojure command
Run local dev server (OSS) lein run clojure -M:run
Run local dev server (EE) lein with-profile +ee run clojure -M:run:ee
Run local dev server (OSS + drivers) lein with-profile +include-all-drivers run clojure -M:run:drivers
Run tests (OSS) lein test clojure -X:dev:test
Run tests (EE) lein with-profile +ee test clojure -X:dev:ee:ee-dev:test
Run driver tests (OSS) DRIVERS=postgres lein test DRIVERS=postgres clojure -X:dev:drivers:drivers-dev:test
Run driver tests (EE) DRIVERS=postgres lein with-profile +ee test DRIVERS=postgres clojure -X:dev:ee:ee-dev:drivers:drivers-dev:test
Start REPL (OSS + tests) lein repl clojure -A:dev
Start REPL (EE + tests) lein with-profile +ee repl clojure -A:dev:ee:ee-dev
Start REPL (OSS + tests + drivers + driver tests) lein with-profile +include-all-drivers repl clojure -A:dev:drivers:drivers-dev
Start nREPL (OSS + tests) lein repl clojure -M:dev:nrepl
Run namespace-checker linter lein check-namespace-decls clojure -X:dev:ee:ee-dev:drivers:drivers-dev:namespace-checker
Compile + check all namespaces lein check clojure -M:dev:ee:ee-dev:drivers:drivers-dev:check
Run Eastwood linter lein eastwood clojure -X:dev:ee:ee-dev:drivers:drivers-dev:eastwood
Run Eastwood linter on a single namespace clj -X:dev:ee:ee-dev:drivers:drivers-dev:eastwood :namespaces '[metabase.util]'
Run whitespace linter clojure -T:whitespace-linter lint
Run clj-kondo clj-kondo --parallel --lint src/ shared/src enterprise/backend/src --config lint-config.edn (if installed) or clojure -Sdeps '{:deps {clj-kondo/clj-kondo {:mvn/version "RELEASE"}}}' -m clj-kondo.main --config lint-config.edn --parallel --lint src/ shared/src enterprise/backend/src (slower; no installation required)
Run Cloverage lein cloverage clojure -X:dev:ee:ee-dev:test:cloverage
Build Uberjar (does not build FE/i18n/etc.) (OSS) lein uberjar clojure -T:build uberjar
Build Uberjar (EE) lein with-profile +ee uberjar clojure -T:build uberjar :edition :ee
Run H2 shell lein h2 clojure -M:h2

Custom env var values

Instead of ~/.lein/profiles.clj, you can set equivalent JVM system property flags by adding an alias to ~/.clojure/deps.edn. Environ will automatically convert JVM properties like mb.db.host to equivalent keys like :mb-db-host in the environ.core/env map. Here is an example ~/.clojure/deps.edn:

{:aliases
 {:user
  {:jvm-opts
   ["-Dmb.db.host=localhost"
    "-Dmb.db.user=cam"
    "-Dmb.db.dbname=metabase"
    "-Dmb.db.pass=cam"
    ;; Postgres
    "-Dmb.db.type=postgres"
    "-Dmb.db.port=5432"
    ;; test DBs
    "-Dmb.oracle.test.host=<REDACTED>"
    "-Dmb.oracle.test.password=<REDACTED>"
    "-Dmb.oracle.test.user=<REDACTED>"
    "-Dmb.oracle.test.sid=ORCL"]}}}

You must add the :user profile to the commands above. I configured the default CIDER command in .dir-locals.el to include the :user automatically; please feel free to tweak editor config files for other editors to do the same.

Environ will attempt to read the .lein-env file (Git-ignored by default) if one is present. lein-environ plugin will have created this file previously if you've ran with lein before. Be sure to delete this file if present so it doesn't ignore the properties you specify.

Note Environ preference order is .lein-env, env variables, system properties. E.g. MB_DB_TYPE=h2 ... will not override a -Dmb.db.type=postgres flag from a :user profile. They easy way to fix this is to create another profile specifying alternative values, e.g.

{:aliases
 {:user
  {:jvm-opts
   ["-Dmb.db.host=localhost"
    "-Dmb.db.user=cam"
    "-Dmb.db.dbname=metabase"
    "-Dmb.db.type=postgres"
    "-Dmb.db.port=5432"
    ;; ... a bunch of other stuff ....
   ]}

  :user/h2
  {:jvm-opts
   ["-Dmb.db.type=h2"]}}}

then compose profiles when running commands, e.g.

# run tests with Postgres
clojure -X:dev:test:user

# run tests with H2 (override :mb-db-type :postgres)
clojure -X:dev:test:user:user/h2

Running specific tests

You can run tests against a single namespace or directory, or one test specifically, by passing :only [argument]:

Arguments to clojure -X are read in as EDN; for things other than plain symbols or numbers you usually need to wrap them in single quotes in your shell. Our test runner uses this argument to determine where to look for tests. Here's how different EDN forms are interpreted as our test runner:

Arg type Example Description
Unqualified Symbol my.namespace-test Run all tests in this namespace
Qualified Symbol my.namespace-test/my-test Run one specific test
String '"test/metabase/api"' Run all tests in test namespaces in this directory (including subdirectories)
Vector of symbols/strings '[my.namespace "test/metabase/some_directory"]' Union of tests found by the individual items in the vector

Example commands:

Description Example
Run tests in a specific namespace clojure -X:dev:test :only my.namespace-test
Run a specific test clojure -X:dev:test :only my.namespace-test/my-test
Run tests in a specific directory (including subdirectories) clojure -X:dev:test :only '"test/metabase/api"'
Run tests in 2 namespaces clojure -X:dev:test :only '[my.namespace-test my.other.namespace-test]'

Running Tests from the REPL

You can run tests with out test runner from the REPL by invoking the underlying test runner functions directly:

;; Load the test runner
(require 'metabase.test-runner)

;; Run a specific test
(require 'metabase.public-settings-test)
(metabase.test-runner/run [#'metabase.public-settings-test/site-locale-test])

;; Run tests in a specific namespace (`find-tests` accepts various arg types listed above)
(metabase.test-runner/run (metabase.test-runner/find-tests 'metabase.public-settings-test))

JVM Arguments

To pass JVM arguments (ex: for remote debugging, heap size, etc.), use the -J argument to clojure, which can be given multiple times for multiple parameters. Ex:

clojure -J-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=15375 -M:<aliases>

See the official docs for full details.

CIDER development

The default aliases used for CIDER-based development (specified in .dir-locals.el) are :dev:drivers:drivers-dev:ee:ee-dev:user, e.g.

clojure -M:dev:drivers:drivers-dev:ee:ee-dev:user:cider/nrepl

Note that this differs from the previous Leiningen defaults in that it includes EE sources by default. You can exclude EE sources by connecting with C-u M-x cider-jack-in and removing the :ee and :ee-dev aliases.

Important Debugging Note

If you run into failures when running commands (such as "class not found" errors) you may need to clear the local classpath cache files. You can do this by running

for file in `find . -name .cpcache`; do rm -rf "$file"; done

(You can also pass the -Sforce command to clojure to ignore the cached classpaths; however it's better just to clear them out and have it recalculate the correct ones.)

The various scripts in ./bin will clear outdated .cpcache directories for you automatically.

Known Caveats

  • JUnit output in CI is not as detailed as our JUnit output in our old test runner; I'll fix this as a follow-on. fixed
  • The parallel test runner is disabled for now until I work out a few kinks related to data warehouse DB connection pools getting nuked by other threads.

Metabase Logo

Clone this wiki locally