Skip to content

Writing a Driver: Packaging a Driver & Metabase Plugin Basics

Jump to bottom
Cam Saul edited this page Dec 20, 2018 · 2 revisions

Back to the Table of Contents

How are Metabase drivers packaged?

Besides the namespaces that contain a driver's method implementations, most drivers require 3rd-party dependencies, such as a JDBC driver. There are essentially three ways a driver's namespaces and its dependencies can be packaged together:

  • Built-in to the core Metabase project
  • In the same repository as the core Metabase project, but built as separate plugins; the plugins are bundled into the uberjar and extracted into the plugins directory on launch
  • In a different repository, and built as a separate plugin. To use this driver, you must manually copy the plugin into your plugins directory

We'll look at each of those options in future detail in a second, but first, let's cover some basics.

Metabase Plugins

Metabase 0.32 and above ship with a new plugins system that allow plugins to be added to Metabase without rebuilding Metabase itself. A Metabase plugin is a JAR file containing compiled class files and a Metabase plugin manifest. In most cases, plugins are lazily loaded; for drivers, this means they are not initialized until connecting to a database of that type.

Plugins Dir

All plugins live in the Metabase plugins directory, which defaults to the ./plugins in the same directory as metabase.jar, or project.clj when running with Leiningen. For example, your directory structure might look like this:

/Users/cam/metabase/metabase.jar
/Users/cam/metabase/plugins/my-plugin.jar

The plugins directory can be changed by setting the env var MB_PLUGINS_DIR.

Plugin Manifests

Metabase plugin JARs contain a plugin manifest -- a top-level file named metabase-plugin.yaml. When Metabase launches, it iterates over every JAR in the plugins directory, and looks for this file in each; this file tells Metabase what the plugin provides, and how to initialize it. Here's a sample plugin manifest:

info:
  name: Metabase SQLite Driver
  version: 1.0.0-SNAPSHOT-3.25.2
  description: Allows Metabase to connect to SQLite databases.
driver:
  name: sqlite
  display-name: SQLite
  lazy-load: true
  parent: sql-jdbc
  connection-properties:
    - name: db
      display-name: Filename
      placeholder: /home/camsaul/toucan_sightings.sqlite
      required: true
init:
  - step: load-namespace
    namespace: metabase.driver.sqlite
  - step: register-jdbc-driver
    class: org.sqlite.JDBC

Most of these details should be self explanatory. In particular, we're interested in the driver section -- this tells Metabase that the plugin defines a driver named :sqlite, that has :sql-jdbc as a parent; Metabase's plugin system calls driver/register! using these details. The plugin also lists the display name and connection properties for the driver; the plugin system creates implementations for driver/display-name and driver/connection-properties, respectively, using this information.

Take a look at the complete Metabase Plugin Manifest Reference for more details -- every option is listed and explained in depth, and I'd rather not repeat all of that here. There are many more options you can specify, including defining requirements on other plugins or certain classes.

Lazy Loading

Note that the driver in the example above is listed as lazy-load: true -- this means that while the method implementation mentioned above are created when Metabase launches, actually initializing the driver is not done until the first time someone attempts to connect to a database of that type. (A driver can also be lazy-load: false, but this will simply make Metabase take longer to launch and eat up more memory. Don't do this.)

Plugin Initialization

Plugin initialization is handled automatically when appropriate. When initializing a plugin, Metabase first adds the to the classpath, then it performs any of the steps listed in the init section of the plugin manifest, in order. In this manifest, there are two steps, a load-namespace step, and a register-jdbc-driver step:

init:
  - step: load-namespace
    namespace: metabase.driver.sqlite
  - step: register-jdbc-driver
    class: org.sqlite.JDBC

Loading Namespaces

For your driver, you'll need to add one or more load-namespace steps to tell Metabase which namespaces contain your driver method implementations -- for the example in Chapter 1, we'd need one for com.mycompany.metabase.driver.foxpro98. load-namespace calls require the normal Clojure way, meaning it will load other namespaces listed in the :require section of its namespace declaration as needed. If your driver's method implementations are split across multiple namespaces, make sure they'll get loaded as well -- you can either have the main namespace handle this (e.g. by including them in the :require form in the namespace declaration) or by adding additional load-namespace steps.

Registering JDBC Drivers

Drivers that use a JDBC driver under the hood will need to add a register-jdbc-driver step as well.

For those interested: a technical explanation of why you must register JDBC drivers this way

Java's JDBC DriverManager will not use JDBC drivers loaded with something other than the system ClassLoader, which effectively only means it will only use JDBC driver classes that are packaged as part of the core Metabase uberjar. Since the system classloader does not allow you to the classpath at runtime, Metabase uses a custom ClassLoader to initialize plugins. Thus the DriverManager any JDBC drivers loaded from these plugins won't work directly. However, we ship a JDBC proxy driver class as part of Metabase itself that can wrap other JDBC drivers; when calling register-jdbc-driver, Metabase actually registers a new instance of the proxy class that forwards method calls to the actual JDBC driver. DriverManager is perfectly fine with this.

Further Reading

For more details on the internals of Metabase plugins, the pull request where I introduced the feature offers an excellent overview of how things work. Take a look if you would like to learn more.

Driver Initialization with metabase.driver/initialize!

All Drivers, even ones not packaged as part of plugins, can include additional code to be executed once and only once when the driver is first initialized (before the first time a connection to a DB of that type is made) by implementing metabase.driver/initialize!. (In fact, this is how driver lazy loading is itself implemented.) There are few cases where you should actually need to do this, but if you need to allocate some sort of resources or set certain system properties, you can do it here.

Different Ways to Ship Drivers

Now that we understand what Metabase plugins are, let's look at the different ways you can ship Metabase drivers:

Drivers built-in to the core Metabase project

This is the simplest method of shipping drivers; it's used for the :postgres, :h2, and :mysql drivers, as well as common parents like :sql and :sql-jdbc. (In fact, before Metabase 0.32, all drivers were shipped this way.)

With this method, dependencies (i.e., JDBC drivers) are included in the core project's project.clj, and the drivers themselves are in the found in the same place all other Metabase source is. The file layout will look something like:

metabase/project.clj                         ; <- deps go in here
metabase/src/metabase/driver/mysql.clj       ; <- main driver namespace
metabase/test/metabase/test/data/mysql.clj   ; <- test extensions
metabase/test/metabase/driver/mysql_test.clj ; <- driver-specific tests go here

The only reason these drivers are shipped this way is that these three databases are also supported as application databases, meaning their dependencies would be part of the core Metabase project anyway. When writing a driver for 3rd-party consumption, or one you hope to have merged into the core Metabase project, DO NOT write your driver this way. There are a lot of good reasons to write drivers as separate plugins -- for one, lazy loading improves launch speed and memory consumption.

It might be helpful to start writing a driver this way, since you don't need to work about writing a plugin manifest; but before shipping it, you'll have to move things around to support one of the other delivery methods mentioned below. Thus I'd recommend against starting a driver this way, even for exploratory purposes. We will not accept any pull requests for drivers packaged this way (i.e., built in to the core product).

The only situation where you'd might consider shipping a driver this way is as part of a custom fork of Metabase, perhaps because it's only intended for in-house use. (Even if you'll still have to publish the source for it to comply with the AGPL).

Drivers shipped as part of the core Metabase repo, but packaged as plugins

Most drivers that ship as part of Metabase are shipped this way, and if you want to submit your driver as a PR, this is the only way we'll accept.

A typical file layout looks something like:

metabase/modules/drivers/mongo/project.clj                         ; <- deps go in here
metabase/modules/drivers/mongo/resources/metabase-plugin.yaml      ; <- plugin manifest
metabase/modules/drivers/mongo/src/metabase/driver/mongo.clj       ; <- main driver namespace
metabase/modules/drivers/mongo/test/metabase/test/data/mongo.clj   ; <- test extensions
metabase/modules/drivers/mongo/test/metabase/driver/mongo_test.clj ; <- driver-specific tests go here

Note that the structure is fundamentally the same as that of built-in drivers, but everything has been moved into a modules/drivers/<driver-name> subdirectory, and a plugin manifest file is added. All drivers packaged this way must follow that directory structure.

project.clj

With this method, drivers are actually a separate Leiningen project, albeit one in the same Git repository as the core Metabase project. As a separate Leiningen project, it must have a separate project.clj; here's Mongo's, for example:

(defproject metabase/mongo-driver "1.0.0-3.5.0"
  :min-lein-version "2.5.0"

  :dependencies
  [[com.novemberain/monger "3.5.0"]]

  :profiles
  {:provided
   {:dependencies [[metabase-core "1.0.0-SNAPSHOT"]]}

   :uberjar
   {:auto-clean    true
    :aot           :all
    :javac-options ["-target" "1.8", "-source" "1.8"]
    :target-path   "target/%s"
    :uberjar-name  "mongo.metabase-driver.jar"}})

Not that it includes the dependencies (monger) for the driver as well as a dependency on the metabase-core project (we'll explain this more in a second) as well as a profile for building the uberjar. The version is 1.0.0-3.5.0 -- the formula I've used here is <actual-driver-version>-<dependencies-version>, but you can use whatever version numbers you feel appropriate; just know the plugin system assumes semantic versioning (e.g. 1.10 is newer than 1.2).

Installing metabase-core locally

The dependency on metabase-core makes all namespaces that are part of the core Metabase project (e.g. metabase.driver) available for use in the driver itself. By putting this dependency in a provided profile, lein uberjar won't include that dependency in the built driver.

Note that Metabase is not currently available in Clojars or other plugin repositories -- you'll have to install it locally before working on a driver. You can do this by running

lein install-for-building-drivers

from the root of the core Metabase repository. For now, metabase-core has one version -- 1.0.0-SNAPSHOT -- so this is what your driver should specify. As APIs get locked down in the near future and we ship a Metabase 1.0 release, we'll ship real [metabase-core "1.0.0"] (and so forth) dependencies, and most likely publish them on Clojars, meaning you'll be able to skip this step; for now, stick to [metabase-core "1.0.0-SNAPSHOT"]. I'll update this guide when this changes.

Building a driver plugin shipped as part of the core Metabase repo

A helpful script is included as part of Metabase to build drivers packaged this way:

./bin/build-driver.sh mongo

This will take care of everything and copy the resulting file to ./resources/modules/mongo.metabase-driver.jar. You can also build the JAR using

LEIN_SNAPSHOTS_IN_RELEASE=true lein uberjar

from the modules/drivers/mongo directory; you'll have to copy it into resources/modules yourself to have it included with the Metabase uberjar if you're building it locally.

Drivers shipped this way are bundled up inside the uberjar under the modules directory (anything in resources gets included in the uberjar); anything JARs in the modules/ directory of the uberjar is extracted into the plugins directory when Metabase starts.

Working with the driver from the REPL and in CIDER

Having to install metabase-core locally, and build driver uberjars would be obnoxious, especially if you had to repeat it to test every change. Luckily, you can use an included Leiningen profile, include-all-drivers, to merge the driver's source paths, test paths, and dependencies into the core Metabase project, letting you run commands as if everything was part of one giant project:

lein with-profiles +include-all-drivers repl

This currently works for a variety of tasks, such as repl, test, and our various linters. Note it is not currently set up to work when running from source (i.e. with lein run or lein ring server) -- you'll need to rebuild the driver and install it in your ./plugins directory instead, and restart when you make changes. This may be fixed in the future, but in the meantime if you want to avoid the slow feedback loop, consider developing your driver using an interactive REPL such as CIDER instead (discussed below), or developing your driver as a "built-in" driver as described above and repackaging it as plugin once everything is finished.

When developing with Emacs and CIDER sending the universal prefix argument to cider-jack-in (i.e. running it with C-u M-x cider-jack-in) will prompt you for the command it should use to start the NREPL process; you can add with-profiles +include-all-drivers to the beginning of the command to include source paths for your driver.

Of course, you can also work on the driver directly from its modules/drivers/<driver> directory -- just note that you won't be able to run tests from that directory, or work on them -- driver test extensions require code in metabase/test, which is not bundled up with metabase-core; the only way for your driver to have access to the namespaces is to use with-profiles +include-all-drivers to simulate an uber-project.

Drivers shipped as 3rd-party plugins

Package a driver this way if you plan on shipping it as a plugin and don't plan on submitting it as a PR. Fundamentally, the structure is similar to plugins shipped as part of Metabase, but in a separate repo rather than the modules/drivers/ directory, and without test extensions or tests (at least, without ones that piggyback off the core project's test functionality):

./project.clj                    ; <- deps go in here
./resources/metabase-plugin.yaml ; <- plugin manifest
./src/metabase/driver/sudoku.clj ; <- main driver namespace

Building a driver like this is largely the same as plugins shipped as part of Metabase -- install metabase-core locally, then build the driver using lein uberjar. Copy the resulting JAR file into your plugins directory, and you're off to the races.

Chapter Summary

  • There are three ways to package a Metabase driver:
    • built-in to the core project
    • as a separate plugin in the same repo as, and bundled with, the core project
    • as a separate plugin shipped separately
  • Plugins are JARs that live in the plugins directory, by default ./plugins, or set with MB_PLUGINS_DIR
  • Every plugin has a manifest called metabase-plugin.yaml
  • If you want to submit your driver as a PR, write it as a separate plugin in the same repo as the core project, because we won't accept PRs for built-in drivers

Next Up

Chapter 3: Implementing metabase.driver methods

Metabase Logo

Clone this wiki locally