Writing a Driver: Packaging a Driver & Metabase Plugin Basics
Besides the namespaces that contain a driver's method implementations, most drivers require 3rd-party dependencies, such as a JDBC driver. There are essentially three ways a driver's namespaces and its dependencies can be packaged together:
- Built-in to the core Metabase project
- In the same repository as the core Metabase project, but built as separate plugins; the plugins are bundled into the uberjar and extracted into the
plugins
directory on launch - In a different repository, and built as a separate plugin. To use this driver, you must manually copy the plugin into your
plugins
directory
We'll look at each of those options in future detail in a second, but first, let's cover some basics.
Metabase 0.32 and above ship with a new plugins system that allow plugins to be added to Metabase without rebuilding Metabase itself. A Metabase plugin is a JAR file containing compiled class files and a Metabase plugin manifest. In most cases, plugins are lazily loaded; for drivers, this means they are not initialized until connecting to a database of that type.
All plugins live in the Metabase plugins directory, which defaults to the ./plugins
in the same directory as metabase.jar
, or project.clj
when running with Leiningen. For example, your directory structure might look like this:
/Users/cam/metabase/metabase.jar
/Users/cam/metabase/plugins/my-plugin.jar
The plugins directory can be changed by setting the env var MB_PLUGINS_DIR
.
Metabase plugin JARs contain a plugin manifest -- a top-level file named metabase-plugin.yaml
. When Metabase launches, it iterates over every JAR in the plugins directory, and looks for this file in each; this file tells Metabase what the plugin provides, and how to initialize it. Here's a sample plugin manifest:
info:
name: Metabase SQLite Driver
version: 1.0.0-SNAPSHOT-3.25.2
description: Allows Metabase to connect to SQLite databases.
driver:
name: sqlite
display-name: SQLite
lazy-load: true
parent: sql-jdbc
connection-properties:
- name: db
display-name: Filename
placeholder: /home/camsaul/toucan_sightings.sqlite
required: true
init:
- step: load-namespace
namespace: metabase.driver.sqlite
- step: register-jdbc-driver
class: org.sqlite.JDBC
Most of these details should be self explanatory. In particular, we're interested in the driver
section -- this tells Metabase that the plugin defines a driver named :sqlite
, that has :sql-jdbc
as a parent; Metabase's plugin system calls driver/register!
using these details. The plugin also lists the display name and connection properties for the driver; the plugin system creates implementations for driver/display-name
and driver/connection-properties
, respectively, using this information.
Take a look at the complete Metabase Plugin Manifest Reference for more details -- every option is listed and explained in depth, and I'd rather not repeat all of that here. There are many more options you can specify, including defining requirements on other plugins or certain classes.
Note that the driver in the example above is listed as lazy-load: true
-- this means that while the method implementation mentioned above are created when Metabase launches, actually initializing the driver is not done until the first time someone attempts to connect to a database of that type. (A driver can also be lazy-load: false
, but this will simply make Metabase take longer to launch and eat up more memory. Don't do this.)
Plugin initialization is handled automatically when appropriate. When initializing a plugin, Metabase first adds the to the classpath, then it performs any of the steps listed in the init
section of the plugin manifest, in order. In this manifest, there are two steps, a load-namespace
step, and a register-jdbc-driver
step:
init:
- step: load-namespace
namespace: metabase.driver.sqlite
- step: register-jdbc-driver
class: org.sqlite.JDBC
For your driver, you'll need to add one or more load-namespace
steps to tell Metabase which namespaces contain your driver method implementations -- for the example in Chapter 1, we'd need one for com.mycompany.metabase.driver.foxpro98
. load-namespace
calls require
the normal Clojure way, meaning it will load other namespaces listed in the :require
section of its namespace declaration as needed. If your driver's method implementations are split across multiple namespaces, make sure they'll get loaded as well -- you can either have the main namespace handle this (e.g. by including them in the :require
form in the namespace declaration) or by adding additional load-namespace
steps.
Drivers that use a JDBC driver under the hood will need to add a register-jdbc-driver
step as well.
Java's JDBC DriverManager
will not use JDBC drivers loaded with something other than the system ClassLoader
, which effectively only means it will only use JDBC driver classes that are packaged as part of the core Metabase uberjar. Since the system classloader does not allow you to the classpath at runtime, Metabase uses a custom ClassLoader
to initialize plugins. Thus the DriverManager any JDBC drivers loaded from these plugins won't work directly. However, we ship a JDBC proxy driver class as part of Metabase itself that can wrap other JDBC drivers; when calling register-jdbc-driver
, Metabase actually registers a new instance of the proxy class that forwards method calls to the actual JDBC driver. DriverManager
is perfectly fine with this.
For more details on the internals of Metabase plugins, the pull request where I introduced the feature offers an excellent overview of how things work. Take a look if you would like to learn more.
All Drivers, even ones not packaged as part of plugins, can include additional code to be executed once and only once when the driver is first initialized (before the first time a connection to a DB of that type is made) by implementing metabase.driver/initialize!
. (In fact, this is how driver lazy loading is itself implemented.) There are few cases where you should actually need to do this, but if you need to allocate some sort of resources or set certain system properties, you can do it here.
Now that we understand what Metabase plugins are, let's look at the different ways you can ship Metabase drivers:
This is the simplest method of shipping drivers; it's used for the :postgres
, :h2
, and :mysql
drivers, as well as common parents like :sql
and :sql-jdbc
. (In fact, before Metabase 0.32, all drivers were shipped this way.)
With this method, dependencies (i.e., JDBC drivers) are included in the core project's project.clj
, and the drivers themselves are in the found in the same place all other Metabase source is. The file layout will look something like:
metabase/project.clj ; <- deps go in here
metabase/src/metabase/driver/mysql.clj ; <- main driver namespace
metabase/test/metabase/test/data/mysql.clj ; <- test extensions
metabase/test/metabase/driver/mysql_test.clj ; <- driver-specific tests go here
The only reason these drivers are shipped this way is that these three databases are also supported as application databases, meaning their dependencies would be part of the core Metabase project anyway. When writing a driver for 3rd-party consumption, or one you hope to have merged into the core Metabase project, DO NOT write your driver this way. There are a lot of good reasons to write drivers as separate plugins -- for one, lazy loading improves launch speed and memory consumption.
It might be helpful to start writing a driver this way, since you don't need to work about writing a plugin manifest; but before shipping it, you'll have to move things around to support one of the other delivery methods mentioned below. Thus I'd recommend against starting a driver this way, even for exploratory purposes. We will not accept any pull requests for drivers packaged this way (i.e., built in to the core product).
The only situation where you'd might consider shipping a driver this way is as part of a custom fork of Metabase, perhaps because it's only intended for in-house use. (Even if you'll still have to publish the source for it to comply with the AGPL).
Most drivers that ship as part of Metabase are shipped this way, and if you want to submit your driver as a PR, this is the only way we'll accept.
A typical file layout looks something like:
metabase/modules/drivers/mongo/project.clj ; <- deps go in here
metabase/modules/drivers/mongo/resources/metabase-plugin.yaml ; <- plugin manifest
metabase/modules/drivers/mongo/src/metabase/driver/mongo.clj ; <- main driver namespace
metabase/modules/drivers/mongo/test/metabase/test/data/mongo.clj ; <- test extensions
metabase/modules/drivers/mongo/test/metabase/driver/mongo_test.clj ; <- driver-specific tests go here
Note that the structure is fundamentally the same as that of built-in drivers, but everything has been moved into a modules/drivers/<driver-name>
subdirectory, and a plugin manifest file is added. All drivers packaged this way must follow that directory structure.
With this method, drivers are actually a separate Leiningen project, albeit one in the same Git repository as the core Metabase project. As a separate Leiningen project, it must have a separate project.clj
; here's Mongo's, for example:
(defproject metabase/mongo-driver "1.0.0-3.5.0"
:min-lein-version "2.5.0"
:dependencies
[[com.novemberain/monger "3.5.0"]]
:profiles
{:provided
{:dependencies [[metabase-core "1.0.0-SNAPSHOT"]]}
:uberjar
{:auto-clean true
:aot :all
:javac-options ["-target" "1.8", "-source" "1.8"]
:target-path "target/%s"
:uberjar-name "mongo.metabase-driver.jar"}})
Not that it includes the dependencies (monger
) for the driver as well as a dependency on the metabase-core
project (we'll explain this more in a second) as well as a profile for building the uberjar. The version is 1.0.0-3.5.0
-- the formula I've used here is <actual-driver-version>-<dependencies-version>
, but you can use whatever version numbers you feel appropriate; just know the plugin system assumes semantic versioning (e.g. 1.10
is newer than 1.2
).
The dependency on metabase-core
makes all namespaces that are part of the core Metabase project (e.g. metabase.driver
) available for use in the driver itself. By putting this dependency in a provided
profile, lein uberjar
won't include that dependency in the built driver.
Note that Metabase is not currently available in Clojars or other plugin repositories -- you'll have to install it locally before working on a driver. You can do this by running
lein install-for-building-drivers
from the root of the core Metabase repository. For now, metabase-core
has one version -- 1.0.0-SNAPSHOT
-- so this is what your driver should specify. As APIs get locked down in the near future and we ship a Metabase 1.0 release, we'll ship real [metabase-core "1.0.0"]
(and so forth) dependencies, and most likely publish them on Clojars, meaning you'll be able to skip this step; for now, stick to [metabase-core "1.0.0-SNAPSHOT"]
. I'll update this guide when this changes.
A helpful script is included as part of Metabase to build drivers packaged this way:
./bin/build-driver.sh mongo
This will take care of everything and copy the resulting file to ./resources/modules/mongo.metabase-driver.jar
. You can also build the JAR using
LEIN_SNAPSHOTS_IN_RELEASE=true lein uberjar
from the modules/drivers/mongo
directory; you'll have to copy it into resources/modules
yourself to have it included with the Metabase uberjar if you're building it locally.
Drivers shipped this way are bundled up inside the uberjar under the modules
directory (anything in resources
gets included in the uberjar); anything JARs in the modules/
directory of the uberjar is extracted into the plugins directory when Metabase starts.
Having to install metabase-core
locally, and build driver uberjars would be obnoxious, especially if you had to repeat it to test every change. Luckily, you can use an included Leiningen profile, include-all-drivers
, to merge the driver's source paths, test paths, and dependencies into the core Metabase project, letting you run commands as if everything was part of one giant project:
lein with-profiles +include-all-drivers repl
This currently works for a variety of tasks, such as repl
, test
, and our various linters. Note it is not currently set up to work when running from source (i.e. with lein run
or lein ring server
) -- you'll need to rebuild the driver and install it in your ./plugins
directory instead, and restart when you make changes. This may be fixed in the future, but in the meantime if you want to avoid the slow feedback loop, consider developing your driver using an interactive REPL such as CIDER instead (discussed below), or developing your driver as a "built-in" driver as described above and repackaging it as plugin once everything is finished.
When developing with Emacs and CIDER sending the universal prefix argument to cider-jack-in
(i.e. running it with C-u M-x cider-jack-in
) will prompt you for the command it should use to start the NREPL process; you can add with-profiles +include-all-drivers
to the beginning of the command to include source paths for your driver.
Of course, you can also work on the driver directly from its modules/drivers/<driver>
directory -- just note that you won't be able to run tests from that directory, or work on them -- driver test extensions require code in metabase/test
, which is not bundled up with metabase-core
; the only way for your driver to have access to the namespaces is to use with-profiles +include-all-drivers
to simulate an uber-project.
Package a driver this way if you plan on shipping it as a plugin and don't plan on submitting it as a PR. Fundamentally, the structure is similar to plugins shipped as part of Metabase, but in a separate repo rather than the modules/drivers/
directory, and without test extensions or tests (at least, without ones that piggyback off the core project's test functionality):
./project.clj ; <- deps go in here
./resources/metabase-plugin.yaml ; <- plugin manifest
./src/metabase/driver/sudoku.clj ; <- main driver namespace
Building a driver like this is largely the same as plugins shipped as part of Metabase -- install metabase-core
locally, then build the driver using lein uberjar
. Copy the resulting JAR
file into your plugins directory, and you're off to the races.
- There are three ways to package a Metabase driver:
- built-in to the core project
- as a separate plugin in the same repo as, and bundled with, the core project
- as a separate plugin shipped separately
- Plugins are JARs that live in the plugins directory, by default
./plugins
, or set withMB_PLUGINS_DIR
- Every plugin has a manifest called
metabase-plugin.yaml
- If you want to submit your driver as a PR, write it as a separate plugin in the same repo as the core project, because we won't accept PRs for built-in drivers
- Backend
- Metabase Developer Reference
- Product Management
- QA and Testing
- Writing A Driver
- Driver Notices
- REST API Notices
- Writing style guide for documentation and blog posts (WIP)