Writing a Driver: The Basics

Back to the Table of Contents

Familiarizing yourself with Clojure

Metabase drivers are written in Clojure, like Metabase itself. If you have no experience writing Clojure, I highly recommend you take a bit of time and familiarize yourself with the language a bit before diving in. If you're really ambitious, the book Clojure for the Brave and True comes highly recommended, and it's free online. If you don't feel like reading a whole book, Mark Volkmann's Clojure tutorial is another good starting point.

Having an in-depth understanding of Clojure is less important when writing JDBC-based drivers because their implementation is simpler -- much of the work is already for you -- but it would still be helpful to understand what things like multimethods are.

What does a Metabase driver do?

Let's start with what a Metabase driver does. At its core, a Metabase driver does four things:

• Provides basic information such as the capabilities of the database and the connection properties to ask for we someone tries to add a database of that type
• Provides information about the schema of the database -- the tables (or equivalent), columns in those tables, and foreign key relationships, for databases that support foreign keys.
  • This functionality is used by the Metabase sync process and stored in the application database
  • The stored information is used in the visual Query Builder and other places to show users what tables/columns/etc. are available
• Compiles our in-house query language, MBQL, into native queries.
  • MBQL queries are generated by the visual Query Builder
  • The Metabase query processor uses this functionality to convert MBQL queries to native queries
• Executes native queries and returns results.
  • Like the MBQL to native functionality, this is used by the query processor.

How are Metabase drivers written?

Let's take a high-level look at how Metabase drivers are written. Take a look at this sample code, and we'll discuss at everything in there in detail:

(ns com.mycompany.metabase.driver.foxpro98
  (:require [metabase.driver :as driver]))

(driver/register! :foxpro98)

(defmethod driver/display-name :foxpro98 [_]
  "Visual FoxPro '98")

Of course, because you took my advice at the beginning of this chapter and have already familiarized yourself with Clojure 😉, you already know exactly what's happening here. We define a namespace, call some function called register!, and define a multimethod implementation for the dispatch value :foxpro98. Nothing out of the ordinary. But why?

Driver namespaces

Typically, each Metabase driver lives in its own namespace -- com.mycompany.metabase.driver.foxpro98 in this case. All core Metabase drivers live in metabase.driver.* namespaces, but feel free to name yours whatever you want, unless your goal is to have it merged into the core Metabase repo, in which case you'll need to stick to metabase.driver.<driver-name>. It's probably best to use something resembling the Java package naming conventions.

Many drivers are further broken out into additional namespaces, especially larger drivers. Commonly, a driver will have a query-processor namespace (e.g. com.mycompany.metabase.driver.foxpro98.query-processor) that contains the logic for converting MBQL queries into native ones. This is often the most complicated part of a driver, and separating that logic from everything else can help make things easier to work with. Some drivers also have a separate sync namespace that has implementations for methods used by the sync process.

Now that we've covered namespaces, what goes in them?

Driver registration

The very first call after the namespace declaration in the example above was a call to metabase.driver/register-driver!:

(driver/register! :foxpro98)

All Metabase drivers must be registered before use. You will probably not need to do this manually using the call to register! because this is done automatically for drivers shipped as separate plugins; this will be discussed more in later chapters.

metabase.driver Multimethods

The metabase.driver namespace defines a series of multimethods, and drivers provide implementations for them, as in our example:

(defmethod driver/display-name :foxpro98 [_]
  "Visual FoxPro '98")

The four main features of a Metabase driver described above are all implemented by multimethod implementations. All of these methods dispatch off of the keyword name of a driver -- :foxpro98 in our case. In fact, that's all a Metabase driver is -- a keyword! There are no classes or objects to be seen -- just a single keyword.

You can browser the metabase.driver namespace for a complete list of multimethods you might consider implementing. You do not need to implement all of them -- some are optional -- but be sure to read the docstring for each method and decide whether you need to implement it. Ones that are optional are clearly described as such.

Getting a list of all driver multimethods

To quickly look up a list of all driver multimethods, you can run the command

lein run driver-methods

which will print a list of all driver namespaces and multimethods. This includes many things like sql and sql-jdbc multimethods, as well as test extension multimethods. Test extensions will be covered in a later chapter.

Parent drivers

One last thing to note: a lot of drivers share a lot in common, and writing complete implementations for sync methods and the like would involve a lot of code duplication. Thus many high-level features are partially or fully implemented in shared "parent" drivers, such as the most common parent, :sql-jdbc. A "parent" driver is analogous to a superclass in OO programming.

You can define a driver parent when you register it:

(driver/register! :postgres, :parent :sql-jdbc)

Note that for drivers packaged as separate plugins, you do not need to do this; instead, list the parent in the plugin manifest (discussed in upcoming chapters).

Parents like :sql-jdbc are intended as a common abstract "base class" for drivers that can share much of their implementation; in the case of :sql-jdbc, it's intended for SQL-based drivers that use a JDBC driver under the hood.:sql-jdbc and other parents provide implementations for many of the methods needed to power the four main features of a Metabase driver. In fact, :sql-jdbc provides implementations of things like driver/execute-query, so a driver using it as a parent does not need to provide one itself. However, various parent drivers define their own multimethods to implement.

Parent drivers to be aware of

Here's a few parent drivers you should be aware of:

• :sql-jdbc can be used as the parent for SQL-based databases with a JDBC driver.
  • :sql-jdbc implements most of the four main features, but instead you must implement sql-jdbc multimethods found in metabase.driver.sql-jdbc.* namespaces, as well as some methods in metabase.driver.sql.* namespaces
• :sql is itself the parent of :sql-jdbc; it can be used for SQL-based databases that do not have a JDBC driver, such as BigQuery.
  • :sql implements a significant chunk of driver functionality, but you must implement some methods found in metabase.driver.sql.* namespaces to use it
• Drivers that use Google's API, such as BigQuery and Google Analytics, can use the :google driver as a parent
• Some drivers use other "concrete" drivers as their parent -- for example, :redshift uses :postgres as a parent, only supplying method implementations to override postgres ones where needed

Calling parent driver implementations

You can get a parent driver's implementation for a method by using get-method:

(defmethod driver/mbql->native :bigquery [driver query]
  ((get-method driver/mbql-native :sql) driver query))

This is the equivalent of calling super.someMethod() is OO-programming. Note that it is important to pass the driver argument to the parent implementation as-is, so any methods called by that method used the correct implementation. Here's two ways of calling parents that you should avoid:

(defmethod driver/mbql->native :bigquery [_ query]
  ;; BAD! If :sql's implementation of mbql->native calls any other methods, it won't use the :bigquery implementation
  ((get-method driver/mbql->native :sql) :sql query))

(defmethod driver/mbql->native :bigquery [_ query]
  ;; BAD! If someone else creates a driver using :bigquery as a parent, any methods called by :sql's implementation
  ;; of mbql->native will use :bigquery method implementations instead of custom ones for that driver 
  ((get-method driver/mbql->native :sql) :bigquery query))

Multiple parents

The astute may have noticed that BigQuery is mentioned as having both :sql and :google as a parent -- the equivalent of OO "multiple inheritance". This is allowed and helpful! You can define a driver with multiple parents as follows:

(driver/register! :bigquery, :parent #{:sql :google})

For drivers shipped as a plugin, this is done in the plugin manifest, which will be described in a later chapter.

In some cases, both parents may provide an implementation for a method; to fix this ambiguity, simply provide an implementation for your driver and pass them to the preferred parent driver's implementation as described above.

Chapter Summary

• Familiarize yourself with Clojure before proceeding
• Metabase drivers do four things:
  • provide basic information like capabilities of the DB, connection properties needed to connect to it, etc.
  • provide information about the schema of the DB
  • transpile MBQL queries to native queries
  • run native queries and return results
• Metabase drivers typically live in their own namespace; some also have additional query-processor or sync namespaces
• Drivers must be registered before use; this is done automatically for drivers packaged as separate plugins
• Drivers are nothing more than a series of multimethod implementations
• Drivers can have one or more parent drivers to use shared functionality
  • The most common parents are :sql and :sql-jdbc
  • Most parent drivers define their own set of multimethods you must implement
  • Use get-method to call a parent driver implementation; be sure to pass the driver argument as-is

Next Up

Chapter 2: Packaging a Driver & Metabase Plugin Basics