Enable SQL Backend

[TheSecMaven]

Description

This resolves #688 and enables users to perform a normal upgrade of smallstep, and the new schema will be migrated to. At which point users can begin to query for certificates the application has signed on a go forward basis.

Note, some tests fail at present because GetCertificate changed to use Context in some places. Figured we could discuss before changing that, or handle however your team prefers.

[CLAassistant]

All committers have signed the CLA.

[TheSecMaven]

You will see that some methods haven't changed to a SQL backend fully, and implement the old NoSQL methods under the sql.DB. I think there is an opportunity to slow roll and migrate to a SQL schema as time passes. This at least enables SQL for storing certificates, which is a great start and can be built on top of.

[dopey]

Hey @mkkeffeler the way that we're beginning this implementation isn't quite what we're looking for.

What we want (and would consider merging) is a SQL implementation of the following three interfaces:

• Auth DB:

  certificates/db/db.go

  Lines 44 to 57 in 8df9f62

  	// AuthDB is an interface over an Authority DB client that implements a nosql.DB interface.
  	type AuthDB interface {
  	IsRevoked(sn string) (bool, error)
  	IsSSHRevoked(sn string) (bool, error)
  	Revoke(rci *RevokedCertificateInfo) error
  	RevokeSSH(rci *RevokedCertificateInfo) error
  	GetCertificate(serialNumber string) (*x509.Certificate, error)
  	StoreCertificate(crt *x509.Certificate) error
  	UseToken(id, tok string) (bool, error)
  	IsSSHHost(name string) (bool, error)
  	StoreSSHCertificate(crt *ssh.Certificate) error
  	GetSSHHostPrincipals() ([]string, error)
  	Shutdown() error
  	}

• ACME DB:

  certificates/acme/db.go

  Lines 15 to 40 in 8df9f62

  	// DB is the DB interface expected by the step-ca ACME API.
  	type DB interface {
  	CreateAccount(ctx context.Context, acc *Account) error
  	GetAccount(ctx context.Context, id string) (*Account, error)
  	GetAccountByKeyID(ctx context.Context, kid string) (*Account, error)
  	UpdateAccount(ctx context.Context, acc *Account) error
  	CreateNonce(ctx context.Context) (Nonce, error)
  	DeleteNonce(ctx context.Context, nonce Nonce) error
  	CreateAuthorization(ctx context.Context, az *Authorization) error
  	GetAuthorization(ctx context.Context, id string) (*Authorization, error)
  	UpdateAuthorization(ctx context.Context, az *Authorization) error
  	CreateCertificate(ctx context.Context, cert *Certificate) error
  	GetCertificate(ctx context.Context, id string) (*Certificate, error)
  	CreateChallenge(ctx context.Context, ch *Challenge) error
  	GetChallenge(ctx context.Context, id, authzID string) (*Challenge, error)
  	UpdateChallenge(ctx context.Context, ch *Challenge) error
  	CreateOrder(ctx context.Context, o *Order) error
  	GetOrder(ctx context.Context, id string) (*Order, error)
  	GetOrdersByAccountID(ctx context.Context, accountID string) ([]string, error)
  	UpdateOrder(ctx context.Context, o *Order) error
  	}

• Admin DB:

  certificates/authority/admin/db.go

  Lines 57 to 70 in 2d5bfd3

  	// DB is the DB interface expected by the step-ca Admin API.
  	type DB interface {
  	CreateProvisioner(ctx context.Context, prov *linkedca.Provisioner) error
  	GetProvisioner(ctx context.Context, id string) (*linkedca.Provisioner, error)
  	GetProvisioners(ctx context.Context) ([]*linkedca.Provisioner, error)
  	UpdateProvisioner(ctx context.Context, prov *linkedca.Provisioner) error
  	DeleteProvisioner(ctx context.Context, id string) error
  	CreateAdmin(ctx context.Context, admin *linkedca.Admin) error
  	GetAdmin(ctx context.Context, id string) (*linkedca.Admin, error)
  	GetAdmins(ctx context.Context) ([]*linkedca.Admin, error)
  	UpdateAdmin(ctx context.Context, admin *linkedca.Admin) error
  	DeleteAdmin(ctx context.Context, id string) error
  	}

Here's an example of one method on the Auth DB interface implemented with SQL:

func (d *DB) GetCertificate(serialNumber string) (*x509.Certificate, error) {
	var raw []byte

	if err := d.db.QueryRow("SELECT certificate FROM certificates WHERE serial = $1", serialNumber).Scan(&raw); err != nil {
		return nil, errors.Wrap(err, "error retrieving certificate")
	}

	cert, err := x509.ParseCertificate(raw)
	if err != nil {
		return nil, errors.Wrap(err, "error parsing certificate")
	}

	return cert, nil
}

There's ~40 methods across all 3 interfaces that will need to be implemented (with unit tests). Let me know if that does / doesn't make sense.

Note: this is a big code change and it's possible we won't have time to prioritize and thoroughly review it until next year.

[maraino]

There's more to what @dopey said. That works for a custom implementation of the start of the ca (ca/ca.go), the rest of the code stays the same, but to propper implement the functionality, there are some architectural changes that we need to make. For example to be able to register and load these interfaces in the right place.