Sigstore APIs/data formats: proto-first or JSON-first?

[znewman01]

Sigstore has a few traditional APIs:

• fulcio.proto
• rekor/openapi.yaml
• ctfe.sigstore.dev exposes an RFC 9162 API for certificate transparency

It's also got a few non-traditional "APIs," especially data formats:

• how cosign stores data in OCI for "signatures" (includes certificates and a "bundle" as well), SBOMs, attestations, vulnerability-scan attestations, generic predicates for in-toto attestations
• a new, cross-client bundle is proposed in cosign#2204 and this doc

There's not a lot of consistency across all of these, and there are some awkward bits:

$ COSIGN_EXPERIMENTAL=1 cosign sign-blob /dev/null --bundle bundle
[...]
$ cat /tmp/bundle | jq -r '.cert' | base64 -d
-----BEGIN CERTIFICATE-----
MIICnTCCAiOgAwIBAgIUK3wdhqhQg0vRtpHvYmSoCv/4/bYwCgYIKoZIzj0EAwMw
[...]

Yes, that's base64'd PEM (which is, yes, already base64'd).

While there are relatively few consumers of this API, we have the opportunity to make some improvements. I believe that the new bundle format is looking good (thanks, @ kommendorkapten and reviewers) except we're still a little confused about how to encode things.

A key point of disagreement is: do we try to make the proto representation idiomatic, or the JSON representation idiomatic? I posit that we can't have both in the same format, and that trying to have both (following the spirit of this comment) is an underlying cause of a lot of inconsistencies and awkwardness.

Background on encoding

This section is a whirlwind tour of encodings for cryptographic structures, data formats, and APIs. It's not going to be able to do a good job, so click through if you really want to learn more. But hopefully it will help set the stage.

Data definitions all start with an interface definition language (IDL). This is a formal way to write down the format of data. In Sigstore, we care about three:

1. Protocol buffers: used to define the Fulcio API, and the in-progress new bundle format.
2. Abstract Syntax Notation 1 (ASN.1): used for a lot of cryptographic and web structures (see this introduction for more details).
3. OpenAPI: used for the Rekor API.

You can also skip the IDL, and just write a server that handles different routes as CTFE does; I recommend against this, because the IDL is a nice, declarative way to document and define an interface all at once.

At the end of this process, you've described a representation of an abstract thing. For instance, a "birthday" is an abstract concept; an (int, int) tuple representing month and day is the representation. Note that at this point we still don't know how to write our abstract concept as bits and bytes! For that, we need an encoding. The relevant ones here are:

• Distinguished Encoding Rules (DER): a common way to turn all of the ASN.1 data types (bytes, integers, sequences/sets, dates, times, objects, null, choice) into binary data (see this introduction for more details).
• JavaScript Object Notation (JSON): a human-readable format supporting strings, numbers, arrays, objects, bools, and null.
• hex: turns binary data into text by representing each byte as one of the following characters: 0123456789ABCDEF.
• base64: turns binary data into text by representing every 3 bytes as a sequence of 4 characters (64 different ASCII characters).
• Privacy-Enhanced Mail (PEM): bytes are hard to send over email, and email servers used to (still do?) wrap lines, so PEM takes binary data (typically DER-encoded), base64 encodes it, wraps the lines, and adds a header/footer.

You may compose these encodings. For instance, you may take DER-encoded value and PEM-encode it. Some of these encodings, like JSON, don't support binary data, so such data must be encoded in a text-friendly manner.

Sigstore is moving towards proto for its interfaces (source: I looked at two data points and drew a line). Protos have their own encoding, but also support serialization to JSON, and this is often how they're used for remote procedure calls (RPCs) over the web.

Example encodings

Here is a table of possible encodings of most types that appear in either the bundle, Fulcio API, or Rekor API, including representations I was able to find and some reasonable alternatives; it also has my overly-opinionated take on how idiomatic the encodings to proto and JSON are.

Thing to Encode	Representation	Encoding	Proto type	Idiomatic?	JSON value	Idiomatic?	Seen in
Data	Opaque	None	bytes	✅	string, base64	❌	Rekor (Rekord)
		JSON+base64	bytes	❌	string, base64+base64	🤮	Cosign signature bundle
Certificate	X.509	PEM	string	✅(a)	string, PEM	✅(a)	Fulcio, Cosign signature
			bytes	❌	string, base64+PEM	🤮	Rekor
		DER	bytes	✅	string, base64+DER(b)	❌	CTFE
Certificate chain (list of certs)	List of X.509	list of...	repeated	see above	list of...	see above	Fulcio
		concatenated PEM	string	🤷	string, concatenated PEM	🤷	Cosign signature
Certificate signing request (CSR)	PKCS#10	DER	bytes	✅	string, base64+DER(b)	❌
		PEM	string	✅(a)	string, PEM	✅(a)	Fulcio
Public key	Opaque(c)	PEM	string(d)	🤷	string, PEM	✅(a)	Fulcio, Rekor
	PKIX	PEM	string	✅(a)	string, PEM	✅(a)
		DER	bytes	✅	string, base64+DER(b)		Fulcio
			string(d)	❓	❓	❓	Fulcio
Algorithm type	Enum	enum	enum	✅	string, enum name (SHA2_256)	✅(e)	Fulcio, Rekor (HashedRekord)
Hash digest	Opaque(c)	none	bytes	✅	string, base64	🤷	Cosign signature, CTFE
		hex	string	❌	string, hex	🤷	Rekor
		base64	string	❌	string, base64	🤷
	Prefixed string (sha256:...)	base64	string	❌	string	🤷	OCI
Merkle proof (list of hash digest)	List of binary	list of...	repeated	see above	list of...	see above	Rekor
Signature	Opaque(c)	none	bytes	✅	string, base64	🤷	Cosign signature, Fulcio, Rekor (HashedRekord), CTFE
		hex	string	❌	string, hex	🤷
		base64	string	❌	string, base64	🤷
		JSON+base64	byte	❌	string, base64+base64	🤮	Cosign signature bundle
	PGP	PEM	byte	❌	string, base64+PEM	🤮	Rekor (Rekord)
	SSH	PEM	byte	❌	string, base64+PEM	🤮	Rekor (Rekord)
	Minisign	base64	byte	❌	string, base64+base64	🤮	Rekor (Rekord)
Signed Tree Head (STH)	STH	DER	bytes	✅	string	✅
	Log checkpoint	none	string	✅	string	✅	Rekor
CT log ID	Opaque	none	bytes	✅	string, base64	🤷
	SHA256(DER(pubkey))	none	bytes	✅	string, base64	🤷	Rekor
		JSON+hex	string	❌	string, hex+base64	🤮	Cosign signature bundle
RFC 3161 Timestamp	TSTInfo	DER	bytes	✅	string, base64+DERe	❌
Signed Certificate Timestamp	SCT	DER	bytes	✅	string, base64+DERe	❌	CTFE
	AddChainResponse	JSON	bytes	❌	string, base64+JSON	🤮	Fulcio
Log Entry	LogEntry	object	message	✅	object	✅	Rekor
Log Entry (body)	Varies by Rekor type	object	message	✅	object	✅	Rekor
Signed Entry Timestamp (signature)	Signature over the above	None	bytes	✅	string, base64	🤷	Rekor
	N/A (signed timestamp + STH)	N/A	N/A	N/A	N/A	N/A	Proposed for Rekor
UUID	Binary	none	bytes	✅	string, base64	❌
		hex	string	❌	string, hex	❌	Rekor
		base64	string	❌	string, base64	❌
	UUID String	string	string	✅	string	✅
Shard identifier	[TreeID] (uint64)	hex	string	❌	string, hex	❌	Rekor
		int	string	❌	string, int	❌	Rekor
			bytes	✅	string, base64	❌
			int	✅	int	✅

^(a) @haydentherapper likes PEM, but IMO the point of PEM is that it's for when you don't have an easy way to encode bytes or communicate the data format; in the proto setting, we have both. PEM parsing can be the source of bugs, and in some languages, PEM requires an additional dependency over common cryptographic libraries.

^(b) PEM is just DER + base64 with line wrapping and a header/footer line, so this is almost PEM.

^(c) Algorithm-dependent, opaque binary blob.

^(d) You can't really encode DER directly into a string; I haven't figured out how to actually trigger this code path from the API.

^(e) Stringly-typed, so you could consider that idiomatic if you like that sort of thing.

Some arguments for each

Arguments for a proto-first worldview:

• Security is a top priority, and easy-to-use (and hard-to-misuse) interfaces help with security. We should be very opinionated, and protos make that easier.
• As you can see from the table above, JSON's limited type system means that very little feels natural. JSON-based libraries will often wind up in the situation of "I have a base64 blob, but I don't know what it is because I only know the field name," which is how you wind up with parsing issues.

For JSON-first worldview:

• Some client libraries might not use protobufs (sigstore-python doesn't, as of the time of this writing). We may want to encourage this, as it allows for dependency-light implementations.
• JSON is what end users will interact with; we want them to have an easy experience (counterpoint: we might not want end users interacting with these things at all, for fear that they'll think they know how to verify something when actually they don't).
• JSON is easy to feed into other tools like jq.

For a proto-first worldview, but use PEM for certs/keys:

• Sometimes you might not know how to interpret plain bytes; PEM gives you a hint in the header (though: we can always stick the format in an enum next to the data).

I'm sure I'm missing a lot; that's why this is a discussion! Please weigh in 😄

Open Questions

• How much do we care about...
  • client library authors using proto?
  • client library authors parsing JSON manually?
  • end users who want to inspect API responses and data formats?
  • (options include: top priority, try to help them out where we can, indifferent, actively discourage)
• From the perspective of library code: we get many of these things pre-encoded from Rekor/Fulcio. Should we decode them, or just pass through? The former is more work, but typically results in a more idiomatic result. Also, many of the encodings chosen by Rekor/Fulcio are...suboptimal. But the latter is easier as a client.
• Generally should we wrap things (transparent in JSON)? Group things and use oneofs (annoying in JSON)?
• Do we care about exposing an HTTP-first interface? If so, could we do this by writing either a gRPC<->idiomatic HTTP proxy? Or a shim API handler?
• Based on my experience curating this table, the documentation for these APIs is pretty rough. Regardless of the outcome here, I hope to take a pass and clarify the docs.
• Is a leaf certificate part of the certificate chain? In the Cosign signature format, you provide both and they're disjoint. Fulcio gives you one, shared chain.

[kommendorkapten]

Very well written post, with tons of good information. Thanks!

Disclaimer: I'm in favour of using Protobuf, and always rely on protojson during de-/serialization.
Pros:

• With protobuf codegeneration you get native data structures "for free", with the idiomatic data type.
• Always using protojson would IMHO turn all ❌ to ✅ in the 7:th column (Idiomatic with JSON) as after JSON deserialization the developer would not need to perform any extra base64 deserialization (compare this to relying on the "regular" JSON deserializer).
• When using protojson some pitfalls are avoided, it's common to encode integers into JSON numbers, which will loose precision for integers larger than 2^53 - 1, protojson solves this and always encodes int64 as string.
• Message reuse (which forces consistency). Some data types (like certificate chains) are used in more than one place. With protobuf it's easy to reuse messages, which will enforce consistent naming and data types across all formats.

Even if the formal definition is done in protobuf, the canonical serialization can still be protojson generated JSON, and so JSON would be what the majority of the developers would interact with. We should also have all services expose both gPRC and HTTP APIs, e.g. Grafeas does this by using grpc-gateway: https://github.com/grafeas/grafeas/blob/master/go/v1beta1/server/server.go#L75

How much do we care about... client library authors using proto?

IMHO we should encourage all libraries to rely on the generated code via protobuf

How much do we care about... client library authors parsing JSON manually?

IMHO we should discourage all libraries to manually parse JSON and always rely on protojson.

How much do we care about... end users who want to inspect API responses and data formats?

I think we should. And with by offering HTTP API with JSON (via the grpc-gateway) this should be fairly easy to achieve.

From the perspective of library code: we get many of these things pre-encoded from Rekor/Fulcio. Should we decode them, or just pass through? The former is more work, but typically results in a more idiomatic result.

Initially I'm in favour of the former (decode into to the protobuf generated format). With the future in mind, when all Sigstore services offers APIs defined by protobufs, this will not be an issue as they would natively be in the expected protojson format (or via gRPC direct).

Group things and use oneofs (annoying in JSON)?

I'm in favour of this. It simplifies mental model as we can add context without having long and hard to read attribute names.

Do we care about exposing an HTTP-first interface? If so, could we do this by writing either a gRPC<->idiomatic HTTP proxy? Or a shim API handler?

IMHO yes, see above on using grpc-gateway.

Based on my experience curating this table, the documentation for these APIs is pretty rough. Regardless of the outcome here, I hope to take a pass and clarify the docs.

I can help with this if wanted, as by defining the Sigstore bundle I think some documentation/comments have already been improved.

Is a leaf certificate part of the certificate chain? In the Cosign signature format, you provide both and they're disjoint. Fulcio gives you one, shared chain.

Having all certificates in the chain as done by Fulcio seems easier. But I don't have a strong opinion here.

Protobuf has quite widespread support of languages, the official protbuf compiler supports:

• C++
• Dart
• Go
• Java
• Kotlin
• Python
• Ruby
• C#
• Objective-C
• PHP

Other implementations exists for:

• JavaScript https://github.com/protobufjs/protobuf.js
• Rust https://github.com/stepancheg/rust-protobuf https://github.com/tokio-rs/prost

[bobcallaway]

Great writeup - I am in favor of protobuf, and continuing the pattern started in fulcio of using the grpc-gateway to expose both GRPC and HTTP/JSON interfaces.

[patflynn]

+1000. Thanks for this thorough and clarifying write-up @znewman01 . In the community meeting I voted for JS because from my point of view, the current user facing API is the bundle, and IMO as long as it's stored as JS we should optimize that representation to be easy to parse and use... because folks inevitably will treat it as the API even if we'd rather they use it through our tools. I like @kommendorkapten arguments towards using protojson, as it sounds like it might allow us to have our cake and eat it too.

Offline we chatted about potentially making the bundle opaque to discourage user slicing and dicing and minimize footguns. In that model cosign and the SDKs are the only API, at which point I think we could go all-in on proto and considerably simplify client code.

[znewman01]

protojson, as it sounds like it might allow us to have our cake and eat it too.

IMO it doesn't. Proto-aware clients are equally idiomatic in this case, but proto-unaware clients (e.g., jq) still look exactly like the later columns above (that's the encoding I was describing).

If we have a proto API, we should definitely use protojson for any JSON-interop layer. But it doesn't solve the problems with JSON-unaware clients.

[patflynn]

@znewman01 just to make sure I understand, do you think we should forego JSON interop and store the bundle is an opaque proto binary?

[znewman01]

There are two separate questions, I think:

1. API: are we going to expose a JSON gRPC API? The consensus seems to be yes, and if we do, we'll use the standard proto serialization (protojson). Then, any proto-aware clients get the nice experience even though we're doing JSON. This is a little bit at the expense of any proto-unware JSON clients, but we seem to have made our peace with that.
2. Bundle format: when I store something on disk, how do I encode it? Options include (a) JSON, idiomatic, (b) JSON, protojson, (c) some other proto encoding that will come out opaque.

Question 2 seems unresolved. I think we probably don't want to go with (a), because that seems to encourage. I have a slight preference for (c), because it actively discourages folks from poking around, which means they're going through tools which should remove most footguns. But I think (b) is a reasonable compromise, and does allow "Sigstore verification the hard way" types of exercises (cf. this gist).

[patflynn]

I guess they could use protoc for their hard way journeys :).

What's not clear to me with the bundle option (c) is how a json API consuming client then turns around and produces proto. If the client ultimately needs to generate a proto, why wouldn't it just use protos e2e rather than use json at the api?

If (c) feels like a big change/stretch, and folks are all comfortable with (b), I'd be happy with taking that route if it gets us to consensus quickly. Frankly I don't have a strong sense of which way to go apart from my question above.

[znewman01]

Yeah, that's a good point. I'm happy enough with (b) and that seems less controversial/easier to implement so maybe that's my vote for now.

[patflynn]

@kommendorkapten @bobcallaway anybody want to throw in 2c? You all good with (b)?

[kommendorkapten]

I'm 100% behind @znewman01 's last comment, and my understanding is too that this is the consensus.

With that said, I recall certificates being discussed as a special case, and PEM encoding was preferred. What's the current take here? I'm ok with either PEM encoding (string) or the DER data (bytes). In terms of size when encoded into JSON, they are almost the same, as PEM as we know is just base64(DER) + headers and newlines.

As we will only transfer certificates and not public keys, the PEM header will not give us any added value, as it will only specify that it's a certificate, which we already know, as it's in the Certificate field.

@haydentherapper your thoughts on this (PEM vs DER bytes) on certificates?

I'll go on and update the PR for the bundle, and as part of that I'll rip out the messages which can be shared into a separate file to make it clear how we can start to organize things.

[kommendorkapten]

If we ever see ourselves needing an opaque format, we could define a format around the wire format, with appropriate headers. But until we have a real use-case for it, I rather not.

[znewman01]

I'm happy to move the cert-specific discussion over to the bundle PR.

Here's my 2c on certs:

• Prefer encoding certs as DER in a bytes field: IMO the point of PEM is for when you don't know what type an object is, and that's not true here. Also, handling PEM just means converting to DER and then handling DER, so feels like we're introducing another layer for no reason. If we've decided that JSON API clients aren't a priority then I think the confusion issue is gone.
  • I could live with string fields containing PEM
• Prefer encoding cert chains as repeated bytes with DER inside
  • Strong preference against cert chains as string (newline-separated PEM)
  • I could live with repeated string containing PEM

If we ever see ourselves needing an opaque format, we could define a format around the wire format, with appropriate headers. But until we have a real use-case for it, I rather not.

I think that's a fair judgment and I'm happy to move forward with that. However, the use case IMO is preventative: to stop people from trying to poke around inside. So once we start storing things on disk in JSON, the cat's out of the bag somewhat 🤷

(Think of it somewhat like QUIC's approach to preventing ossification 🙂)

[haydentherapper]

I don't feel incredibly strongly anymore, because either will work. Fulcio returns a PEM certificate chain as repeated string, and I would probably prefer to do the same in clients, but I also don't mind if we say that clients should unwrap the certificates into repeated DER bytes.

[kommendorkapten]

During manual grok-ing with the JSON files, having certificates as base64(DER) is easy to work with, openssl(1) can as expected take raw DER bytes as both input and produce them.

Even from a developer perspective, this removes a tiny bit of extra work, as the PEM decoding is no longer needed.

I'll go on an update the messages use bytes.