core/crypto design/planning/thoughts

[Yawning]

As core:crypto gets fleshed out, I thought it would be a good idea to come up with a cohesive design/implementation plan.

As a warning, I am somewhat opinionated, and a lot of these things are subjective.

Design goals

Have core/crypto be a library that:

• Has a unified/consistent API (within reason). Hash functions should expose the same API. ECDH should expose the same API, etc.

• Nudges users towards making good decisions regarding primitive selection, and ideally guides them towards using higher level constructs rather than lower level ones (eg: Prefer noise/TLS over chacha20poly1305 over chacha20 + poly1305 by hand over ROT13). Primitives that exist for backward compatibility purposes (ie: are not a good choice for new systems) should be annotated as such, or split off into a separate package.

• Makes it hard to misuse APIs. For example, if the library provides PBKDF2 the output could be an opaque type, so that the user is inclined to use compare_hashed_password(expected, digest: ^HashedPassword) -> bool over mem.compare. Using strong-typing does wonders here, though there is a balance/trade-off between "safe" and "annoying".

• Provides primitive implementations that are resistant to at least timing side-channel attacks. As a example, Golang's crypto/aes leaks the key on systems where the CPU or the standard library lacks support for hardware AES. Instead AES should be implemented correctly (bitsliced portable implementation, hardware accelerated per-target fast implementations) or call native code that does it correctly. Not "merge a Te table based implementation anyway". WASM is explicitly out of scope for this requirement due to this being impossible as currently specified.

• Makes an effort to sanitize sensitive memory, including intermediaries, to mitigate the impact that bugs that disclose memory can have (mem.zero_explicit). How to handle registers and values that get spilled onto the stack is an open question, that I personally would feel is ok to defer for now.

• No "Hipster Crypto". Exotic things like VRFs, ZK proof systems, pairing friendly curves, etc belongs in external libraries.

Primitives and other functionality

This is a (non-comprehensive) list of low-level primitives that are "a reasonable choice for something new" and or "required to interoperate with something widely deployed", that the library can strive towards. Some of these primitives are implemented already.

• Hash functions

  • SHA-1 (backward compatibility only)
  • MD-5 (backward compatibility only)
  • SHA-2 family (SHA256, SHA384, SHA512, SHA512/256)
  • SHA-3 family
  • TupleHash (SP 800-185)
  • BLAKE2 family (BLAKE2s/BLAKE2b)

• eXtendable Output Functions

  • SHAKE
  • cSHAKE (SP 800-185)

• MAC functions

  • Poly1305
  • HMAC
  • KMAC (SP 800-185)
  • GHASH (hard to implement securely)

• Digital signatures (hard to implement securely)

  • Ed25519 (some subtle implementation gotchas)
  • ECDSA (secp256r1, TLS1.3)
  • RSA (notoriously hard to implement securely, TLS1.3)

• Key agreement

  • Elliptic Curve Diffie-Hellman (hard to implement securely)
    • X25519
    • X448
    • secp256r1 (TLS1.3)

• Stream ciphers

  • ChaCha20/XChaCha20
  • CTR mode

• Block ciphers (and non-authenticated modes)

  • AES (notoriously hard to implement securely, TLS1.3)
  • ECB mode

• Authenticated Encryption with Associated Data

  • chacha20poly1305
  • GCM mode (TLS1.3)

• Key Derivation Functions and password hashing

  • HKDF
  • PBKDF2
  • bcrypt
  • Argon2
  • scrypt

• Random number generation

  • Linux
  • Windows
  • Darwin
  • BSDs
  • NIST SP 900-90A Revision 1 DRBGs

These are things that are a "nice to have":

• AEGIS (AEAD, based on AES so hard to implement securely)
• Deoxys-II (AEAD, based on AES so hard to implement securely)
• Ristretto255
• Ed448 (not widely used)
• secp384r1, secp521r1 (pointless tinfoil-hattery, but people do use them. Note secp521r1 isn't allowed by Let's Encrypt.)
• Kekkac (compatibility only)
• xchacha20poly1305 (Might as well, because it's easy)
• OCB mode
• CBC mode (Rather not, but there's likely significant legacy usage)
• Post-Quantum public key cryptography (Want eventually, but standard algorithm definitions not finalized)
  • FIPS 203, Kyber KEM (Wait till standard is finalized)
  • FIPS 204, Dilithium signature (Wait till standard is finalized)
  • FIPS 205, SPHINCS+ signature (Wait till standard is finalized)

Maybe:

• Salsa20/XSalsa20 (Use Chacha20/XChacha20, however scrypt needs Salsla8, so this is cheap to do)
• NaCl box, secretbox (Not sure these are used much anymore)
• BLAKE2x (Not sure if it is popular enough to justify)
• BLAKE3 (It seems moderately popular, but I'm not a huge fan)
• SM3 (Commie Not-Invented-Here SHA2)

These primitives are "rather not" that might need to be implemented for interoperability:

• Non-Elliptic Curve Diffie-Hellman
• secp256k1 (Shitcoin curve)
• CCM mode

The primitives were chosen around what is required to implement/interoperate with:

• Noise Protocol Framework
• TLS 1.3 (https://www.rfc-editor.org/rfc/rfc8446#section-9.1)
• Web PKI standards (RSA, secp256r1, secp384r1, secp521r1)

[zhibog]

An idea just now was to rip out all the context switching stuff from core and leave only the Odin implementation. Then move any binding stuff to core:vendor, exposing the original API of the library as well as the API of the Odin crypto lib. That way core would be cleaned up and simplified + you could easily switch between using Odin or a lib by just changing the import.

[Yawning]

For the core/crypto hash incremental API(s), something like the following could be a starting point:

DIGEST_SIZE :: 32 // Should use consistent naming for constants.

Context :: struct {
  // Opaque members

  _is_initialized: bool,
}

clone :: proc (dst, src: ^Context) {
  assert(src._is_initialized)

  // Deep-copy src->dst (needed to support rolling digests etc).
}

reset :: proc (ctx: ^Context) {
  // Obliterates anything potentially sensitive in ctx with mem.zero_explicit,
  // including but not limited to the hash function state, and any buffered
  // and unprocessed data.

  ctx._is_initialized = false
}

init :: proc (ctx: ^Context, /* additional options go here */) {
  // Initialization.

  ctx._is_initialized = true
}

update :: proc (ctx: ^Context, data: []byte) {
  assert(ctx._is_initialized)

  // Incremental update.
}

final :: proc (ctx: ^Context, dst: []byte) {
  assert(ctx._is_initialized)
  if len(dst) != DIGEST_LEN {
    panic("crypto/myfancyhash: invalid destination digest size")
  }

  // Two "reasonable" choices here, I have no strong opinions as to which is used for hashes,
  // as long as it is consistent.

  // Go stdlib style:
  {
    f_ctx: Context = ---
    clone(&f_ctx, ctx)
    defer reset(&f_ctx)
    // ... Derive the digest with f_ctx (leaving ctx untouched).
  }

  // OpenSSL style:
  {
    defer reset(ctx)
    // .. Derive the digest with ctx, further non-init/reset calls using ctx should at least assert.
  }
}

For the one-shot calls, I have no strong opinions, as long as they are composed on top of the incremental API, and the context used is properly sanitized.

[zhibog]

what's the reasoning for the cloning within final?
do you want to keep the original ctx untouched? if yes - for what reason?

The init, update and final basically already look like that for the most part.
The bool field for the Context struct is a nice addition, clone and reset are probably too.

[Yawning]

I believe it is to make calculating rolling digests easy by default (incrementally hash some data, derive a digest, hash some more), but personally I lean a bit more towards having the caller handle calling clone if they want to do that (so OpenSSL EVP style behavior).

MAC algorithms for the most part can expose a similar external API, though I would probably omit clone (some but not all MAC algorithms fail catastrophically if the key is reused).

[zhibog]

Yes. Agreed, the caller can handle that in their loop and get the data out when they need it if it's needed.

[Yawning]

As a minor nitpick of the existing APIs, should the context struct name be prefixed with the name of the algorithm by default? (eg: Blake2b_Context).

For the primitives I am adding in my branch, I just named the struct Context because the algorithm name is implicit via the package selector (eg: ctx: poly1305.Context). With the current APIs, it would look like ctx: sha3.Sha3_Context which feels somewhat redundant.

[zhibog]

Yes. That's somewhat of a leftover from the context switching thing I didn't address when ripping it out. I'll change them to Context soon. Although in cases where there is more than one struct I still need to name them differently.
Another option would be to combine everything into a single struct but I kind of like the ability to use compile time when statements rather than normal if when checking which one it is

[Yawning]

If I could get some feedback regarding the MAC/AEAD/DH/stream/utility calls in my branch, I would appreciate it.

#1277

[Yawning]

Ok, now that the first round of my primitives are merged, this is my tentative second round plan:

• Go back and add some documentation.
• (X)Salsa20
• NaCl box/secretbox
• Ed25519pure, Ed25519ph, Ed25519ctx
• Add tests against Wycheproof

This will bring core/crypto up to feature parity with TweetNaCl, with the following caveat. There are quite a few different ways in which Ed25519 signature verification is implemented in the wild, stemming from implementations that predate standardization efforts, and the RFC leaving "cofactor multiply or not" up to the implementation. My plan is to follow modern "best practices" here, and as far as I know all of the various incompatibilities do not come into play with well-formed signatures (I can go into more detail here if required).

Do we want to slim down the number of hash functions present in core/crypto, or is it sufficient to annotate the "not really recommended" ones? I marginally prefer the former just from a maintenance burden standpoint, but the nice things about primitives like these is that it tends to be "get correct once" code.

[Kelimion]

Do we want to slim down the number of hash functions present in core/crypto, or is it sufficient to annotate the "not really recommended" ones? I marginally prefer the former just from a maintenance burden standpoint, but the nice things about primitives like these is that it tends to be "get correct once" code.

I would start by limiting them to current ones. The "not really recommended" hash functions can be added one by one with proper annotation.

[zhibog]

I've been thinking about how to best do HMAC + PBKDF2.
I wonder if I should just stick with one algo for each of them, like SHA256. Alternatively do all of SHA1, SHA2.

It could be a little annoying API wise if there are multiple available. It might make sense to split it into different packages.
Otherwise, if it's in a single package we could have different structs. That's how I've already implemented them.

Any thoughts?

[Yawning]

So, assuming you want to use one-shot calls for everything, something like this should cover both the common and uncommon cases.

Hash :: enum {
  Sha256,
  Sha512,
  Sha3,
  // Others redacted for brevity.
}

HashBytes :: proc (dst, src: []byte)

// Note: Minor prototype change required for the one-shot calls to allow for a unified API.
_hash_bytes := map[Hash]HashBytes {
  Sha256 = sha2.hash_bytes_256,
  Sha512 = sha2.hash_bytes_512,
  Sha3 = sha3.hash_bytes,
  // etc
}

_digest_size := map[Hash[int {
  Sha256 = 32, // TODO: Un-magic-number this.
  Sha512 = 64,
  Sha3 = 32,
}

sum_fn :: proc (h: Hash) -> HashBytes {
  return _hash_bytes[h]
}

digest_size :: proc (h: Hash) -> int {
  return _digest_size[h]
}

hash_bytes :: proc (h: Hash, dst, src: []byte) {
  fn := sum_fn(h)
  fn(dst, src)
}


// To deal with oddballs that have configurable digest-size and or take additional parameters
// like BLAKE2s, use parametric polymorphism for HMAC etc to support passing in either a
// Hash (enum) or a HashBytes (proc, caller can provide a wrapper that sets up the correct
// digest size if they are doing something exotic).  For example for HMAC...

Context :: struct {
  _sum_fn: hash.HashBytes.
 _digest_size: int,
  // irrelevant functions omitted for brevity.
}

init_hash :: proc (ctx: ^Context, h: hash.Hash, key: []byte) {
  init_hash_bytes(ctx, hash.sum_fn(h), hash.digest_size(h), key)
}

init_hash_ex :: proc(ctx: ^Context, fn: hash.HashBytes, digest_size: int, key: []byte) {
  ctx._sum_fn = fn
  ctx._digest_size = digetst_size
  // blah blah blah
}

// Past that `ctx._sum_fn(dst, src)` can be used for every single other call, and
// `ctx.digest_size` can be used to size buffers.

The one place where having a unified init/update/final/etc API would be extremely useful is implementing something like TLS, but I personally suspect that doing that is a while away (if only because X.509 and ASN.1 are nighmarish), so the one-shot only unified interface should be more than sufficient for now.

Edit: Oops, forgot that we don't have a garbage collector. Like I said, I'm tired.

[zhibog]

Remember the procs within the algos have a signature like hash_bytes :: proc(data: []byte) -> [SIZE]byte so you won't be able to generalize it as a map like that, using dst and src slices. But that's what your note is for I suppose.

[Yawning]

Yep. It does require changing the public API ever so slightly. I think passing in the destination makes certain use-cases less annoying (eg: including the digest into a structure being build incrementally, since you can pass hash_bytes a slice into an existing buffer), but that probably is a matter of taste.

The unfortunate thing is that using slices gives up a bunch of type-safety, but having to explicitly do lots of copies also has issues of it's own.

[zhibog]

I thought about it after I went to bed and I noticed that it doesn't have to be either / or.
I can just add this as a possible proc to the proc map without any issue.
Here is an example for MD4:

hash_bytes :: proc(data: []byte) -> [16]byte {
    hash: [16]byte
    ctx: Md4_Context
    init(&ctx)
    update(&ctx, data)
    final(&ctx, hash[:])
    return hash
}

hash_bytes_to_buffer :: proc(data, hash: []byte) {
 // NOTE: Maybe assert the block_size of the hash algo to the
 // length of the dst slice to make sure it is big enough to remedy possible issues?
    ctx: Md4_Context
    init(&ctx)
    update(&ctx, data)
    final(&ctx, dst[:])
}

hash :: proc {
    hash_bytes,
    hash_bytes_to_buffer ,
    // omitted the rest
}

This also gives the end user more choices, depending on their use case and it provides you with the API needed.
I think this is a good way to solve it.

So that + having a DIGEST_SIZE constant defined in each package should be enough to make it work.

[zhibog]

Made a PR with the extended API

#1401

[wardjm]

With Apple and likely others to follow, post-quantum stuff is going to start to become more relevant and used. Should we add that to the list?

[Yawning]

Eventually. Though I would say it is a low priority for now for the following reasons:

• Having all the classical primitives require to allow implementing spec compliant TLS1.3 client and server is likely more immediately useful.
• Standardization is still in progress (NIST FIPS 203, 204, and 205 are currently "Initial Public Draft"s), so the algorithms are still subject to change.
• The correct way to deploy PQ KEMs is currently an area of active research, though consensus is "in some sort of hybrid construct". I do have preferences here (https://eprint.iacr.org/2024/039 and the corresponding IETF draft), but that also is a moving target.
• The TLS extension also currently in IETF draft stage.

I'll add them to the list, but with a note to wait till standardization has finished. (Edit: Added)