[wip] Willow

[Frando]

Description

Work in progress! Implementation of https://willowprotocol.org/ for iroh

Protocol

The protocol module contains the data structures and basic interactions as defined by the Willow specs. It is IO-less and not generic, all types are concrete.

• Willow data model proto::willow
  • Struct defintions
  • Spec compliant encoding for entries (maybe, didn't test)
  • Authorizing entries with Meadowcap
• Meadowcap proto::meadowcap
  • Create and verify owned capabilities without delegations
  • Create and verify communal capabilities without delegations
  • Support delegations
• Groupings proto::groupings
  • Ranges, ThreeDRanges and Areas with inclusion checks
  • intersection of ranges
• Willow Geneal Sync Protocol (WGSP) proto::sync
  • Most message type structs written down
  • Private Area Intersection
  • Cryptographic fingerprint (uses only XOR atm)

Session

The session module contains the implementation of the reconciliation session.

• Session setup
• Private area intersection
• Set reconciliation
• Resource control
  • Binding resources
  • Freeing resources
  • Issue initial guarantees
  • Respect issued guarantees
  • Issue new guarantees
• Live data mode between two peers
  • Subscribe to new inserts and send to active sessions
  • Only send entries for matcing shared areas of interest
• Live data mode in a swarm setting
  • Basic swarm managmenet over hyparview
  • ...

Net

The net module opens sessions over iroh-net QUIC connections.

• Commitment exchange
• Separate channels for each logical channel
• Basic session lifecycle (allow to close sessions)

Store

The store module contains backing stores for entries and keys

• Store traits and generic functionality
  • Entry store trait
  • Key store trait
  • Use iroh-blobs for payload storage
  • Store capabilities by hash and do reference counting
  • Expose hook to prevent GC of payloads and/or do reference counting for payloads
• PoC memory store
• redb store
  • port @rklaehn's redb PoC
  • figure out if the traits need to change for our delayed commit strategy

Breaking Changes

Once we integrate, this will be a very much breaking changes for everything around docs: This will break not only APIs, but also the protocol and the storage. We cannot offer automatic migration even, because signatures change. How exactly we deal with this situation is tbd. Likely we will offer an out-of-tree tool to migrate data from the old iroh-docs to the new willow iroh-docs.

Notes & open questions

Change checklist

• Self-review.
• Documentation updates if relevant.
• Tests if relevant.
• All breaking changes documented.