Introduction to Event Sourcing - Self Paced Kit

Event Sourcing is perceived as a complex pattern. Some believe that it's like Nessie, everyone's heard about it, but rarely seen it. In fact, Event Sourcing is a pretty practical and straightforward concept. It helps build predictable applications closer to business. Nowadays, storage is cheap, and information is priceless. In Event Sourcing, no data is lost.

The workshop aims to build the knowledge of the general concept and its related patterns for the participants. The acquired knowledge will allow for the conscious design of architectural solutions and the analysis of associated risks.

The emphasis will be on a pragmatic understanding of architectures and applying it in practice using Marten and EventStoreDB.

1. Introduction to Event-Driven Architectures. Differences from the classical approach are foundations and terminology (event, event streams, command, query).
2. What is Event Sourcing, and how is it different from Event Streaming. Advantages and disadvantages.
3. Write model, data consistency guarantees on examples from Marten and EventStoreDB.
4. Various ways of handling business logic: Aggregates, Command Handlers, functional approach.
5. Projections, best practices and concerns for building read models from events on the examples from Marten and EventStoreDB.
6. Challenges in Event Sourcing and EDA: deliverability guarantees, sequence of event handling, idempotency, etc.
7. Saga, Choreography, Process Manager, distributed processes in practice.
8. Event Sourcing in the context of application architecture, integration with other approaches (CQRS, microservices, messaging, etc.).
9. Good and bad practices in event modelling.
10. Event Sourcing on production, evolution, events' schema versioning, etc.

You can do the workshop as a self-paced kit. That should give you a good foundation for starting your journey with Event Sourcing and learning tools like Marten and EventStoreDB. If you'd like to get full coverage with all nuances of the private workshop, feel free to contact me via email.

Exercises

1. Events definition.
2. Getting State from events.
3. Appending Events:
   • Marten
   • EventStoreDB
4. Getting State from events
   • Marten
   • EventStoreDB
5. Business logic:
   • General
   • Marten
   • EventStoreDB
6. Optimistic Concurrency:
   • Marten
   • EventStoreDB
7. Projections:
   • General
   • Idempotency
   • Eventual Consistency

Related PRs:

• #114
• #125
• #128
• #126
• #129

Fixed Kafka integration to correctly publish and get messages

Fixed Kafka integration to correctly publish and get messages:

• Fixed Kafka producer and consumer to correctly serialize/deserialize send Event with event envelope.
• Upgraded Kafka images to the latest 7.0.1 version.
• Added a basic test for Kafka Producer.
• Switched Kafka UI from landoop/kafka-topics-ui to landoop/kafka-ui.

See details in #120.

Besides that:

• Removed obsolete IIS express run profiles.
• Fixed async method signatures in tests.
• Fixed copy/paste naming issues in Cancel Shopping Carts API tests.

See details in #122.

Various improvements and bug fixes for Event Sourcing Samples

1. Made an alignment of ECommerce samples: #118:

• Renamed Initialize Shopping Cart to Open Shopping Cart to be more closer to real terminology,
• Added Cancel Shopping Cart operation,
• Aligned Product Items implementation.

2. Updated MartenEventPublisher to publish events with metadata #112
3. Fixed Marten Append Scope registration: #111
4. Upgraded Marten to v5 release candidate 1: #115
5. Shift tenant to aggregateId part of streamId to properly handle EventStoreDB category projections: #117

Added Correlation and CausationId support in samples

Implemented correlation id and causation id for tracing operations fully.

Refactored previously existing CorrelationIdMiddleware by adding causation id handling and renamed it to more general TracingMiddleware. It takes correlation id and causation id from HTTP headers and caches them into TracingScope.

TracingScope is also setting logging scope internally to keep Correlation id and causation id in logs. Thanks to that, the logic for tracing setup can be reused in middleware and subscriptions. The new causation id is generated based on the event id in subscriptions. Thanks to that, we can build the tree with a history of event handling.

Updated Optimistic Concurrency Scopes and generalised into AppendScope. It wraps both tracing and expected version handling. Pushed tracing metadata into Marten and EventStoreDB storage.

Replaced custom EventBus instead of Mediator one to have more flexibility (e.g. be able to create new DI and logging scopes).

See more in PRs: #106 and #108.

Added Marten Outbox Pattern/Subscription with custom Projection

Added Marten Outbox Pattern/Subscription with custom Projection Plugged it into samples removing calling EventBus from the repository, by that getting at-least-once processing guarantee.

• MartenSubscription with general subscription logic,
• MartenEventPublisher to publish to in memory bus and forward to e.g. KafkaProducer or external database (e.g. ElasticSearch).

Read more about that in: https://event-driven.io/en/integrating_Marten/.

Other changes:

• Updated Marten to latest v5 alpha
• Disabled API tests parallelisation until better test setup is provided

See more in PR: #104

Removed not working correctly Github Action with Test Results.

Removed not working correctly Github Action with Test Results.

Besides that:

• Added Directory.Build.props and removed some commonly shared settings
• Renamed project Core.Streaming.Kafka into Core.Kafka

Added full Optimistic Concurrency handling in all samples with integration to ETag

Added optimistic concurrency to samples and did a huge all-around refactoring.

The most significant changes:

• Added OptimisticConcurrencyMiddleware and related classes to support full flow based on the ETag. Applied both for Marten and EventStoreDB samples,
• Aligned convention around Records (use them for DTOs with static factory method for validation),
• Introduced StreamEvent for EventStoreDB subscriptions to gather information about stream revision and global position. That can be used for idempotency checks in projections.
• Made MartenExternalProjection, EntityFrameworkProjection idempotent,
• Merged EventStoreDB improvements from Simple EventStoreDB examples into Core project,
• Added NoMediatorEventBus to not rely on the marker interfaces need etc.
• Aligned ECommerce samples:
  • API structure and models
  • The same set of integration tests,
  • Naming and structure conventions
• Added CorrelationIdMiddleware and plugged it initially. More changes will come in the follow-up PR,
• Unified background processing for ESDB Subscriptions and Kafka Consumers

See more in: #100.

Upgraded to Marten v5 and EventStoreDB client 22.0.0

Upgraded packages to the latest version

• Marten v5 alpha,
• EventStoreDB to 22.0.0
• and the rest (e.g. MediatR, RestSharp, etc.)

Upgraded also ESDB docker images to the latest LTS version.

Strongly-Typed ids with Marten

Strongly typed ids (or, in general, a proper type system) can make your code more predictable. It reduces the chance of trivial mistakes, like accidentally changing parameters order of the same primitive type.

So for such code:

var reservationId = "RES/01";
var seatId = "SEAT/22";
var customerId = "CUS/291";

var reservation = new ReservationId (
    reservationId,
    seatId,
    customerId 
);

the compiler won't catch if you switch reservationId with seatId.

If you use strongly typed ids, then compile will catch that issue:

var reservationId = new ReservationId("RES/01");
var seatId = new SeatId("SEAT/22");
var customerId = new CustomerId("CUS/291");

var reservation = new ReservationId (
    reservationId,
    seatId,
    customerId 
);

They're not ideal, as they're usually not playing well with the storage engines. Typical issues are: serialisation, Linq queries, etc. For some cases they may be just overkill. You need to pick your poison.

To reduce tedious, copy/paste code, it's worth defining a strongly-typed id base class, like:

public class StronglyTypedValue<T>: IEquatable<StronglyTypedValue<T>> where T: IComparable<T>
{
    public T Value { get; }

    public StronglyTypedValue(T value)
    {
        Value = value;
    }

    public bool Equals(StronglyTypedValue<T>? other)
    {
        if (ReferenceEquals(null, other)) return false;
        if (ReferenceEquals(this, other)) return true;
        return EqualityComparer<T>.Default.Equals(Value, other.Value);
    }

    public override bool Equals(object? obj)
    {
        if (ReferenceEquals(null, obj)) return false;
        if (ReferenceEquals(this, obj)) return true;
        if (obj.GetType() != this.GetType()) return false;
        return Equals((StronglyTypedValue<T>)obj);
    }

    public override int GetHashCode()
    {
        return EqualityComparer<T>.Default.GetHashCode(Value);
    }

    public static bool operator ==(StronglyTypedValue<T>? left, StronglyTypedValue<T>? right)
    {
        return Equals(left, right);
    }

    public static bool operator !=(StronglyTypedValue<T>? left, StronglyTypedValue<T>? right)
    {
        return !Equals(left, right);
    }
}

Then you can define specific id class as:

public class ReservationId: StronglyTypedValue<Guid>
{
    public ReservationId(Guid value) : base(value)
    {
    }
}

You can even add additional rules:

public class ReservationNumber: StronglyTypedValue<string>
{
    public ReservationNumber(string value) : base(value)
    {
        if (string.IsNullOrEmpty(value) || value.StartsWith("RES/") || value.Length <= 4)
            throw new ArgumentOutOfRangeException(nameof(value));
    }
}

The base class working with Marten, can be defined as:

public abstract class Aggregate<TKey, T>
    where TKey: StronglyTypedValue<T>
    where T : IComparable<T>
{
    public TKey Id { get; set; } = default!;
    
    [Identity]
    public T AggregateId    {
        get => Id.Value;
        set {} 
    }

    public int Version { get; protected set; }

    [JsonIgnore] private readonly Queue<object> uncommittedEvents = new(); 

    public object[] DequeueUncommittedEvents()
    {
        var dequeuedEvents = uncommittedEvents.ToArray();

        uncommittedEvents.Clear();

        return dequeuedEvents;
    }

    protected void Enqueue(object @event)
    {
        uncommittedEvents.Enqueue(@event);
    }
}

Marten requires the id with public setter and getter of string or Guid. We used the trick and added AggregateId with a strongly-typed backing field. We also informed Marten of the Identity attribute to use this field in its internals.

Example aggregate can look like:

public class Reservation : Aggregate<ReservationId, Guid>
{
    public CustomerId CustomerId { get; private set; } = default!;

    public SeatId SeatId { get; private set; } = default!;

    public ReservationNumber Number { get; private set; } = default!;

    public ReservationStatus Status { get; private set; }

    public static Reservation CreateTentative(
        SeatId seatId,
        CustomerId customerId)
    {
        return new Reservation(
            new ReservationId(Guid.NewGuid()),
            seatId,
            customerId,
            new ReservationNumber(Guid.NewGuid().ToString())
        );
    }

    // (...)
}

See the full sample here.

Read more in the article:

• 📝 Using strongly-typed identifiers with Marten
• 📝 Immutable Value Objects are simpler and more useful than you think!

Added Event Schema Versioning samples

Shows how to handle basic event schema versioning scenarios using event and stream transformations (e.g. upcasting):

• Simple mapping
  • New not required property
  • New required property
  • Renamed property
• Upcasting
  • Changed Structure
  • New required property
• Downcasters
• Events Transformations
• Stream Transformation
• Summary

See the whole sample at: https://github.com/oskardudycz/EventSourcing.NetCore/tree/main/Sample/EventsVersioning
And details of changes in: #75