Server-side stream linking and projections for reckon-db.

A *link* is a derived stream computed from one or more source streams by applying an optional filter predicate and an optional transform function. The result is persisted under a stable name and exposed as just-another-subscribable-stream — consumers read it like any user stream.

This is the same primitive Greg Young's EventStoreDB exposes as *projections*: $ce-account (category events for account-* streams), $et-UserCreated (all events of one type), or arbitrary user-defined JavaScript projections that emit into custom destination streams. reckon-db's links cover the same conceptual ground with native Erlang funs instead of a scripting layer.

Why links exist

Reckon-db already has rich typed subscription filters (by_event_type, by_event_pattern, by_event_payload, by_tags) that filter the event stream on the fly. So why a separate "link" mechanism?

• **Transform.** Typed subscriptions can only filter. Links can rewrite events (add fields, re-key, redact, denormalise) before delivery. The transform is server-side, deterministic, and replayable.
• **Materialisation.** Typed subscriptions are recomputed for every consumer. A link computes once, persists the derived events on disk, and serves them like any other stream — N consumers cost N stream-reads, not N×(source-scans + filter + transform).
• **Stable named subscription targets.** Dynamically-created source streams (order-018f6a..., order-018f6b..., ...) can't be subscribed to as a group via the bare-id selector; you'd need a wildcard pattern in the consumer. A link ($link:orders filtered to source pattern order-*) gives you one durable, named, listable target.
• **Replayability.** A new consumer attaching to a link replays the persisted derived events from the start, in order, deterministically — same shape every time, regardless of how the source streams have grown. Typed subscriptions replay by re-scanning the entire global log and re-applying the predicate, which is correct but expensive at scale.
• **Composability.** Links can source from other links — $link:high-value-orders built on top of $link:orders. Useful for incremental refinement of a view.
• **Operational visibility.** Named derived streams show up in store listings (lazyreckon's streams pane, for example) — operators can see "what derived views does this store maintain" at a glance. Typed subscriptions are ephemeral client state and invisible to tooling.

Stream-id namespace

Link streams live in the $ system namespace:

$link:<human-readable-name>

e.g. $link:high-value-orders. The $ prefix is the reckon-db convention for "system / projected stream, not user data" — same role as EventStoreDB's $ce- / $et- / $by_category / $stats- prefixes. User streams follow the <prefix>-<hex> format (e.g. account-018f6a7b8c9d4abc...). Both forms are accepted as subscription selectors via reckon_db_filters:by_stream/1.

Link subscriptions themselves get a related namespace:

$link-sub:<link-name>

so subscriptions managed by the link engine are distinguishable from user subscriptions in operational tooling.

Use cases (worked examples)

1. Materialised category view. Collate every event from all order-<hex> streams into one subscribable stream:

  reckon_db_links:create(my_store, #{
      name   => <<"orders">>,
      source => #{type => stream_pattern, pattern => <<"order-*">>}
  }).
  reckon_db_subscriptions:subscribe(
      my_store, stream, <<"$link:orders">>, <<"orders-projector">>).

2. Filtered derived stream. Just the high-value subset:

  reckon_db_links:create(my_store, #{
      name   => <<"high-value-orders">>,
      source => #{type => stream_pattern, pattern => <<"order-*">>},
      filter => fun(E) ->
          maps:get(total, E#event.data, 0) > 1000
      end
  }).

3. Transform / re-shape. Strip PII from outbound webhook events:

  reckon_db_links:create(my_store, #{
      name      => <<"webhook-feed">>,
      source    => #{type => stream_pattern, pattern => <<"order-*">>},
      transform => fun(E) ->
          D2 = maps:without([customer_email, customer_phone],
                            E#event.data),
          E#event{data = D2}
      end
  }).

4. Composition. Layer a filter on top of an existing link:

  reckon_db_links:create(my_store, #{
      name   => <<"high-value-webhook">>,
      source => #{type => stream, stream_id => <<"$link:webhook-feed">>},
      filter => fun(E) ->
          maps:get(total, E#event.data, 0) > 1000
      end
  }).

When NOT to use a link

• Single-consumer one-off filter — a by_event_type or by_tags subscription is cheaper and avoids persistent derived data.
• Read model that lives in a database (postgres, sqlite, etc) — subscribe directly with the appropriate typed filter and project to the database. The link layer adds no value when the materialised target is elsewhere.
• Aggregation across many sources where the math is non-trivial — links emit one derived event per source event; they don't fold. For aggregation, use a process manager.

Running a link

start/2 starts the link's consumer (reckon_db_link_consumer), which follows the source live and, with backfill => true, writes what the source already held first. Each source event that passes the filter reaches the link stream once, also across a consumer crash or a store restart: the consumer stores, per source stream, the last version it handled. A link stored as running is resumed when the store's leader activates. stop/2 ends the consumer and removes its subscription; a later start/2 picks up from where it stopped, so the link also gets what the source received while it was stopped.

A link that cannot progress (a transform that crashes or returns something other than an event) stops with status error and the reason, and writes nothing for that event; the other links keep running. Fix the cause, then start/2 it again.

Status

Live delivery works for every source type since reckon-db#24; before that a link only ever held its backfill. Treat links as preview while we accumulate real usage.