reckon_db_dcb_reindex (reckon_db v5.11.14)

View Source

One-time re-index of DCB events written before they got secondary index entries (reckon-db #2).

Up to 5.11.10 the DCB append wrote its own by_tag / by_event_type / payload entries but never the [idx] entries of reckon_db_index, so on a store declaring tags, event_type or {meta, Key} the indexed read_by_tags / read_by_event_types / read_by_metadata never returned a DCB event. A DCB-context decision on such a store read an empty context and could not commit into a non-empty boundary. The append now writes those entries; this gives the events written before it theirs.

Once

A marker at [metadata, index, dcb_indexed] lists the kinds whose entries the DCB log already has. A run re-indexes only the declared kinds missing from it, and writes the marker after every batch has landed: a run that dies half way leaves no marker and the next one redoes it (the entries are keyed by path, so writing one twice is harmless). With the marker in place a run is one read. A kind declared later, never covered before, is re-indexed on its own. The marker trusts that every DCB append since it was written came from 5.11.11 or later; when that is not so (a rolling upgrade where an old member appended after the marker, or a kind that was undeclared and declared again), run(StoreId, #{force => true}) ignores it and redoes every declared kind.

Through the Ra leader

Every member of a cluster opens the store, so run/1 works only on the Ra leader and answers not_leader elsewhere; reckon_db_leader runs it, in a process of its own, on activation, which the node monitor does on the leader after a start and after every leadership change. Two runs can still overlap across a leadership change; they stay correct (same entries, same paths, the marker is a union) and each scans the log once.

Summary

Functions

Re-index the DCB log for the declared kinds it has no entries for yet. {ok, not_leader} on a member that is not the Ra leader.

As run/1. With #{force => true} the marker is ignored and every declared kind is re-indexed: the way to redo it when DCB events got in without entries after the marker was written, e.g. appended by a member still on 5.11.10 during a rolling upgrade, or while a kind was undeclared and then declared again.

Types

index_decl/0

-type index_decl() ::
          tags | event_type |
          {meta, Key :: binary()} |
          {payload, Key :: binary()} |
          {payload_hash, Keys :: [binary()]}.

outcome/0

-type outcome() ::
          #{reindexed := non_neg_integer(), kinds := [index_decl()], duration_us := non_neg_integer()}.

Functions

run(StoreId)

-spec run(atom()) -> {ok, outcome() | not_leader} | {error, term()}.

Re-index the DCB log for the declared kinds it has no entries for yet. {ok, not_leader} on a member that is not the Ra leader.

run(StoreId, Opts)

-spec run(atom(), #{force => boolean()}) -> {ok, outcome() | not_leader} | {error, term()}.

As run/1. With #{force => true} the marker is ignored and every declared kind is re-indexed: the way to redo it when DCB events got in without entries after the marker was written, e.g. appended by a member still on 5.11.10 during a rolling upgrade, or while a kind was undeclared and then declared again.