reckon_db_scavenge (reckon_db v5.11.14)
View SourceScavenging and archival for reckon-db
Provides functionality to: - Remove old events beyond a retention period - Optionally archive events before deletion - Maintain stream integrity by requiring snapshots
Use cases: - Reduce storage costs by removing old events - Comply with data retention policies - Archive events to cold storage
Safety guarantees: - By default, requires a snapshot before scavenging - Supports dry-run mode for previewing changes - Telemetry events for monitoring
Summary
Functions
Archive events to a backend, then scavenge.
Preview what would be scavenged without making changes.
Scavenge a single stream.
Scavenge all streams matching a pattern.
Types
-type event() :: #event{event_id :: binary(), event_type :: binary(), stream_id :: binary(), version :: non_neg_integer(), data :: map() | binary(), metadata :: map(), tags :: [binary()] | undefined, timestamp :: integer(), epoch_us :: integer(), data_content_type :: binary(), metadata_content_type :: binary(), prev_event_hash :: binary() | undefined, mac :: mac_value(), signature :: binary() | undefined}.
-type integrity_failure_kind() ::
mac_mismatch | chain_mismatch | missing_integrity | snapshot_anchor_mismatch |
snapshot_mac_mismatch.
-type mac_value() :: {KeyId :: non_neg_integer(), MacBytes :: binary()} | {v2, KeyId :: non_neg_integer(), MacBytes :: binary()} | undefined.
-type scavenge_error() :: store_integrity_violation() | {no_snapshot, binary()} | {stream_not_found, binary()} | reckon_db_stream_path:validation_error().
-type scavenge_opts() :: #{before => integer(), before_version => integer(), keep_versions => pos_integer(), require_snapshot => boolean(), dry_run => boolean()}.
-type scavenge_result() :: #{stream_id := binary(), deleted_count := non_neg_integer(), deleted_versions := {0, non_neg_integer()}, archived := boolean(), archive_key => binary(), dry_run => true}.
-type store_integrity_violation() :: {integrity_violation, #{layer := storage | snapshot, stream_id := binary(), version := non_neg_integer(), kind := integrity_failure_kind(), context := map()}}.
Functions
-spec archive_and_scavenge(atom(), binary(), {module(), term()}, scavenge_opts()) -> {ok, scavenge_result()} | {error, term()}.
Archive events to a backend, then scavenge.
First archives events to the specified backend, then deletes them. This ensures events are preserved before removal.
-spec dry_run(atom(), binary(), scavenge_opts()) -> {ok, scavenge_result()} | {error, term()}.
Preview what would be scavenged without making changes.
-spec scavenge(atom(), binary(), scavenge_opts()) -> {ok, scavenge_result()} | {error, scavenge_error()}.
Scavenge a single stream.
Deletes old events based on the provided options. By default, requires a snapshot to exist (safety measure).
Options: - before`: Delete events with epoch_us before this timestamp - `before_version`: Delete events before this version (alternative to `before`) - `keep_versions`: Always keep at least N latest versions (default: 0) - `require_snapshot`: Require snapshot exists (default: true) - `dry_run`: Preview only, dont delete (default: false)
Example:
%% Delete events older than 1 year, keep last 10 versions
OneYearAgo = erlang:system_time(microsecond) - (365 * 24 * 60 * 60 * 1000000),
{ok, Result} = reckon_db_scavenge:scavenge(my_store, <<"orders-123">>, #{
before => OneYearAgo,
keep_versions => 10
}).-spec scavenge_matching(atom(), binary(), scavenge_opts()) -> {ok, [scavenge_result()]} | {error, term()}.
Scavenge all streams matching a pattern.
Pattern uses shell-like wildcards (e.g., "orders-*"). Applies the same options to all matching streams.