Module store 

Object store WAL construction, recovery, writes and region reads.

Structs

Actor 🔒

The actor that owns the open batch and the sealed batches until they are durable.

DeletingObjects 🔒

The delete attempts in flight, one per object.

DurableWaiter 🔒

A caller waiting until a region is durable through an entry id.

Failing 🔒

The state of an enqueued store whose backlog the object store refuses with an error that requires intervention, see requires_intervention, such as denied access or a missing bucket.

FetchedObject 🔒

An object as recovery fetched it: its key, header and footer, and its segment of release records when the reads of the footer returned it.

FetchedObjects 🔒

The objects recovery fetched, ordered by sequence, and the listed objects that were gone when their header or footer was fetched, with the error the fetch met. A collection of an earlier instance may complete after the listing.

ObjectStoreLogStore

A log store over the immutable WAL objects under one prefix.

PendingAppend 🔒

An append waiting for the object that holds its entries.

Recovered 🔒

What recovery rebuilt from the objects under a prefix.

SealedBatch 🔒

A sealed and encoded batch that holds its object sequence and waits to be created, indexed and acknowledged.

WalHole

A segment a read skipped because it did not decode: the entries of one region in one object are missing from every read of that region.

Enums

Command 🔒

CreateState 🔒

Where the conditional create of a sealed batch stands.

Withdrawal 🔒

Whether the store gave up its start object, see Actor::handle_withdraw.

Constants

APPEND_BUFFER 🔒

Appends queued for the actor; a caller beyond them waits to send.

COMMAND_BUFFER 🔒

CREATE_RETRY_DELAY 🔒

Delay before a create that failed transiently is attempted again in the enqueued acknowledgement mode, where no caller is left to retry it.

FAILING_LOG_INTERVAL 🔒

How often a store logs that the object store still refuses its backlog.

MAX_FAILING_RETRY_DELAY 🔒

Longest delay from a failed attempt of a backlog that the object store refuses with an error that requires intervention, see Failing, to the next attempt. The delay doubles from CREATE_RETRY_DELAY after every attempt. At the cap a dead bucket costs four requests a minute, a conditional create and the read that reconciles it every 30 seconds; a longer cap would keep appends rejected longer after the cause is fixed.

MAX_IN_FLIGHT_CREATES 🔒

MAX_IN_FLIGHT_DELETES 🔒

MAX_KEPT_RELEASE_BYTES 🔒

Largest segment of release records that recovery keeps from the bytes it read for a footer. It keeps them before it knows which objects are on the chain, so the bound caps what objects off the chain hold until recovery ends; a larger segment is read again if its object is on the chain. A release seals its record at once, so a segment usually holds the records of one or a few regions, 16 bytes each.

MAX_SEALED_BATCHES 🔒

Number of sealed batches that may wait to be created or indexed. One admission can seal two batches, the open batch before an append that would exhaust its positions and the append’s own batch, so the actor admits an append only while two more fit and holds it back otherwise: a stalled object store holds callers back instead of growing the backlog.

MIN_FLUSH_INTERVAL 🔒

READ_PREFETCH_BYTES 🔒

Bytes of decoded segments that all reads of a store together fetch ahead; a segment counts as its length, which bounds its payloads, plus a Record per entry. A read always fetches the segment it yields next, as a read that fetched one segment at a time did, and a later segment only while it fits in what is left, so regions replayed in parallel hold one segment each, up to about max_batch_bytes or more for one oversized append, plus at most these bytes, however many regions open at once. While a fetch copies the payloads out of the downloaded segment it briefly holds the segment’s length once more.

READ_PREFETCH_DEPTH 🔒

Number of segments a read fetches ahead of the one whose entries it yields. On S3 a replay that fetched each segment after the previous one spent about 30 ms per object; with eight fetches in flight a segment arrives about every 4 ms. The segments fetched ahead also take their bytes from READ_PREFETCH_BYTES.

RECOVERY_CONCURRENCY 🔒

Number of objects whose footers recovery fetches at a time. On S3 an object costs two reads, its header and its tail, tens of milliseconds per lane however many vCPUs the node has, and recovery time scales with objects divided by concurrency up to 128 lanes without throttling: at 64 about 10,000 objects recover in about 10 s instead of about 100 s at 8, while the reads use about a third of S3’s request rate per prefix, leaving room for the data file reads of start-up. Each lane holds one RECOVERY_TAIL_WINDOW, 4 MiB in all.

RECOVERY_TAIL_WINDOW 🔒

Bytes recovery reads from the end of an object in one request. The window holds the trailer and the footer of an object with up to 1364 regions of 48 bytes each, so a second request for the footer is rare, and usually the segment of release records, which is the last segment.

Traits

WalObjectIo 🔒

Object access of the store, so tests can inject failures.

Functions

chain_ending_at 🔒

Returns, in sequence order, the chain that ends at tip if tip is complete, see select_chain.

durable_entry_ids 🔒

encode_batch 🔒

Encodes the entries of a batch, and its release records as the single entry of RELEASE_REGION_ID, whose id is at position 1 of the object; segments are ordered by region id, so it is the last segment.

epoch_of 🔒

fetch_footer 🔒

Reads the header, trailer and footer of object and verifies them: the header must carry the sequence of the key, the trailer must be well formed, the footer must match the checksum the trailer holds and its segments must tile the object body. Also returns a copy of the segment of release records when the bytes read hold it and it is at most MAX_KEPT_RELEASE_BYTES, which is not decoded here.

fetch_footers 🔒

Fetches and verifies the headers and footers of objects as they are listed, up to concurrency objects at a time, and returns them ordered by object sequence whatever the order the fetches complete in. An object the object store no longer has is returned as vanished; a failed listing or any other failure abandons the remaining fetches.

fetch_segment 🔒

Fetches and decodes the segment entry describes in the object object_seq. A segment that does not decode is fetched once more, since the download rather than the object may be damaged; when it still does not decode the error is returned as Error::InvalidWalObject. A failed fetch is returned as is.

finish_recovery 🔒

Indexes the chain of the fetched objects. Objects off the chain, orphans and objects below the reclaim boundary of the tip, are not indexed, but no later object takes their sequences, nor the sequences of vanished objects. A vanished object at or above the boundary of the tip fails recovery with the error its fetch met: collection never deletes such an object.

is_collected 🔒

Returns whether collection deleted the object object_seq after a read listed it. The set of deleting objects is observed before the catalog, the reverse of the order the actor updates them, so an object that no longer looks deleting but still looks indexed was never collected. An undecided delete is waited for.

is_not_found 🔒

locate_footer 🔒

Verifies the header and trailer of the object object_seq of object_len bytes from its first bytes head and its last bytes tail, and returns the header and the trailer with the range the footer occupies in the object.

positive_bytes 🔒

publish_indexed 🔒

Sets both indexed gauges to the objects catalog holds.

read_releases 🔒

Reads the release records of the indexed objects, which seed the watermarks of their regions, from the segment of RELEASE_REGION_ID of every indexed object that holds one: the bytes in fetched by object sequence, which the reads of the footers returned, or else one range read. The segments of objects that are not indexed are never decoded. A segment that cannot be fetched or does not decode fails recovery, whatever on_corrupted_segment says: skipping it would lose the watermarks it seeds.

record_create 🔒

Counts the outcome of a conditional create and, for the object of a batch sealed at sealed_at, times a success. The create of a batch is counted whether or not the batch is still there to be indexed: an object a poisoned store gave up on is as durable as any other.

record_obsolete 🔒

Records the obsolete watermark of region_id, which never moves down, for reads and in the catalog for the reclaim boundary.

record_skipped_segment 🔒

Records the segment entry describes in the object object_seq as a hole of its region, once per object, counts the skip and warns about it. An object collected meanwhile is skipped without a hole: the catalog is held while the hole is recorded, as a collection holds it while it drops the holes of the object it removes.

recover 🔒

Rebuilds the catalog from object headers and footers, so recovery costs a few small reads per object however large the objects are. Segments are not read, except those of release records, see read_releases; a segment checksum is verified by the read that decodes it. Footers are fetched for up to RECOVERY_CONCURRENCY objects at a time while the listing goes on, and indexed in sequence order, so the catalog checks the entry ranges of every object against its predecessors like a sequential replay. Only the objects on the chain select_chain picks are indexed.

requires_intervention 🔒

Returns true for an object store error that repeating the request cannot settle until someone changes the credentials, the configuration or the bucket: denied access; an invalid configuration, which is how OpenDAL reports a missing S3 bucket; and a write to a location that does not exist, which is how OSS, GCS and Azure Blob Storage report a missing bucket or container. A read that finds nothing is not one: it is what the read that reconciles a create meets when the object is gone, and a repeat creates it. Every other error, an HTTP 400 or 5xx, a timeout or a refused connection, may pass by itself. A create that fails with an error requiring intervention is still repeated, with the backoff Failing describes.

select_chain 🔒

Returns, in sequence order, the objects on the chain that ends at the tip: the complete object with the largest epoch, then the largest sequence.

set_terminal 🔒

Records error as the terminal error unless one is already recorded, and returns the recorded one.

shared 🔒

Wraps an error that several callers receive.

spawn_fetch_segment 🔒

Runs fetch_segment on a task of its own, so the fetch proceeds while the read is busy with earlier segments, and returns the outcome with permit. The task holds the permit, so the reserved bytes are released only with the buffers of the fetch. Dropping the returned future, as a dropped read does, aborts the task; a task that panics fails the fetch with Error::WalSegmentFetch.

stale_conflict 🔒

Resolves a create of this store’s epoch that met a different object: an object of an earlier epoch can never be on the chain, so the create fails like a transient error; an object of the same or a later epoch belongs to another writer and conflict is returned.

start_epoch 🔒

Writes the empty object that starts the epoch of this instance at object_seq, linked to the recovered tip and carrying the recovered reclaim_boundary, and returns it as the tip later objects extend, together with the sequences of earlier-epoch objects it moved past. The epoch is one above the sequence the create claims, so no two instances share one however their opens interleave, and it is above the epoch of every object recovery listed.

terminal 🔒

try_reserve_prefetch_bytes 🔒

Takes from budget the bytes the segment entry describes holds once decoded, if they are free: its length plus a Record per entry.

Type Aliases

AppendResponse 🔒

CreateOutcome 🔒

ObsoleteEntryIds 🔒

QueuedAppend 🔒

StartedRegions 🔒

Regions whose history starts in the store: those created on it and those it opened by accepting a missing history. The store registers a region and the actor decides a withdrawal under this lock, so neither misses the other.

TerminalError 🔒

WalHoles 🔒