log_store/
object_store_wal.rs

1// Copyright 2023 Greptime Team
2//
3// Licensed under the Apache License, Version 2.0 (the "License");
4// you may not use this file except in compliance with the License.
5// You may obtain a copy of the License at
6//
7//     http://www.apache.org/licenses/LICENSE-2.0
8//
9// Unless required by applicable law or agreed to in writing, software
10// distributed under the License is distributed on an "AS IS" BASIS,
11// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12// See the License for the specific language governing permissions and
13// limitations under the License.
14
15//! Durable primitives of a WAL that stores entries as immutable object store
16//! objects.
17//!
18//! Entries of many regions are batched into a single object, written once and
19//! never mutated afterwards. An object is laid out as
20//!
21//! ```text
22//! header | segment (region 1) | ... | segment (region N) | footer | trailer
23//! ```
24//!
25//! The header carries the magic `GTWALOBJ`, the format version, the object
26//! sequence, the epoch of the instance that wrote the object, the link to the
27//! object it extends, and a CRC32 of the header. Each segment holds the
28//! entries of exactly one region, ordered by entry id, and segments are ordered
29//! by region id. The footer indexes every segment with its region id, entry id
30//! range, byte range and CRC32. The fixed-size trailer points at the footer and
31//! carries the CRC32 of the footer and of the whole object, so a reader locates
32//! the footer by reading the fixed-length trailer at the end of the object.
33//!
34//! Object sequences increase monotonically within one prefix and may leave
35//! gaps, so recovery continues after the largest sequence present. An object is
36//! created conditionally: rewriting a sequence with the content it already
37//! holds is a no-op at the object store, while different content under a taken
38//! sequence is a conflict. A create can fail with an unknown outcome and its
39//! object can still appear, so objects form a chain: every object links to the
40//! object it extends by sequence and epoch, and recovery replays only the chain
41//! that ends at the complete object with the largest epoch and sequence. An
42//! object is complete when every link on its chain names a present object of
43//! the recorded epoch, back to an object that starts the chain. Objects off the
44//! chain are orphans, which are never replayed but keep their sequences. Each
45//! open writes an object without segments above every present object before
46//! it accepts writes, and its epoch is one above the sequence of that object,
47//! so no two instances share an epoch and a late object of an earlier
48//! instance never ends the chain.
49//!
50//! Recovery lists the objects, reads and verifies only the header, trailer and
51//! footer of each, and indexes the footers of the chain in sequence order to
52//! rebuild the object catalog, which rejects a sequence it already holds.
53//! Segments are read and checksummed only when a read decodes them.
54//!
55//! Entry ids are object-sequence-major, see [`entry_id`]: the high bits of an
56//! id name the object that holds the entry, the low bits its position among
57//! the entries of its region in that object.
58
59mod batch;
60mod catalog;
61mod format;
62mod io;
63mod store;
64
65pub use batch::entry_id;
66pub use store::ObjectStoreLogStore;
log_store/object_store_wal.rs

log_store/
object_store_wal.rs
