Expand description
GC worker which periodically checks and removes unused/obsolete SST files.
expel time: the time when the file is considered as removed, as in removed from the manifest.
lingering time: the time duration before deleting files after they are removed from manifest.
delta manifest: the manifest files after the last checkpoint that contains the changes to the manifest.
delete time: the time when the file is actually deleted from the object store.
unknown files: files that are not recorded in the manifest, usually due to saved checkpoint which remove actions before the checkpoint.
Structs§
- GcConfig
- GcLimiter
- Limit the amount of concurrent GC jobs on the datanode
- Local
GcWorker - Manifest
Open Config
Functions§
- gen_
partition_ 🔒from_ concurrency - Generate partition prefixes based on concurrency and assume file names are evenly-distributed uuid string, to evenly distribute files across partitions. For example, if concurrency is 2, partition prefixes will be: [“8”] so it divide uuids into two partitions based on the first character. If concurrency is 32, partition prefixes will be: [“08”, “10”, “18”, “20”, “28”, “30”, “38” …, “f0”, “f8”] if concurrency is 1, it returns an empty vector.