Struct LocalGcWorker 

pub struct LocalGcWorker {
    pub(crate) access_layer: AccessLayerRef,
    pub(crate) cache_manager: Option<CacheManagerRef>,
    pub(crate) manifest_mgrs: HashMap<RegionId, RegionManifestManager>,
    pub(crate) opt: GcConfig,
    pub(crate) manifest_open_config: ManifestOpenConfig,
    pub(crate) file_ref_manifest: FileRefsManifest,
    _permit: OwnedSemaphorePermit,
    pub full_file_listing: bool,
}

Fields

access_layer: AccessLayerRef

cache_manager: Option<CacheManagerRef>

manifest_mgrs: HashMap<RegionId, RegionManifestManager>

opt: GcConfig

Lingering time before deleting files.

manifest_open_config: ManifestOpenConfig

file_ref_manifest: FileRefsManifest

Tmp ref files manifest, used to determine which files are still in use by ongoing queries.

Also contains manifest versions of regions when the tmp ref files are generated. Used to determine whether the tmp ref files are outdated.

_permit: OwnedSemaphorePermit

full_file_listing: bool

Whether to perform full file listing during GC. When set to false, GC will only delete files that are tracked in the manifest’s removed_files, which can significantly improve performance by avoiding expensive list operations. When set to true, GC will perform a full listing to find and delete orphan files (files not tracked in the manifest).

Set to false for regular GC operations to optimize performance. Set to true periodically or when you need to clean up orphan files.

Implementations

impl LocalGcWorker

pub async fn try_new( access_layer: AccessLayerRef, cache_manager: Option<CacheManagerRef>, regions_to_gc: BTreeSet<RegionId>, opt: GcConfig, manifest_open_config: ManifestOpenConfig, file_ref_manifest: FileRefsManifest, limiter: &GcLimiterRef, full_file_listing: bool, ) -> Result<Self>

Create a new LocalGcWorker, with regions_to_gc regions to GC. The regions are specified by their RegionId and should all belong to the same table.

pub async fn read_tmp_ref_files( &self, outdated_regions: &mut HashSet<RegionId>, ) -> Result<HashMap<RegionId, HashSet<FileId>>>

Get tmp ref files for all current regions

Outdated regions are added to outdated_regions set

pub async fn run(self) -> Result<GcReport>

Run the GC worker in serial mode, considering list files could be slow and run multiple regions in parallel may cause too many concurrent listing operations.

TODO(discord9): consider instead running in parallel mode

impl LocalGcWorker

pub const CONCURRENCY_LIST_PER_FILES: usize = 1_024usize

concurrency of listing files per region. This is used to limit the number of concurrent listing operations and speed up listing

pub async fn do_region_gc( &self, region_id: RegionId, tmp_ref_files: &HashSet<FileId>, ) -> Result<Vec<FileId>>

Perform GC for the region.

1. Get all the removed files in delta manifest files and their expel times
2. List all files in the region dir concurrently
3. Filter out the files that are still in use or may still be kept for a while
4. Delete the unused files

Note that the files that are still in use or may still be kept for a while are not deleted to avoid deleting files that are still needed.

async fn delete_files( &self, region_id: RegionId, file_ids: &[(FileId, FileId)], ) -> Result<()>

async fn open_mgr_for( &self, region_id: RegionId, ) -> Result<RegionManifestManager>

Get the manifest manager for the region.

pub async fn get_removed_files_expel_times( &self, region_id: RegionId, ) -> Result<BTreeMap<Timestamp, HashSet<FileId>>>

Get all the removed files in delta manifest files and their expel times. The expel time is the time when the file is considered as removed. Which is the last modified time of delta manifest which contains the remove action.

async fn partition_region_files( &self, region_id: RegionId, concurrency: usize, ) -> Result<Vec<(Lister, Option<String>)>>

Create partitioned listers for concurrent file listing based on concurrency level. Returns a vector of (lister, end_boundary) pairs for parallel processing.

async fn list_region_files_concurrent( &self, listers: Vec<(Lister, Option<String>)>, ) -> Result<Vec<Entry>>

Concurrently list all files in the region directory using the provided listers. Returns a vector of all file entries found across all partitions.

fn filter_deletable_files( &self, entries: Vec<Entry>, in_use_filenames: &HashSet<&FileId>, may_linger_filenames: &HashSet<&FileId>, eligible_for_removal: &HashSet<&FileId>, unknown_file_may_linger_until: DateTime<Utc>, ) -> (Vec<FileId>, HashSet<FileId>)

Filter files to determine which ones can be deleted based on usage status and lingering time. Returns a vector of file IDs that are safe to delete.

pub async fn list_to_be_deleted_files( &self, region_id: RegionId, in_used: HashSet<FileId>, recently_removed_files: BTreeMap<Timestamp, HashSet<FileId>>, concurrency: usize, ) -> Result<Vec<FileId>>

Concurrently list unused files in the region dir because there may be a lot of files in the region dir and listing them may take a long time.

When full_file_listing is false, this method will only delete files tracked in recently_removed_files without performing expensive list operations, which significantly improves performance. When full_file_listing is true, it performs a full listing to find and delete orphan files.