Struct LocalGcWorker 

pub struct LocalGcWorker {
    pub(crate) access_layer: AccessLayerRef,
    pub(crate) cache_manager: Option<CacheManagerRef>,
    pub(crate) manifest_mgrs: HashMap<RegionId, RegionManifestManager>,
    pub(crate) opt: FileGcOption,
    pub(crate) manifest_open_config: ManifestOpenConfig,
    pub(crate) file_ref_manifest: TableFileRefsManifest,
}

Fields

access_layer: AccessLayerRef

cache_manager: Option<CacheManagerRef>

manifest_mgrs: HashMap<RegionId, RegionManifestManager>

opt: FileGcOption

Lingering time before deleting files.

manifest_open_config: ManifestOpenConfig

file_ref_manifest: TableFileRefsManifest

Tmp ref files manifest, used to determine which files are still in use by ongoing queries.

Also contains manifest versions of regions when the tmp ref files are generated. Used to determine whether the tmp ref files are outdated.

Implementations

impl LocalGcWorker

pub async fn try_new( access_layer: AccessLayerRef, cache_manager: Option<CacheManagerRef>, regions_to_gc: BTreeSet<RegionId>, opt: FileGcOption, manifest_open_config: ManifestOpenConfig, file_ref_manifest: TableFileRefsManifest, ) -> Result<Self>

Create a new LocalGcWorker, with regions_to_gc regions to GC. The regions are specified by their RegionId and should all belong to the same table.

pub async fn read_tmp_ref_files( &self, outdated_regions: &mut HashSet<RegionId>, ) -> Result<HashMap<RegionId, HashSet<FileId>>>

Get tmp ref files for all current regions

Outdated regions are added to outdated_regions set

pub async fn run(self) -> Result<GcReport>

Run the GC worker in serial mode, considering list files could be slow and run multiple regions in parallel may cause too many concurrent listing operations.

TODO(discord9): consider instead running in parallel mode

impl LocalGcWorker

pub const CONCURRENCY_LIST_PER_FILES: usize = 512usize

concurrency of listing files per region. This is used to limit the number of concurrent listing operations and speed up listing

pub async fn do_region_gc( &self, region_id: RegionId, tmp_ref_files: &HashSet<FileId>, ) -> Result<Vec<FileId>>

Perform GC for the region.

1. Get all the removed files in delta manifest files and their expel times
2. List all files in the region dir concurrently
3. Filter out the files that are still in use or may still be kept for a while
4. Delete the unused files

Note that the files that are still in use or may still be kept for a while are not deleted to avoid deleting files that are still needed.

async fn delete_files( &self, region_id: RegionId, file_ids: &[FileId], ) -> Result<()>

async fn open_mgr_for( &self, region_id: RegionId, ) -> Result<RegionManifestManager>

Get the manifest manager for the region.

pub async fn get_removed_files_expel_times( &self, region_id: RegionId, ) -> Result<BTreeMap<Timestamp, HashSet<FileId>>>

Get all the removed files in delta manifest files and their expel times. The expel time is the time when the file is considered as removed. Which is the last modified time of delta manifest which contains the remove action.

async fn partition_region_files( &self, region_id: RegionId, concurrency: usize, ) -> Result<Vec<(Lister, Option<String>)>>

Create partitioned listers for concurrent file listing based on concurrency level. Returns a vector of (lister, end_boundary) pairs for parallel processing.

async fn list_region_files_concurrent( &self, listers: Vec<(Lister, Option<String>)>, ) -> Result<Vec<Entry>>

Concurrently list all files in the region directory using the provided listers. Returns a vector of all file entries found across all partitions.

fn filter_deletable_files( &self, entries: Vec<Entry>, in_use_filenames: &HashSet<&FileId>, may_linger_filenames: &HashSet<&FileId>, all_files_appear_in_delta_manifests: &HashSet<&FileId>, unknown_file_may_linger_until: DateTime<Utc>, ) -> (Vec<FileId>, HashSet<FileId>)

Filter files to determine which ones can be deleted based on usage status and lingering time. Returns a vector of file IDs that are safe to delete.

pub async fn list_to_be_deleted_files( &self, region_id: RegionId, in_used: HashSet<FileId>, recently_removed_files: BTreeMap<Timestamp, HashSet<FileId>>, concurrency: usize, ) -> Result<Vec<FileId>>

Concurrently list unused files in the region dir because there may be a lot of files in the region dir and listing them may take a long time.