Struct FileReferenceManager

pub struct FileReferenceManager {
    node_id: Option<u64>,
    files_per_table: DashMap<TableId, TableFileRefs>,
}

Manages all file references in one datanode. It keeps track of which files are referenced and group by table ids. And periodically update the references to tmp file in object storage. This is useful for ensuring that files are not deleted while they are still in use by any query.

Fields

node_id: Option<u64>

Datanode id. used to determine tmp ref file name.

files_per_table: DashMap<TableId, TableFileRefs>

TODO(discord9): use no hash hasher since table id is sequential.

Implementations

impl FileReferenceManager

pub fn new(node_id: Option<u64>) -> Self

fn ref_file_set(&self, table_id: TableId) -> Option<HashSet<FileRef>>

pub(crate) async fn get_snapshot_of_unmanifested_refs( &self, table_id: TableId, region_map: &Arc<RegionMap>, ) -> Result<TableFileRefsManifest>

Gets all ref files for the given table id, excluding those already in region manifest.

It’s safe if manifest version became outdated when gc worker is called, as gc worker will check the changes between those two versions and act accordingly to make sure to get the real truly tmp ref file sets at the time of old manifest version.

TODO(discord9): Since query will only possible refer to files in latest manifest when it’s started, the only true risks is files removed from manifest between old version(when reading refs) and new version(at gc worker), so in case of having outdated manifest version, gc worker should make sure not to delete those files(Until next gc round which will use the latest manifest version and handle those files normally). or perhaps using a two-phase commit style process where it proposes a set of files for deletion and then verifies no new references have appeared before committing the delete.

gc worker could do this:

1. if can get the files that got removed from old manifest to new manifest, then shouldn’t delete those files even if they are not in tmp ref file, other files can be normally handled(deleted if not in use, otherwise keep) and report back allow next gc round to handle those files with newer tmp ref file sets.
2. if can’t get the files that got removed from old manifest to new manifest(possible if just did a checkpoint), then can do nothing as can’t sure whether a file is truly unused or just tmp ref file sets haven’t report it, so need to report back and try next gc round to handle those files with newer tmp ref file sets.

pub fn add_file(&self, file_meta: &FileMeta)

Adds a new file reference. Also records the access layer for the table if not exists. The access layer will be used to upload ref file to object storage.

pub fn remove_file(&self, file_meta: &FileMeta)

Removes a file reference. If the reference count reaches zero, the file reference will be removed from the manager.

Trait Implementations

impl Debug for FileReferenceManager

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.