Struct BloomFilterIndexer

pub struct BloomFilterIndexer {
    creators: HashMap<ColumnId, BloomFilterCreator>,
    temp_file_provider: Arc<TempFileProvider>,
    codec: IndexValuesCodec,
    aborted: bool,
    stats: Statistics,
    global_memory_usage: Arc<AtomicUsize>,
}

The indexer for the bloom filter index.

Fields

creators: HashMap<ColumnId, BloomFilterCreator>

The bloom filter creators.

temp_file_provider: Arc<TempFileProvider>

The provider for intermediate files.

codec: IndexValuesCodec

Codec for decoding primary keys.

aborted: bool

Whether the indexing process has been aborted.

stats: Statistics

The statistics of the indexer.

global_memory_usage: Arc<AtomicUsize>

The global memory usage.

Implementations

impl BloomFilterIndexer

pub fn new( sst_file_id: FileId, metadata: &RegionMetadataRef, intermediate_manager: IntermediateManager, memory_usage_threshold: Option<usize>, ) -> Result<Option<Self>>

Creates a new bloom filter indexer.

pub async fn update(&mut self, batch: &mut Batch) -> Result<()>

Updates index with a batch of rows. Garbage will be cleaned up if failed to update.

TODO(zhongzc): duplicate with mito2::sst::index::inverted_index::creator::InvertedIndexCreator

pub async fn finish( &mut self, puffin_writer: &mut <FsPuffinManager<Arc<BoundedStager<FileId>>, ObjectStorePuffinFileAccessor> as PuffinManager>::Writer, ) -> Result<(usize, u64)>

Finishes index creation and cleans up garbage. Returns the number of rows and bytes written.

TODO(zhongzc): duplicate with mito2::sst::index::inverted_index::creator::InvertedIndexCreator

pub async fn abort(&mut self) -> Result<()>

Aborts index creation and clean up garbage.

TODO(zhongzc): duplicate with mito2::sst::index::inverted_index::creator::InvertedIndexCreator

async fn do_update(&mut self, batch: &mut Batch) -> Result<()>

async fn do_finish( &mut self, puffin_writer: &mut <FsPuffinManager<Arc<BoundedStager<FileId>>, ObjectStorePuffinFileAccessor> as PuffinManager>::Writer, ) -> Result<()>

TODO(zhongzc): duplicate with mito2::sst::index::inverted_index::creator::InvertedIndexCreator

async fn do_cleanup(&mut self) -> Result<()>

async fn do_finish_single_creator( col_id: &ColumnId, creator: &mut BloomFilterCreator, puffin_writer: &mut <FsPuffinManager<Arc<BoundedStager<FileId>>, ObjectStorePuffinFileAccessor> as PuffinManager>::Writer, ) -> Result<u64>

Data flow of finishing index:

                              (In Memory Buffer)
                                   ┌──────┐
 ┌─────────────┐                   │ PIPE │
 │             │ write index data  │      │
 │ IndexWriter ├──────────────────►│ tx   │
 │             │                   │      │
 └─────────────┘                   │      │
                 ┌─────────────────┤ rx   │
 ┌─────────────┐ │ read as blob    └──────┘
 │             │ │
 │ PuffinWriter├─┤
 │             │ │ copy to file    ┌──────┐
 └─────────────┘ └────────────────►│ File │
                                   └──────┘

TODO(zhongzc): duplicate with mito2::sst::index::inverted_index::creator::InvertedIndexCreator

pub fn memory_usage(&self) -> usize

Returns the memory usage of the indexer.

pub fn column_ids(&self) -> impl Iterator<Item = ColumnId> + use<'_>

Returns the column ids to be indexed.