Struct InvertedIndexer 

pub struct InvertedIndexer {
    index_creator: Box<dyn InvertedIndexCreator>,
    temp_file_provider: Arc<TempFileProvider>,
    codec: IndexValuesCodec,
    value_buf: Vec<u8>,
    pk_offsets: SparseOffsetsCache,
    stats: Statistics,
    aborted: bool,
    memory_usage: Arc<AtomicUsize>,
    indexed_column_ids: Vec<(ColumnId, String)>,
    metadata: RegionMetadataRef,
}

InvertedIndexer creates inverted index for SST files.

Fields

index_creator: Box<dyn InvertedIndexCreator>

The index creator.

temp_file_provider: Arc<TempFileProvider>

The provider of intermediate files.

codec: IndexValuesCodec

Codec for decoding primary keys.

value_buf: Vec<u8>

Reusable buffer for encoding index values.

pk_offsets: SparseOffsetsCache

Scratch offsets shared by indexed tags of one sparse primary key.

stats: Statistics

Statistics of index creation.

aborted: bool

Whether the index creation is aborted.

memory_usage: Arc<AtomicUsize>

The memory usage of the index creator.

indexed_column_ids: Vec<(ColumnId, String)>

Ids of indexed columns and their encoded target keys.

metadata: RegionMetadataRef

Region metadata for column lookups.

Implementations

impl InvertedIndexer

pub fn new( sst_file_id: FileId, metadata: &RegionMetadataRef, intermediate_manager: IntermediateManager, memory_usage_threshold: Option<usize>, segment_row_count: NonZeroUsize, indexed_column_ids: HashSet<ColumnId>, ) -> Self

Creates a new InvertedIndexer. Should ensure that the number of tag columns is greater than 0.

pub async fn update(&mut self, batch: &mut Batch) -> Result<()>

Updates index with a batch of rows. Garbage will be cleaned up if failed to update.

pub async fn update_flat(&mut self, batch: &RecordBatch) -> Result<()>

Updates the inverted index with the given flat format RecordBatch.

async fn do_update_flat(&mut self, batch: &RecordBatch) -> Result<()>

pub(crate) async fn finish( &mut self, puffin_writer: &mut <FsPuffinManager<Arc<BoundedStager<RegionIndexId>>, ObjectStorePuffinFileAccessor> as PuffinManager>::Writer, ) -> Result<(usize, u64)>

Finishes index creation and cleans up garbage. Returns the number of rows and bytes written.

pub async fn abort(&mut self) -> Result<()>

Aborts index creation and clean up garbage.

async fn do_update(&mut self, batch: &mut Batch) -> Result<()>

async fn do_finish( &mut self, puffin_writer: &mut <FsPuffinManager<Arc<BoundedStager<RegionIndexId>>, ObjectStorePuffinFileAccessor> as PuffinManager>::Writer, ) -> Result<()>

Data flow of finishing index:

                              (In Memory Buffer)
                                   ┌──────┐
 ┌─────────────┐                   │ PIPE │
 │             │ write index data  │      │
 │ IndexWriter ├──────────────────►│ tx   │
 │             │                   │      │
 └─────────────┘                   │      │
                 ┌─────────────────┤ rx   │
 ┌─────────────┐ │ read as blob    └──────┘
 │             │ │
 │ PuffinWriter├─┤
 │             │ │ copy to file    ┌──────┐
 └─────────────┘ └────────────────►│ File │
                                   └──────┘

async fn do_cleanup(&mut self) -> Result<()>

pub fn column_ids(&self) -> impl Iterator<Item = ColumnId> + '_

pub fn memory_usage(&self) -> usize