Struct ParquetReaderBuilder

pub struct ParquetReaderBuilder {
    file_dir: String,
    file_handle: FileHandle,
    object_store: ObjectStore,
    predicate: Option<Predicate>,
    projection: Option<Vec<ColumnId>>,
    cache_strategy: CacheStrategy,
    inverted_index_applier: Option<Arc<InvertedIndexApplier>>,
    bloom_filter_index_applier: Option<Arc<BloomFilterIndexApplier>>,
    fulltext_index_applier: Option<Arc<FulltextIndexApplier>>,
    expected_metadata: Option<RegionMetadataRef>,
}

Parquet SST reader builder.

Fields

file_dir: String

SST directory.

file_handle: FileHandle

object_store: ObjectStore

predicate: Option<Predicate>

Predicate to push down.

projection: Option<Vec<ColumnId>>

Metadata of columns to read.

None reads all columns. Due to schema change, the projection can contain columns not in the parquet file.

cache_strategy: CacheStrategy

Strategy to cache SST data.

inverted_index_applier: Option<Arc<InvertedIndexApplier>>

Index appliers.

bloom_filter_index_applier: Option<Arc<BloomFilterIndexApplier>>

fulltext_index_applier: Option<Arc<FulltextIndexApplier>>

expected_metadata: Option<RegionMetadataRef>

Expected metadata of the region while reading the SST. This is usually the latest metadata of the region. The reader use it get the correct column id of a column by name.

Implementations

impl ParquetReaderBuilder

pub fn new( file_dir: String, file_handle: FileHandle, object_store: ObjectStore, ) -> ParquetReaderBuilder

Returns a new ParquetReaderBuilder to read specific SST.

pub fn predicate(self, predicate: Option<Predicate>) -> ParquetReaderBuilder

Attaches the predicate to the builder.

pub fn projection( self, projection: Option<Vec<ColumnId>>, ) -> ParquetReaderBuilder

Attaches the projection to the builder.

The reader only applies the projection to fields.

pub fn cache(self, cache: CacheStrategy) -> ParquetReaderBuilder

Attaches the cache to the builder.

pub(crate) fn inverted_index_applier( self, index_applier: Option<Arc<InvertedIndexApplier>>, ) -> Self

Attaches the inverted index applier to the builder.

pub(crate) fn bloom_filter_index_applier( self, index_applier: Option<Arc<BloomFilterIndexApplier>>, ) -> Self

Attaches the bloom filter index applier to the builder.

pub(crate) fn fulltext_index_applier( self, index_applier: Option<Arc<FulltextIndexApplier>>, ) -> Self

Attaches the fulltext index applier to the builder.

pub fn expected_metadata( self, expected_metadata: Option<RegionMetadataRef>, ) -> Self

Attaches the expected metadata to the builder.

pub async fn build(&self) -> Result<ParquetReader>

Builds a ParquetReader.

This needs to perform IO operation.

pub(crate) async fn build_reader_input( &self, metrics: &mut ReaderMetrics, ) -> Result<(FileRangeContext, BTreeMap<usize, Option<RowSelection>>)>

Builds a FileRangeContext and collects row groups to read.

This needs to perform IO operation.

fn get_region_metadata( file_path: &str, key_value_meta: Option<&Vec<KeyValue>>, ) -> Result<RegionMetadata>

Decodes region metadata from key value.

async fn read_parquet_metadata( &self, file_path: &str, file_size: u64, ) -> Result<Arc<ParquetMetaData>>

Reads parquet metadata of specific file.

async fn row_groups_to_read( &self, read_format: &ReadFormat, parquet_meta: &ParquetMetaData, metrics: &mut ReaderFilterMetrics, ) -> BTreeMap<usize, Option<RowSelection>>

Computes row groups to read, along with their respective row selections.

async fn prune_row_groups_by_fulltext_index( &self, row_group_size: usize, parquet_meta: &ParquetMetaData, output: &mut BTreeMap<usize, Option<RowSelection>>, metrics: &mut ReaderFilterMetrics, ) -> bool

Prunes row groups by fulltext index. Returns true if the row groups are pruned.

fn group_row_ids( row_ids: BTreeSet<u32>, row_group_size: usize, num_row_groups: usize, ) -> Vec<(usize, Vec<usize>)>

Groups row IDs into row groups, with each group’s row IDs starting from 0.

async fn prune_row_groups_by_inverted_index( &self, row_group_size: usize, parquet_meta: &ParquetMetaData, output: &mut BTreeMap<usize, Option<RowSelection>>, metrics: &mut ReaderFilterMetrics, ) -> bool

Applies index to prune row groups.

TODO(zhongzc): Devise a mechanism to enforce the non-use of indices as an escape route in case of index issues, and it can be used to test the correctness of the index.

fn prune_row_groups_by_minmax( &self, read_format: &ReadFormat, parquet_meta: &ParquetMetaData, output: &mut BTreeMap<usize, Option<RowSelection>>, metrics: &mut ReaderFilterMetrics, ) -> bool

Prunes row groups by min-max index.

async fn prune_row_groups_by_bloom_filter( &self, parquet_meta: &ParquetMetaData, output: &mut BTreeMap<usize, Option<RowSelection>>, metrics: &mut ReaderFilterMetrics, ) -> bool

fn prune_row_groups_by_rows( parquet_meta: &ParquetMetaData, rows_in_row_groups: Vec<(usize, Vec<usize>)>, output: &mut BTreeMap<usize, Option<RowSelection>>, filtered_row_groups: &mut usize, filtered_rows: &mut usize, )

Prunes row groups by rows. The rows_in_row_groups is like a map from row group to a list of row ids to keep.

fn prune_row_groups_by_ranges( parquet_meta: &ParquetMetaData, ranges_in_row_groups: impl Iterator<Item = (usize, impl Iterator<Item = Range<usize>>)>, output: &mut BTreeMap<usize, Option<RowSelection>>, filtered_row_groups: &mut usize, filtered_rows: &mut usize, )

Prunes row groups by ranges. The ranges_in_row_groups is like a map from row group to a list of row ranges to keep.