Enum ReadFormat 

pub enum ReadFormat {
    PrimaryKey(PrimaryKeyReadFormat),
    Flat(FlatReadFormat),
}

Helper to read parquet formats.

Variants

PrimaryKey(PrimaryKeyReadFormat)

The parquet is in the old primary key format.

Flat(FlatReadFormat)

The parquet is in the new flat format.

Implementations

impl ReadFormat

pub fn new_primary_key( metadata: RegionMetadataRef, column_ids: impl Iterator<Item = ColumnId>, ) -> Self

Creates a helper to read the primary key format.

pub fn new_flat( metadata: RegionMetadataRef, column_ids: impl Iterator<Item = ColumnId>, num_columns: Option<usize>, file_path: &str, skip_auto_convert: bool, ) -> Result<Self>

Creates a helper to read the flat format.

pub fn new( region_metadata: RegionMetadataRef, projection: Option<&[ColumnId]>, flat_format: bool, num_columns: Option<usize>, file_path: &str, skip_auto_convert: bool, ) -> Result<ReadFormat>

Creates a new read format.

pub(crate) fn as_primary_key(&self) -> Option<&PrimaryKeyReadFormat>

pub(crate) fn as_flat(&self) -> Option<&FlatReadFormat>

pub(crate) fn arrow_schema(&self) -> &SchemaRef

Gets the arrow schema of the SST file.

This schema is computed from the region metadata but should be the same as the arrow schema decoded from the file metadata.

pub(crate) fn metadata(&self) -> &RegionMetadataRef

Gets the metadata of the SST.

pub(crate) fn projection_indices(&self) -> &[usize]

Gets sorted projection indices to read.

pub fn min_values( &self, row_groups: &[impl Borrow<RowGroupMetaData>], column_id: ColumnId, ) -> StatValues

Returns min values of specific column in row groups.

pub fn max_values( &self, row_groups: &[impl Borrow<RowGroupMetaData>], column_id: ColumnId, ) -> StatValues

Returns max values of specific column in row groups.

pub fn null_counts( &self, row_groups: &[impl Borrow<RowGroupMetaData>], column_id: ColumnId, ) -> StatValues

Returns null counts of specific column in row groups.

pub(crate) fn column_values( row_groups: &[impl Borrow<RowGroupMetaData>], column: &ColumnMetadata, column_index: usize, is_min: bool, ) -> Option<ArrayRef>

Returns min/max values of specific columns. Returns None if the column does not have statistics. The column should not be encoded as a part of a primary key.

pub(crate) fn column_null_counts( row_groups: &[impl Borrow<RowGroupMetaData>], column_index: usize, ) -> Option<ArrayRef>

Returns null counts of specific columns. The column should not be encoded as a part of a primary key.

pub(crate) fn set_override_sequence(&mut self, sequence: Option<SequenceNumber>)

Sets the sequence number to override.

pub(crate) fn new_override_sequence_array( &self, length: usize, ) -> Option<ArrayRef>

Creates a sequence array to override.