Struct FlatReadFormat

pub struct FlatReadFormat {
    metadata: RegionMetadataRef,
    arrow_schema: SchemaRef,
    format_projection: FormatProjection,
    column_id_to_sst_index: HashMap<ColumnId, usize>,
    override_sequence: Option<SequenceNumber>,
    convert_format: Option<FlatConvertFormat>,
}

Helper for reading the flat SST format with projection.

It only supports flat format that stores primary keys additionally.

Fields

metadata: RegionMetadataRef

The metadata stored in the SST.

arrow_schema: SchemaRef

SST file schema.

format_projection: FormatProjection

Projection computed for the format.

column_id_to_sst_index: HashMap<ColumnId, usize>

Column id to index in SST.

override_sequence: Option<SequenceNumber>

Sequence number to override the sequence read from the SST.

convert_format: Option<FlatConvertFormat>

Optional format converter for handling flat format conversion.

Implementations

impl FlatReadFormat

pub fn new( metadata: RegionMetadataRef, column_ids: impl Iterator<Item = ColumnId>, convert_to_flat: bool, ) -> FlatReadFormat

Creates a helper with existing metadata and column_ids to read.

pub(crate) fn set_override_sequence(&mut self, sequence: Option<SequenceNumber>)

Sets the sequence number to override.

pub fn projected_index_by_id(&self, column_id: ColumnId) -> Option<usize>

Index of a column in the projected batch by its column id.

pub fn min_values( &self, row_groups: &[impl Borrow<RowGroupMetaData>], column_id: ColumnId, ) -> StatValues

Returns min values of specific column in row groups.

pub fn max_values( &self, row_groups: &[impl Borrow<RowGroupMetaData>], column_id: ColumnId, ) -> StatValues

Returns max values of specific column in row groups.

pub fn null_counts( &self, row_groups: &[impl Borrow<RowGroupMetaData>], column_id: ColumnId, ) -> StatValues

Returns null counts of specific column in row groups.

pub(crate) fn arrow_schema(&self) -> &SchemaRef

Gets the arrow schema of the SST file.

This schema is computed from the region metadata but should be the same as the arrow schema decoded from the file metadata.

pub(crate) fn metadata(&self) -> &RegionMetadataRef

Gets the metadata of the SST.

pub(crate) fn projection_indices(&self) -> &[usize]

Gets sorted projection indices to read.

pub(crate) fn format_projection(&self) -> &FormatProjection

Gets the projection.

pub(crate) fn new_override_sequence_array( &self, length: usize, ) -> Option<ArrayRef>

Creates a sequence array to override.

pub(crate) fn convert_batch( &self, record_batch: RecordBatch, override_sequence_array: Option<&ArrayRef>, ) -> Result<RecordBatch>

Convert a record batch to apply flat format conversion and override sequence array.

Returns a new RecordBatch with flat format conversion applied first (if enabled), then the sequence column replaced by the override sequence array.

pub(crate) fn need_convert_to_flat( file_path: &str, num_columns: usize, metadata: &RegionMetadata, ) -> Result<bool>

Checks whether the batch from the parquet file needs to be converted to match the flat format.

• file_path is the path to the parquet file, for error message.
• num_columns is the number of columns in the parquet file.
• metadata is the region metadata (always assumes flat format).

fn get_stat_values( &self, row_groups: &[impl Borrow<RowGroupMetaData>], column_id: ColumnId, is_min: bool, ) -> StatValues