Struct FlatProjectionMapper

pub struct FlatProjectionMapper {
    metadata: RegionMetadataRef,
    output_schema: SchemaRef,
    column_ids: Vec<ColumnId>,
    batch_schema: Vec<(ColumnId, ConcreteDataType)>,
    is_empty_projection: bool,
    batch_indices: Vec<usize>,
    input_arrow_schema: SchemaRef,
}

Handles projection and converts batches in flat format with correct schema.

This mapper support duplicate and unsorted projection indices. The output schema is determined by the projection indices.

Fields

metadata: RegionMetadataRef

Metadata of the region.

output_schema: SchemaRef

Schema for converted RecordBatch to return.

column_ids: Vec<ColumnId>

Ids of columns to project. It keeps ids in the same order as the projection indices to build the mapper. The mapper won’t deduplicate the column ids.

batch_schema: Vec<(ColumnId, ConcreteDataType)>

Ids and DataTypes of columns of the expected batch. We can use this to check if the batch is compatible with the expected schema.

is_empty_projection: bool

true If the original projection is empty.

batch_indices: Vec<usize>

The index in flat format RecordBatch for each column in the output RecordBatch.

input_arrow_schema: SchemaRef

Precomputed Arrow schema for input batches.

Implementations

impl FlatProjectionMapper

pub fn new( metadata: &RegionMetadataRef, projection: impl Iterator<Item = usize>, ) -> Result<Self>

Returns a new mapper with projection. If projection is empty, it outputs RecordBatch without any column but only a row count. SELECT COUNT(*) FROM table is an example that uses an empty projection. DataFusion accepts empty RecordBatch and only use its row count in this query.

pub fn all(metadata: &RegionMetadataRef) -> Result<Self>

Returns a new mapper without projection.

pub(crate) fn metadata(&self) -> &RegionMetadataRef

Returns the metadata that created the mapper.

pub(crate) fn column_ids(&self) -> &[ColumnId]

Returns ids of projected columns that we need to read from memtables and SSTs.

pub(crate) fn field_column_start(&self) -> usize

Returns the field column start index in output batch.

pub(crate) fn batch_schema(&self) -> &[(ColumnId, ConcreteDataType)]

Returns ids of columns of the batch that the mapper expects to convert.

pub(crate) fn input_arrow_schema(&self) -> SchemaRef

pub(crate) fn output_schema(&self) -> SchemaRef

Returns the schema of converted RecordBatch. This is the schema that the stream will output. This schema may contain less columns than FlatProjectionMapper::column_ids().

pub(crate) fn empty_record_batch(&self) -> RecordBatch

Returns an empty RecordBatch.

pub(crate) fn convert(&self, batch: &RecordBatch) -> Result<RecordBatch>

Converts a flat format RecordBatch to a normal RecordBatch.

The batch must match the projection using to build the mapper.