Module format 

Format to store in parquet.

We store three internal columns in parquet:

• __primary_key, the primary key of the row (tags). Type: dictionary(uint32, binary)
• __sequence, the sequence number of a row. Type: uint64
• __op_type, the op type of the row. Type: uint8

The schema of a parquet file is:

field 0, field 1, ..., field N, time index, primary key, sequence, op type

We stores fields in the same order as RegionMetadata::field_columns().

Structs

FormatProjection 🔒

Helper to compute the projection for the SST.

PrimaryKeyReadFormat

Helper for reading the SST format.

PrimaryKeyWriteFormat 🔒

Helper for writing the SST format with primary key.

Enums

StatValues

Values of column statistics of the SST.

Constants

FIXED_POS_COLUMN_NUM 🔒

Number of columns that have fixed positions.

INTERNAL_COLUMN_NUM 🔒

Number of internal columns.

Functions

column_null_counts 🔒

Returns null counts of specific columns. The column should not be encoded as a part of a primary key.

column_values 🔒

Returns min/max values of specific columns. Returns None if the column does not have statistics. The column should not be encoded as a part of a primary key.

need_override_sequence 🔒

Checks if sequence override is needed based on all row groups’ statistics. Returns true if ALL row groups have sequence min-max values of 0.

parquet_row_group_time_range 🔒

Gets the min/max time index of the row group from the parquet meta. It assumes the parquet is created by the mito engine.

primary_key_offsets 🔒

Compute offsets of different primary keys in the array.

Type Aliases

PrimaryKeyArray 🔒

Arrow array type for the primary key dictionary.

PrimaryKeyArrayBuilder 🔒

Builder type for primary key dictionary array.