Module format

Format to store in parquet.

We store three internal columns in parquet:

• __primary_key, the primary key of the row (tags). Type: dictionary(uint32, binary)
• __sequence, the sequence number of a row. Type: uint64
• __op_type, the op type of the row. Type: uint8

The schema of a parquet file is:

field 0, field 1, ..., field N, time index, primary key, sequence, op type

We stores fields in the same order as RegionMetadata::field_columns().

Structs

• ReadFormat
  Helper for reading the SST format.
• WriteFormat 🔒
  Helper for writing the SST format.

Constants

• FIXED_POS_COLUMN_NUM 🔒
  Number of columns that have fixed positions.

Functions

• new_primary_key_array 🔒
  Creates a new array for specific primary_key.
• parquet_row_group_time_range 🔒
  Gets the min/max time index of the row group from the parquet meta. It assumes the parquet is created by the mito engine.
• primary_key_offsets 🔒
  Compute offsets of different primary keys in the array.

Type Aliases

• PrimaryKeyArray 🔒
  Arrow array type for the primary key dictionary.