convert_bulk_part

Function convert_bulk_part 

Source
pub fn convert_bulk_part(
    part: BulkPart,
    region_metadata: &RegionMetadataRef,
    primary_key_codec: Arc<dyn PrimaryKeyCodec>,
    schema: SchemaRef,
    store_primary_key_columns: bool,
) -> Result<Option<BulkPart>>
Expand description

Converts a BulkPart that is unordered and without encoded primary keys into a BulkPart with the same format as produced by BulkPartConverter.

This function takes a BulkPart where:

  • For dense encoding: Primary key columns may be stored as individual columns
  • For sparse encoding: The __primary_key column should already be present with encoded keys
  • The batch may not be sorted

And produces a BulkPart where:

  • Primary key columns are optionally stored (depending on store_primary_key_columns and encoding)
  • An encoded __primary_key dictionary column is present
  • The batch is sorted by (primary_key, timestamp, sequence desc)

§Arguments

  • part - The input BulkPart to convert
  • region_metadata - Region metadata containing schema information
  • primary_key_codec - Codec for encoding primary keys
  • schema - Target schema for the output batch
  • store_primary_key_columns - If true and encoding is not sparse, stores individual primary key columns

§Returns

Returns None if the input part has no rows, otherwise returns a new BulkPart with encoded primary keys and sorted data.