Struct FlatLastNonNull

pub struct FlatLastNonNull {
    field_column_start: usize,
    filter_deleted: bool,
    buffer: Option<BatchLastRow>,
    contains_delete: bool,
}

Dedup strategy that keeps the last non-null field for the same key.

Fields

field_column_start: usize

The start index of field columns:

filter_deleted: bool

Filter deleted rows.

buffer: Option<BatchLastRow>

Buffered batch to check whether the next batch have duplicated rows with this batch. Fields in the last row of this batch may be updated by the next batch. The buffered batch should contain no duplication.

contains_delete: bool

Whether the last row range contains a delete operation. If so, we don’t need to update null fields.

Implementations

impl FlatLastNonNull

pub fn new(field_column_start: usize, filter_deleted: bool) -> Self

Creates a new strategy with the given filter_deleted flag.

fn dedup_one_batch( batch: RecordBatch, field_column_start: usize, prev_batch_contains_delete: bool, ) -> Result<(RecordBatch, bool)>

Remove duplications from the batch without considering the previous and next rows. Returns a tuple containing the deduplicated batch and a boolean indicating whether the last range contains deleted rows.

fn dedup_by_partitions( batch: RecordBatch, partitions: &Partitions, field_column_start: usize, op_types: &UInt8Array, first_range_contains_delete: bool, ) -> Result<(RecordBatch, bool)>

Remove depulications for each partition. Returns a tuple containing the deduplicated batch and a boolean indicating whether the last range contains deleted rows.

fn compute_field_indices( ranges: &[Range<usize>], field_array: &ArrayRef, op_types: &UInt8Array, first_range_contains_delete: bool, ) -> UInt64Array

Returns an array of indices of the latest non null value for each input range. If all values in a range are null, the returned index is unspecific. Stops when encountering a delete operation and ignores all subsequent rows.

fn last_range_has_delete(ranges: &[Range<usize>], op_types: &UInt8Array) -> bool

Checks whether the last range contains a delete operation.

Trait Implementations

impl RecordBatchDedupStrategy for FlatLastNonNull

fn push_batch( &mut self, batch: RecordBatch, metrics: &mut DedupMetrics, ) -> Result<Option<RecordBatch>>

Pushes a batch to the dedup strategy. Returns a batch if the strategy ensures there is no duplications based on the input batch.

fn finish(&mut self, metrics: &mut DedupMetrics) -> Result<Option<RecordBatch>>

Finishes the deduplication process and returns any remaining batch.