Module dedup 

Utilities to remove duplicate rows from a sorted batch.

Structs

DedupMetrics

Metrics for deduplication.

LastFieldsBuilder 🔒

Buffer to store fields in the last row to merge.

LastNonNull 🔒

Dedup strategy that keeps the last non-null field for the same key.

LastNonNullIter 🔒

An iterator that dedup rows by LastNonNull strategy. The input iterator must returns sorted batches.

Traits

DedupMetricsReport

Trait for reporting dedup metrics.

DedupStrategy

Strategy to remove duplicate rows from sorted batches.

Functions

filter_deleted_from_batch 🔒

Removes deleted rows from the batch and updates metrics.