Module dedup

Utilities to remove duplicate rows from a sorted batch.

Structs

BatchLastRow 🔒

State of the last row in a batch for dedup.

DedupMetrics 🔒

Metrics for deduplication.

DedupReader 🔒

A reader that dedup sorted batches from a source based on the dedup strategy.

LastFieldsBuilder 🔒

Buffer to store fields in the last row to merge.

LastNonNull 🔒

Dedup strategy that keeps the last non-null field for the same key.

LastNonNullIter 🔒

An iterator that dedup rows by LastNonNull strategy. The input iterator must returns sorted batches.

LastRow 🔒

Dedup strategy that keeps the row with latest sequence of each key.

Traits

DedupStrategy 🔒

Strategy to remove duplicate rows from sorted batches.

Functions

filter_deleted_from_batch 🔒

Removes deleted rows from the batch and updates metrics.