Module dedup

Utilities to remove duplicate rows from a sorted batch.

Structs

• BatchLastRow 🔒
  State of the last row in a batch for dedup.
• DedupMetrics 🔒
  Metrics for deduplication.
• DedupReader 🔒
  A reader that dedup sorted batches from a source based on the dedup strategy.
• LastFieldsBuilder 🔒
  Buffer to store fields in the last row to merge.
• LastNonNull 🔒
  Dedup strategy that keeps the last non-null field for the same key.
• LastNonNullIter 🔒
  An iterator that dedup rows by LastNonNull strategy. The input iterator must returns sorted batches.
• LastRow 🔒
  Dedup strategy that keeps the row with latest sequence of each key.

Traits

• DedupStrategy 🔒
  Strategy to remove duplicate rows from sorted batches.

Functions

• filter_deleted_from_batch 🔒
  Removes deleted rows from the batch and updates metrics.