Module parallel_writer 

Arrow to Parquet writer that encodes the columns of a row group in parallel.

Structs

ParallelArrowWriter

Writes Parquet like [parquet::arrow::AsyncArrowWriter], but encodes the columns of each row group on up to parallelism blocking tasks.

Enums

EncodeRuntime

Runtime whose blocking pool encodes the columns of a writer.

Functions

abort_encoding_tasks 🔒

Aborts queued blocking work and joins running work before its budget is released.

assign_fields 🔒

Splits fields into at most parallelism tasks, giving the most expensive field to the least loaded task first. Fields cost nothing before any row group is measured, which spreads them evenly.

encode_fields 🔒

Encodes fields of all batches and closes their column writers.

Type Aliases

EncodedFields 🔒

Output of one encode task: column chunks by leaf index, and the time spent on each field.

FieldWriters 🔒

Column writers of one top-level field: its index and its leaf writers.