Struct ParallelArrowWriter 

pub struct ParallelArrowWriter<W> {

    file_writer: SerializedFileWriter<Vec<u8>>,
    factory: Arc<ArrowRowGroupWriterFactory>,
    arrow_schema: SchemaRef,
    leaf_fields: Vec<usize>,
    field_costs: Vec<Duration>,
    async_writer: W,
    max_row_group_rows: usize,
    parallelism: usize,
    runtime: EncodeRuntime,
    buffered: Vec<RecordBatch>,
    buffered_rows: usize,
    encoding: Vec<JoinHandle<Result<(Vec<(usize, ArrowColumnChunk)>, Vec<(usize, Duration)>)>>>,
    num_row_groups: usize,
}

Writes Parquet like [parquet::arrow::AsyncArrowWriter], but encodes the columns of each row group on up to parallelism blocking tasks.

A row group encodes in the background while the next one is buffered, so at most two row groups of input are held in memory. Every column writer sees the same input as with [ArrowWriter], so the output is byte-for-byte identical. The bytes of a row group reach the async writer when the next row group starts or on Self::flush, one row group later than with AsyncArrowWriter.

Fields

file_writer: SerializedFileWriter<Vec<u8>>

Assembles row groups and the footer into an in-memory buffer.

factory: Arc<ArrowRowGroupWriterFactory>

Creates the column writers of each row group.

arrow_schema: SchemaRef

leaf_fields: Vec<usize>

Top-level field of each leaf column.

field_costs: Vec<Duration>

Encode time of each field in the last row group, used to balance tasks.

async_writer: W

max_row_group_rows: usize

parallelism: usize

runtime: EncodeRuntime

buffered: Vec<RecordBatch>

Slices of input buffered for the next row group.

buffered_rows: usize

encoding: Vec<JoinHandle<Result<(Vec<(usize, ArrowColumnChunk)>, Vec<(usize, Duration)>)>>>

Tasks encoding the previous row group.

num_row_groups: usize

Implementations

impl<W: AsyncFileWriter> ParallelArrowWriter<W>

pub fn supports(props: &WriterProperties) -> bool

Returns whether the writer can reproduce [ArrowWriter]’s row groups for props, which holds only when row groups are limited by row count alone.

pub fn try_new( async_writer: W, arrow_schema: SchemaRef, props: WriterProperties, parallelism: usize, runtime: EncodeRuntime, ) -> Result<Self>

Creates a writer. props must be supported.

pub async fn write(&mut self, batch: &RecordBatch) -> Result<()>

Buffers batch, starting to encode each row group once it is full.

pub async fn flush(&mut self) -> Result<()>

Writes all buffered rows as a row group, like [ArrowWriter::flush].

pub async fn close(self) -> Result<ParquetMetaData>

Writes the remaining rows and the footer, then completes the async writer.

pub(crate) async fn abort(self)

Discards buffered input and waits for encoders without writing a failed SST.

async fn encode_buffered(&mut self) -> Result<()>

Starts encoding the buffered rows once the previous row group is written.

async fn finish_encoding(&mut self) -> Result<()>

Waits for the row group being encoded and appends it to the file.

async fn write_buffered_bytes(&mut self) -> Result<()>

Moves the bytes assembled so far to the async writer.