Struct BatchBuilder

pub struct BatchBuilder {
    schema: SchemaRef,
    batches: Vec<(usize, RecordBatch)>,
    cursors: Vec<BatchCursor>,
    indices: Vec<(usize, usize)>,
}

Provides an API to incrementally build a [RecordBatch] from partitioned [RecordBatch]

Fields

schema: SchemaRef

The schema of the RecordBatches yielded by this stream

batches: Vec<(usize, RecordBatch)>

Maintain a list of [RecordBatch] and their corresponding stream

cursors: Vec<BatchCursor>

The current BatchCursor for each stream

indices: Vec<(usize, usize)>

The accumulated stream indexes from which to pull rows Consists of a tuple of (batch_idx, row_idx)

Implementations

impl BatchBuilder

pub fn new(schema: SchemaRef, stream_count: usize, batch_size: usize) -> Self

Create a new BatchBuilder with the provided stream_count and batch_size

pub fn push_batch(&mut self, stream_idx: usize, batch: RecordBatch)

Append a new batch in stream_idx

pub fn push_row(&mut self, stream_idx: usize)

Append the next row from stream_idx

pub fn len(&self) -> usize

Returns the number of in-progress rows in this BatchBuilder

pub fn is_empty(&self) -> bool

Returns true if this BatchBuilder contains no in-progress rows

pub fn schema(&self) -> &SchemaRef

Returns the schema of this BatchBuilder

pub fn build_record_batch(&mut self) -> Result<Option<RecordBatch>>

Drains the in_progress row indexes, and builds a new RecordBatch from them

Will then drop any batches for which all rows have been yielded to the output

Returns None if no pending rows

pub fn take_remaining_rows( &mut self, stream_idx: usize, next: Option<RecordBatch>, ) -> RecordBatch

Slice and take remaining rows from the last batch of stream_idx and push the next batch if available.

fn retain_batches(&mut self)

Trait Implementations

impl Debug for BatchBuilder

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.