Struct SendableRecordBatchMapper

pub struct SendableRecordBatchMapper {
    pub(crate) inner: SendableRecordBatchStream,
    pub(crate) mapper: fn(RecordBatch, &SchemaRef, &SchemaRef) -> Result<RecordBatch>,
    pub(crate) schema: SchemaRef,
    pub(crate) apply_mapper: bool,
}

A wrapper that maps a RecordBatchStream to a new RecordBatchStream by applying a function to each RecordBatch.

The mapper function is applied to each RecordBatch in the stream. The schema of the new RecordBatchStream is the same as the schema of the inner RecordBatchStream after applying the schema mapper function. The output ordering of the new RecordBatchStream is the same as the output ordering of the inner RecordBatchStream. The metrics of the new RecordBatchStream is the same as the metrics of the inner RecordBatchStream if it is not None.

Fields

inner: SendableRecordBatchStream

mapper: fn(RecordBatch, &SchemaRef, &SchemaRef) -> Result<RecordBatch>

The mapper function is applied to each RecordBatch in the stream. The original schema and the mapped schema are passed to the mapper function.

schema: SchemaRef

The schema of the new RecordBatchStream is the same as the schema of the inner RecordBatchStream after applying the schema mapper function.

apply_mapper: bool

Whether the mapper function is applied to each RecordBatch in the stream.

Implementations

impl SendableRecordBatchMapper

pub fn new( inner: SendableRecordBatchStream, mapper: fn(RecordBatch, &SchemaRef, &SchemaRef) -> Result<RecordBatch>, schema_mapper: fn(SchemaRef) -> (SchemaRef, bool), ) -> Self

Creates a new SendableRecordBatchMapper with the given inner RecordBatchStream, mapper function, and schema mapper function.

Trait Implementations

impl RecordBatchStream for SendableRecordBatchMapper

fn name(&self) -> &str

fn schema(&self) -> SchemaRef

fn output_ordering(&self) -> Option<&[OrderOption]>

fn metrics(&self) -> Option<RecordBatchMetrics>

impl Stream for SendableRecordBatchMapper

type Item = Result<RecordBatch, Error>

Values yielded by the stream.

fn poll_next( self: Pin<&mut Self>, cx: &mut Context<'_>, ) -> Poll<Option<Self::Item>>

Attempt to pull out the next value of this stream, registering the current task for wakeup if the value is not yet available, and returning None if the stream is exhausted.

fn size_hint(&self) -> (usize, Option<usize>)

Returns the bounds on the remaining length of the stream.