Struct ExternalSorter

pub struct ExternalSorter {
    index_name: String,
    temp_file_provider: Arc<dyn ExternalTempFileProvider>,
    segment_null_bitmap: Bitmap,
    values_buffer: BTreeMap<Bytes, (Bitmap, usize)>,
    total_row_count: usize,
    segment_row_count: NonZeroUsize,
    current_memory_usage: usize,
    current_memory_usage_threshold: Option<usize>,
    global_memory_usage: Arc<AtomicUsize>,
    global_memory_usage_sort_limit: Option<usize>,
}

ExternalSorter manages the sorting of data using both in-memory structures and external files. It dumps data to external files when the in-memory buffer crosses a certain memory threshold.

Fields

index_name: String

The index name associated with the sorting operation

temp_file_provider: Arc<dyn ExternalTempFileProvider>

Manages creation and access to external temporary files

segment_null_bitmap: Bitmap

Bitmap indicating which segments have null values

values_buffer: BTreeMap<Bytes, (Bitmap, usize)>

In-memory buffer to hold values and their corresponding bitmaps until memory threshold is exceeded

total_row_count: usize

Count of all rows ingested so far

segment_row_count: NonZeroUsize

The number of rows per group for bitmap indexing which determines how rows are batched for indexing. It is used to determine which segment a row belongs to.

current_memory_usage: usize

Tracks memory usage of the buffer

current_memory_usage_threshold: Option<usize>

The threshold of current memory usage below which the buffer is not dumped, even if the global memory usage exceeds global_memory_usage_sort_limit. This allows for smaller buffers to remain in memory, providing a buffer against unnecessary dumps to external files, which can be costly in terms of performance. None indicates that only the global memory usage threshold is considered for dumping the buffer.

global_memory_usage: Arc<AtomicUsize>

Tracks the global memory usage of all sorters

global_memory_usage_sort_limit: Option<usize>

The memory usage limit that, when exceeded by the global memory consumption of all sorters, necessitates a reassessment of buffer retention. Surpassing this limit signals that there is a high overall memory pressure, potentially requiring buffer dumping to external storage for memory relief. None value indicates that no specific global memory usage threshold is established for triggering buffer dumps.

Implementations

impl ExternalSorter

pub fn new( index_name: String, temp_file_provider: Arc<dyn ExternalTempFileProvider>, segment_row_count: NonZeroUsize, current_memory_usage_threshold: Option<usize>, global_memory_usage: Arc<AtomicUsize>, global_memory_usage_sort_limit: Option<usize>, ) -> Self

Constructs a new ExternalSorter

pub fn factory( temp_file_provider: Arc<dyn ExternalTempFileProvider>, current_memory_usage_threshold: Option<usize>, global_memory_usage: Arc<AtomicUsize>, global_memory_usage_sort_limit: Option<usize>, ) -> SorterFactory

Generates a factory function that creates new ExternalSorter instances

fn push_not_null( &mut self, value: BytesRef<'_>, segment_index_range: RangeInclusive<usize>, ) -> usize

Pushes the non-null values to the values buffer and sets the bits within the specified range in the given bitmap to true. Returns the memory usage difference of the buffer after the operation.

async fn may_dump_buffer(&mut self, memory_diff: usize) -> Result<()>

Checks if the in-memory buffer exceeds the threshold and offloads it to external storage if necessary

fn segment_index_range(&self, n: usize) -> RangeInclusive<usize>

Determines the segment index range for the row index range [row_begin, row_begin + n - 1]

fn segment_index(&self, row_index: usize) -> usize

Determines the segment index for the given row index

Trait Implementations

impl Sorter for ExternalSorter

fn push_n<'life0, 'life1, 'async_trait>( &'life0 mut self, value: Option<BytesRef<'life1>>, n: usize, ) -> Pin<Box<dyn Future<Output = Result<()>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Pushes n identical values into the sorter, adding them to the in-memory buffer and dumping the buffer to an external file if necessary

fn output<'life0, 'async_trait>( &'life0 mut self, ) -> Pin<Box<dyn Future<Output = Result<SortOutput>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Finalizes the sorting operation, merging data from both in-memory buffer and external files into a sorted stream

fn push<'life0, 'life1, 'async_trait>( &'life0 mut self, value: Option<BytesRef<'life1>>, ) -> Pin<Box<dyn Future<Output = Result<()>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Inputs a non-null or null value into the sorter. Should be equivalent to calling push_n with n = 1