Struct BloomFilterCreator

pub struct BloomFilterCreator {
    rows_per_segment: usize,
    accumulated_row_count: usize,
    cur_seg_distinct_elems: HashSet<Bytes>,
    cur_seg_distinct_elems_mem_usage: usize,
    finalized_bloom_filters: FinalizedBloomFilterStorage,
    finalized_row_count: usize,
    global_memory_usage: Arc<AtomicUsize>,
}

BloomFilterCreator is responsible for creating and managing bloom filters for a set of elements. It divides the rows into segments and creates bloom filters for each segment.

Format

The bloom filter creator writes the following format to the writer:

+--------------------+--------------------+-----+----------------------+----------------------+
| Bloom filter 0     | Bloom filter 1     | ... | BloomFilterMeta      | Meta size            |
+--------------------+--------------------+-----+----------------------+----------------------+
|<- bytes (size 0) ->|<- bytes (size 1) ->| ... |<- json (meta size) ->|<- u32 LE (4 bytes) ->|

Fields

rows_per_segment: usize

The number of rows per segment set by the user.

accumulated_row_count: usize

Row count that added to the bloom filter so far.

cur_seg_distinct_elems: HashSet<Bytes>

A set of distinct elements in the current segment.

cur_seg_distinct_elems_mem_usage: usize

The memory usage of the current segment’s distinct elements.

finalized_bloom_filters: FinalizedBloomFilterStorage

Storage for finalized Bloom filters.

finalized_row_count: usize

Row count that finalized so far.

global_memory_usage: Arc<AtomicUsize>

Global memory usage of the bloom filter creator.

Implementations

impl BloomFilterCreator

pub fn new( rows_per_segment: usize, intermediate_provider: Arc<dyn ExternalTempFileProvider>, global_memory_usage: Arc<AtomicUsize>, global_memory_usage_threshold: Option<usize>, ) -> Self

Creates a new BloomFilterCreator with the specified number of rows per segment.

PANICS

rows_per_segment <= 0

pub async fn push_n_row_elems( &mut self, nrows: usize, elems: impl IntoIterator<Item = Bytes>, ) -> Result<()>

Adds multiple rows of elements to the bloom filter. If the number of accumulated rows reaches rows_per_segment, it finalizes the current segment.

pub async fn push_row_elems( &mut self, elems: impl IntoIterator<Item = Bytes>, ) -> Result<()>

Adds a row of elements to the bloom filter. If the number of accumulated rows reaches rows_per_segment, it finalizes the current segment.

pub async fn finish(&mut self, writer: impl AsyncWrite + Unpin) -> Result<()>

Finalizes any remaining segments and writes the bloom filters and metadata to the provided writer.

pub fn memory_usage(&self) -> usize

Returns the memory usage of the creating bloom filter.

async fn finalize_segment(&mut self) -> Result<()>

Trait Implementations

impl Drop for BloomFilterCreator

fn drop(&mut self)

Executes the destructor for this type.