Skip to main content

sst_info_from_file_meta

Function sst_info_from_file_meta 

Source
fn sst_info_from_file_meta(meta: &FileMeta) -> SstInfo
Expand description

Builds an SstInfo for the remote-compaction path by copying the scalar fields that FileMeta also carries.

Remote compaction runs off-process and ships MergeOutput over the wire; MergeOutput::sst_infos is #[serde(skip)] because the parquet footer (ParquetMetaData) is not serde-serializable. To avoid feeding the hook default (zero) values for real SSTs, this rebuilds the seven fields FileMeta and SstInfo share: file_id, time_range, file_size, max_row_group_uncompressed_size, num_rows, num_row_groups, num_series.

Two fields stay default and cannot be recovered on the datanode:

  • file_metadata — the parquet footer, whose column min/max/null-count statistics are required by hooks building richer artifacts (e.g. an Iceberg manifest). That data is produced by the compactor’s writer and exists only in the freshly-written SST on object storage.
  • index_metadataFileMeta tracks indexes via available_indexes / indexes, not as an [IndexOutput].

A hook that needs column stats must fetch the footer from object storage (the CompactionRegion’s access_layer reaches the store), or the footer must be shipped over the wire (revisit the #[serde(skip)] on sst_infos). num_rows may read 0 for legacy FileMetas where the count is unknown.