Module loki Copy item path Source JsonStreamItem π LOKI_INIT_SCHEMAS π LokiJsonParser π LokiMiddleItem This is the holder of the loki lines parsed from json or protobuf.
The generic here is either [VrlValue] or Vec .
Depending on the target destination, this can be converted to LokiRawItem or LokiPipeline . LokiPbParser LokiPipeline This is the line item prepared for the pipeline engine. LokiRawItem This is the line item for the Loki raw ingestion.
Weβll persist the line in its whole, set labels into tags,
and structured metadata into a big JSON. PbStreamItem LABEL_KEY π LINES_KEY π LOKI_LINE_COLUMN π LOKI_LINE_COLUMN_NAME π LOKI_PIPELINE_LABEL_PREFIX π LOKI_PIPELINE_METADATA_PREFIX π LOKI_STRUCTURED_METADATA_COLUMN π LOKI_TABLE_NAME π STREAMS_KEY π extract_item π This is the flow of the Loki ingestion.
+ββββ+
| bytes |
+ββββ+
|
+βββββββββββ+βββββββββββ+
| | |
| JSON content type | PB content type |
+βββββββββββ+βββββββββββ+
| | |
| JsonStreamItem | PbStreamItem |
| stream: serde_json | stream: adapter |
+βββββββββββ+βββββββββββ+
| | |
| MiddleItem<serde_json> | MiddleItem |
+βββββββββββ+βββββββββββ+
\ /
\ /
\ /
+βββββββββββ+
| MiddleItem |
+βββββββββββ+
|
+ββββββββ+ββββββββ+
| |
+ββββββ+ +βββββββ+
| LokiRawItem | | LokiPipelineItem |
+ββββββ+ +βββββββ+
| |
+ββββββ+ +βββββββ+
| Loki ingest | | run_pipeline |
+ββββββ+ +βββββββ+ init_row π loki_ingest parse_loki_labels since weβre hand-parsing the labels, if any error is encountered, weβll just skip the label
note: pub here for bench usage
ref: process_labels π prost_ts_to_nano π