Skip to main content

Module native_compile

Module native_compile 

Source
Expand description

Compiles a PromQL AST into a [NativeTree].

This is the front half of the native evaluation model: it reads the parsed statement and writes down what it means as bound nodes. The native planner uses this compiler to build a tree for standalone execution. Its accepted expressions are described by the recursive rule below:

<expression>                                     one of, recursively:
  <selector>                                     e.g. x, x offset -5m
  <range function>(<matrix>)                     e.g. rate(x[5m]), rate(x[5m:1m] @ 1234)
  quantile_over_time(<constant>, <matrix>)       e.g. quantile_over_time(0.9, x[5m])
  predict_linear(<matrix>, <constant>)           e.g. predict_linear(x[5m], 300)
  double_exponential_smoothing(<matrix>, <constant>, <constant>)
                                                 e.g. double_exponential_smoothing(x[5m], .5, .1)
  <math function>(<expression>[, <constant>]*)   e.g. abs(rate(x[5m])), clamp_min(x, 0)
  <round or clamp>(<expression>, <scalar>+)      e.g. round(x, time() / 10), clamp(x, 0, time())
  <histogram function>(<expression>)             e.g. histogram_count(x)
  <sort>(<expression>[, <string>]*)              e.g. sort_desc(x), sort_by_label(x, "job")
  absent(<expression>)                           e.g. absent(x{job="a"})
  absent_over_time(<matrix>)                     e.g. absent_over_time(x{job="a"}[5m])
  <histogram>(<constant>+, <expression>)         e.g. histogram_quantile(0.9, rate(x[5m]))
  <label function>(<expression>, <string>*)       e.g. label_replace(x, "a", "$1", "b", "(.*)")
  <expression> <operator> <scalar>               e.g. x * 100, x > bool time()
  <scalar> <operator> <expression>               e.g. 1 / x, time() - timestamp(x)
  <scalar> <operator> <scalar>                   e.g. time() - 60, time() > bool 1
  time(), pi(), vector(<scalar>)                 e.g. vector(time()), pi() * x
  <date function>()                              e.g. hour(), which is `hour(vector(time()))`
  -<expression>                                  e.g. -x
  <expression> <set operator> <expression>       e.g. x and on(job) y, x unless y
  info(<expression>[, {<matchers>}])             e.g. info(rate(x[5m]), {version=~".+"})
  <aggregator> [(by|without)(<labels>)] (<expression>)
                                                 e.g. avg by(job)(rate(x[5m]) * 100)
  <aggregator> (<expression>) [(by|without)(<labels>)]
                                                 e.g. count (x) by(job)
  quantile(<constant>, <expression>) [(by|without)(<labels>)]
                                                 e.g. quantile(0.9, x) by(job)
  <selector aggregator>(<constant>, <expression>) [(by|without)(<labels>)]
                                                 e.g. topk(3, x) by(job), limit_ratio(0.5, x)
  count_values(<string>, <expression>) [(by|without)(<labels>)]
                                                 e.g. count_values("v", x)

<matrix>    <selector>[<range>]<modifiers>       a range selector: raw samples
          | <expression>[<range>:<step>]<modifiers>
                                                 a subquery: <expression> evaluated on a grid;
                                                 the step may be omitted as `[<range>:]`
<scalar>    <constant> | time() | <scalar> <operator> <scalar>
<modifiers>                                      nothing, `offset <duration>`, `@ <anchor>`, or
                                                 both, in either order — e.g. `x[5m] @ 1234
                                                 offset 1m`

where <set operator> is and, or or unless, optionally with on(...) or ignoring(...), <aggregator> is sum, avg, count, min, max, group, stddev, stdvar or quantile, <selector aggregator> is topk, bottomk, limitk or limit_ratio, the function names are range_signature’s and math_signature’s, and which side of the matrix a range function’s constants go on is RangeSignature’s: before it for quantile_over_time, after it for predict_linear and double_exponential_smoothing. A math function’s constants always follow its vector.

Parentheses are transparent wherever they appear — rate((x[5m])) is rate(x[5m]) — because they say how a statement was typed rather than what it asks for.

A subquery’s <expression> is any shape above, so a subquery nests inside a subquery on a grid of its own, as in rate(rate(x[5m:1m])[10m:30s]); a <step> left out means Prometheus’s default evaluation interval, one minute. Its offset and its @ anchor are its own, not the inner expression’s, so rate(x[5m:1m] @ end() offset 1m) reads one fixed interval. A <selector> outside a <matrix> takes an offset and an optional @ anchor; x @ 100 selects at one fixed instant across the evaluation grid, as does rate(x[5m] @ 100) for its window.

Those shapes become Input (raw samples) → Instant (the last sample within the statement’s lookback), Input (raw samples) → Window (the range function over the statement’s grid), <expression> → Window whose input is evaluated (the subquery, whose input grid is its own), <expression> → Map (a pointwise math, native histogram or label function over what its argument evaluates to) and, over any of those, Binary (an arithmetic or comparison operation between vectors or with a constant on one side), Scalar (that constant), Set (and, or or unless between two vectors), Info (the expression and an Instant of the info metric reporting sample times), Aggregate (one of the nine aggregators that fold a group’s values into one value per grid point, optionally grouped) and Vector (an operation over the whole vector at each point: a sort, absent, a classic histogram function, a selecting aggregator or count_values). Everything else is a [DataFusionError::NotImplemented] that names what was found and quotes the shapes above, because rejecting a query the compiler does not understand is the only honest alternative to compiling it into something else.

The compiler binds a subquery’s child grid and RangeWindow independently of runtime execution: compilation describes the anchored interval, while runtime support determines how that bound plan is consumed.

Two decisions keep the recursion from growing a branch per query. A constant argument — a quantile, a rounding step, a clamp bound — is folded out of the statement by constant_value, so quantile_over_time(1 - 0.1, x[5m]) binds the number it means rather than the expression it was written as. A math function is bound by math_op from a list of names, so clamp_min(abs(rate(x[5m])), 0) is two maps over one window because the arguments nest, not because the compiler knows the query. Arithmetic inside a fold is the kernel operation [BinaryOp::kind] names, so the number this compiler binds and the number the kernel computes cannot disagree about %, ^ or atan2.

Two more are about what a node means rather than about which node to build. A binary operation keeps its operands in the order they were written, because most operators are not commutative: 1 / x is not x / 1, and which side the constant is on is the only thing that tells them apart. return_bool is carried as its own field, so x > 0 (which filters) and x > bool 0 (which reports zero or one) are one node with one field changed. A negation then needs no node of its own: -x binds as the multiplication by -1 that it is, which the multiplication kernel already computes.

Vector matching is bound with each selector’s own input, including the labels and cardinality written on a vector–vector operation. A scalar has no input of its own: it is bound next to the vector that gives it something to broadcast over, while a statement whose value is only a constant is bound by compile_statement as that constant on the statement’s grid.

An aggregate is bound as [AggregateStage::Complete], which is a claim about the whole execution rather than about the node: it says that every contribution to every group is in this execution’s input. The caller has to keep that claim. One tree per partition does not: handing every partition the same Complete tree sums each partition’s part and reports it as the group’s whole, so a partitioned caller has to feed whole groups to one execution or split the aggregation into Partial and Final instead. What a Complete node guarantees here is only the arithmetic — that the values it folded are all the values it was given.

The statement’s grid, the selector’s lookback, window, offset and anchor are the only numbers derived here; the legality of the tree is not checked here. [NativeTree::try_new] is the single authority on that, so a node added later cannot be legal in one place and illegal in another. What is refused before a tree exists is the scan’s shape: no shape below can consume chunks of different series that interleave, so an Interleaved ordering is refused once, at the entry, and [NativeNode::validate] refuses the same ordering for a tree built some other way.

Structs§

CompiledInput
One compiled selector occurrence and the scan layout bound for it.
CompiledTree
Context 🔒
Shared statement, per-selector binding callback, occurrence records, ordering and current grid.
InfoInput 🔒
The info series info(v[, {selector}]) reads, as a selector, and the selector its kernel combines by.
SelectorRequirement
SubqueryGrid 🔒
The grid a subquery’s inner expression is evaluated on, and the window that reads it.

Enums§

MathSignature 🔒
What a call to a math function has to be: the constants it reads after its vector, and how to bind them.
RangeSignature 🔒
What a call to a range function has to be: where its range selector sits, how many constants it reads, and how they bind.
SelectorTiming

Constants§

ABSENT_OVER_TIME 🔒
The PromQL name of absent_over_time, which is not a range function here: it reads a matrix but answers with absent’s vector operation rather than with one value per series.
DEFAULT_SUBQUERY_STEP_MS 🔒
The step a subquery names when it names none, as in x[5m:].
PRESENCE_FOLD 🔒
The range function absent_over_time folds a window with, to learn whether a series has a sample in it.
SUPPORTED 🔒
The shapes this compiler accepts, quoted in every rejection.
VECTOR_FIELD 🔒
The field vector(s) reports its value under: a scalar names no column, so it is the planner’s default value column.

Functions§

absent_labels_of 🔒
The labels absent(expr) and absent_over_time(m[d]) answer with, as Prometheus createLabelsForAbsentFunction derives them: from the matchers of a vector or range selector, and none for any other expression.
absent_over_time_call 🔒
Binds absent_over_time(m[d]) into absent’s own vector operation, over a presence read of the window.
aggregate_expr 🔒
Compiles an aggregation into an aggregate node over the expression it groups.
argument_position 🔒
How a rejection names the argument a parameter was read from.
arity_error 🔒
The rejection of a math call that wrote given arguments where signature takes another count.
binary_op 🔒
The binary operator a PromQL token denotes, or None when there is no node for it.
binary_operation 🔒
Compiles a binary operation, preserving written operand order and vector matching modifiers.
bind_evaluated_binary 🔒
Uses the ordinary binary binder for two evaluated frame inputs.
bind_selector 🔒
bind_vector_binary 🔒
Binds the same vector operation over compiled or already evaluated operands.
broadcast 🔒
Wraps two compiled operands in the binary node that applies op to them.
compile
Compiles stmt into a tree that reads input and evaluates the statement’s expression over it.
compile_call 🔒
Compiles a range, pointwise, histogram or label function call.
compile_expr 🔒
Compiles expr into the node it denotes, recursing into whatever it is made of.
compile_selectors
Compiles stmt into a tree and returns a binding record for each selector occurrence.
compile_selectors_with_grid 🔒
Compiles stmt with an explicit evaluation grid. See compile_statement_with_grid.
compile_selectors_with_grid_and_value_field 🔒
compile_selectors_with_value_field 🔒
compile_statement
Compiles a statement that reads no data — a constant, time(), vector(time()), hour() — to a tree without inputs, on the statement’s grid.
compile_statement_with_grid 🔒
Compiles stmt with an explicit evaluation grid, instead of the one its bounds derive.
constant_argument 🔒
The constant argument folds to, or a rejection naming the argument and where it was read.
constant_value 🔒
The number expr folds to, or None when its value is not the statement’s to know.
data_free_value_field 🔒
grid_endpoints 🔒
grid_error 🔒
grouping 🔒
How a by or without modifier identifies a group, in the tree’s own terms.
histogram_function 🔒
The native histogram function name denotes, or None when it is not one.
info_call 🔒
Compiles info(v[, {selector}]) as its base v and the instant selection of the info series it reads.
info_input 🔒
Splits the arguments of info into the selector of its info series and the kernel’s selector.
info_reference 🔒
The @ anchor and offset the info series are selected at: those of the first selector of expr when every vector-producing path of it reaches a selector with the same effective reference time, and none otherwise, as Prometheus infoSeriesSelectTimestampAndOffset decides.
instant_selection 🔒
Compiles a bare selector into an instant selection over the raw input.
instant_selection_of 🔒
label_names 🔒
label_rewrite 🔒
The label change label_replace or label_join makes, and the vector it is made to, or None when name is neither.
lookback_millis 🔒
The statement’s lookback in milliseconds, which is how far back an instant selection reads.
map_call 🔒
Wraps op over the value of vector, which is the first argument of a math, native histogram or label function.
match_op 🔒
matching_labels 🔒
The labels on(...) or ignoring(...) names; with neither, every label but __name__.
math_op 🔒
The pointwise kernel name denotes over its vector, and that vector, or None when this compiler has no math kernel for that name.
math_signature 🔒
The signature the PromQL name calls, or None when this compiler has no math kernel for it.
millis 🔒
A millisecond magnitude as an i64, or None when it does not fit.
negate 🔒
Compiles a negation as the multiplication it is: -x is -1 * x.
not_implemented 🔒
A rejection of a shape outside SUPPORTED.
parameterized_call 🔒
Binds round, clamp, clamp_min or clamp_max whose parameters are not all constants, or returns None for any other call. Prometheus reads each parameter at every step, so each one is bound as the scalar node it compiles to; with constant parameters the call is a [MathOp].
parameterized_histogram_call 🔒
Compiles a classic histogram call with at least one varying scalar argument.
range_function 🔒
The range function a PromQL name denotes, or None when this compiler has no kernel for it.
range_millis 🔒
The window length of a range selector, which has to be longer than zero to select anything.
range_signature 🔒
The signature the PromQL name calls, or None when this compiler has no range kernel for it.
refuses_modifier 🔒
Whether a binary operation carries a modifier this compiler refuses to bind.
scalar_call 🔒
Binds time(), vector(s) and a date function called without an argument, or returns None when the call is none of them.
scalar_function_call 🔒
Binds max_of(a, b) or min_of(a, b), whose arguments are both scalars, or returns None for any other call.
scalar_matching 🔒
The matching a vector-and-constant operation records.
scalar_node 🔒
Binds value as a constant scalar on the grid of the expression being compiled.
selector_anchor_ms 🔒
The selector’s @ anchor in milliseconds since the Unix epoch, or None when it has none.
selector_offset_ms 🔒
The selector’s offset in milliseconds; positive reads earlier samples.
set_op 🔒
The set operator a PromQL token denotes, or None when it is not one.
set_operation 🔒
Compiles and, or or unless between two vectors, matched by the labels the query wrote.
statement_grid 🔒
The grid the statement is evaluated on: (end - start) / step + 1 points, the last of which does not pass end.
subquery_child_grid 🔒
Derives the SubqueryGrid of subquery, whose outer statement is evaluated on outer.
subquery_window 🔒
Binds function over subquery, whose inner expression is evaluated on a grid of its own.
timestamp_call 🔒
Compiles timestamp, projecting selected source times for a selector and evaluation times for an already evaluated vector.
unparen 🔒
The expression a parenthesised one wraps, however many parentheses were written.
validate_inputs 🔒
vector_aggregate 🔒
Binds topk, bottomk, limitk, limit_ratio or count_values as a vector operation over the expression it aggregates.
vector_call 🔒
Wraps the vector operation op over the value of vector, which keeps its type as a map does.
vector_matching 🔒
The matching a vector–vector operation records.
vector_of 🔒
vector(s): the scalar input as one series without labels, under one value field.
vector_op 🔒
The vector operation a sort, absent or a classic histogram function binds, and the vector it reads, or None when name is none of them.
window_call 🔒
Binds a range function over the matrix it is given, and the window that matrix opens.
window_selection 🔒
Binds function over the range selector vs[range] into a window over the raw input.