Expand description
Compiles a PromQL AST into a [NativeTree].
This is the front half of the native evaluation model: it reads the parsed statement and writes down what it means as bound nodes. The native planner uses this compiler to build a tree for standalone execution. Its accepted expressions are described by the recursive rule below:
<expression> one of, recursively:
<selector> e.g. x, x offset -5m
<range function>(<matrix>) e.g. rate(x[5m]), rate(x[5m:1m] @ 1234)
quantile_over_time(<constant>, <matrix>) e.g. quantile_over_time(0.9, x[5m])
predict_linear(<matrix>, <constant>) e.g. predict_linear(x[5m], 300)
double_exponential_smoothing(<matrix>, <constant>, <constant>)
e.g. double_exponential_smoothing(x[5m], .5, .1)
<math function>(<expression>[, <constant>]*) e.g. abs(rate(x[5m])), clamp_min(x, 0)
<round or clamp>(<expression>, <scalar>+) e.g. round(x, time() / 10), clamp(x, 0, time())
<histogram function>(<expression>) e.g. histogram_count(x)
<sort>(<expression>[, <string>]*) e.g. sort_desc(x), sort_by_label(x, "job")
absent(<expression>) e.g. absent(x{job="a"})
absent_over_time(<matrix>) e.g. absent_over_time(x{job="a"}[5m])
<histogram>(<constant>+, <expression>) e.g. histogram_quantile(0.9, rate(x[5m]))
<label function>(<expression>, <string>*) e.g. label_replace(x, "a", "$1", "b", "(.*)")
<expression> <operator> <scalar> e.g. x * 100, x > bool time()
<scalar> <operator> <expression> e.g. 1 / x, time() - timestamp(x)
<scalar> <operator> <scalar> e.g. time() - 60, time() > bool 1
time(), pi(), vector(<scalar>) e.g. vector(time()), pi() * x
<date function>() e.g. hour(), which is `hour(vector(time()))`
-<expression> e.g. -x
<expression> <set operator> <expression> e.g. x and on(job) y, x unless y
info(<expression>[, {<matchers>}]) e.g. info(rate(x[5m]), {version=~".+"})
<aggregator> [(by|without)(<labels>)] (<expression>)
e.g. avg by(job)(rate(x[5m]) * 100)
<aggregator> (<expression>) [(by|without)(<labels>)]
e.g. count (x) by(job)
quantile(<constant>, <expression>) [(by|without)(<labels>)]
e.g. quantile(0.9, x) by(job)
<selector aggregator>(<constant>, <expression>) [(by|without)(<labels>)]
e.g. topk(3, x) by(job), limit_ratio(0.5, x)
count_values(<string>, <expression>) [(by|without)(<labels>)]
e.g. count_values("v", x)
<matrix> <selector>[<range>]<modifiers> a range selector: raw samples
| <expression>[<range>:<step>]<modifiers>
a subquery: <expression> evaluated on a grid;
the step may be omitted as `[<range>:]`
<scalar> <constant> | time() | <scalar> <operator> <scalar>
<modifiers> nothing, `offset <duration>`, `@ <anchor>`, or
both, in either order — e.g. `x[5m] @ 1234
offset 1m`where <set operator> is and, or or unless, optionally with on(...) or ignoring(...),
<aggregator> is sum, avg, count, min, max, group, stddev, stdvar or
quantile, <selector aggregator> is topk, bottomk, limitk or limit_ratio, the
function names are range_signature’s and math_signature’s, and which side
of the matrix a range function’s constants go on is RangeSignature’s: before it for
quantile_over_time, after it for predict_linear and double_exponential_smoothing. A math
function’s constants always follow its vector.
Parentheses are transparent wherever they appear — rate((x[5m])) is rate(x[5m]) — because
they say how a statement was typed rather than what it asks for.
A subquery’s <expression> is any shape above, so a subquery nests inside a subquery on a grid
of its own, as in rate(rate(x[5m:1m])[10m:30s]); a <step> left out means Prometheus’s default
evaluation interval, one minute. Its offset and its @ anchor are its own, not the inner
expression’s, so rate(x[5m:1m] @ end() offset 1m) reads one fixed interval. A <selector>
outside a <matrix> takes an offset and an optional @ anchor; x @ 100 selects at one
fixed instant across the evaluation grid, as does rate(x[5m] @ 100) for its window.
Those shapes become Input (raw samples) → Instant (the last sample within the statement’s
lookback), Input (raw samples) → Window (the range function over the statement’s grid),
<expression> → Window whose input is evaluated (the subquery, whose input grid is its own),
<expression> → Map (a pointwise math, native histogram or label function over what its
argument evaluates to) and, over
any of those, Binary (an arithmetic or comparison operation between vectors or with a
constant on one side), Scalar (that constant), Set (and, or or unless between two
vectors), Info (the expression and an Instant of the info metric reporting sample times),
Aggregate (one of the nine aggregators that
fold a group’s values into one value per grid point, optionally grouped) and Vector (an
operation over the whole vector at each point: a sort, absent, a classic histogram function, a
selecting aggregator or count_values). Everything else is a
[DataFusionError::NotImplemented] that names what was found and quotes the shapes above,
because rejecting a query the compiler does not understand is the only honest alternative to
compiling it into something else.
The compiler binds a subquery’s child grid and RangeWindow independently of runtime execution:
compilation describes the anchored interval, while runtime support determines how that bound
plan is consumed.
Two decisions keep the recursion from growing a branch per query. A constant argument — a
quantile, a rounding step, a clamp bound — is folded out of the statement by constant_value,
so quantile_over_time(1 - 0.1, x[5m]) binds the number it means rather than the expression it
was written as. A math function is bound by math_op from a list of names, so
clamp_min(abs(rate(x[5m])), 0) is two maps over one window because the arguments nest, not
because the compiler knows the query. Arithmetic inside a fold is the kernel operation
[BinaryOp::kind] names, so the number this compiler binds and the number the kernel computes
cannot disagree about %, ^ or atan2.
Two more are about what a node means rather than about which node to build. A binary operation
keeps its operands in the order they were written, because most operators are not commutative:
1 / x is not x / 1, and which side the constant is on is the only thing that tells them
apart. return_bool is carried as its own field, so x > 0 (which filters) and x > bool 0
(which reports zero or one) are one node with one field changed. A negation then needs no node of
its own: -x binds as the multiplication by -1 that it is, which the multiplication kernel
already computes.
Vector matching is bound with each selector’s own input, including the labels and cardinality
written on a vector–vector operation. A scalar has no input of its own: it is bound next to the
vector that gives it something to broadcast over, while a statement whose value is only a
constant is bound by compile_statement as that constant on the statement’s grid.
An aggregate is bound as [AggregateStage::Complete], which is a claim about the whole execution
rather than about the node: it says that every contribution to every group is in this execution’s
input. The caller has to keep that claim. One tree per partition does not: handing every
partition the same Complete tree sums each partition’s part and reports it as the group’s
whole, so a partitioned caller has to feed whole groups to one execution or split the aggregation
into Partial and Final instead. What a Complete node guarantees here is only the arithmetic
— that the values it folded are all the values it was given.
The statement’s grid, the selector’s lookback, window, offset and anchor are the only numbers
derived here; the legality of the tree is not checked here. [NativeTree::try_new] is the
single authority on that, so a node added later cannot be legal in one place and illegal in
another. What is refused before a tree exists is the scan’s shape: no shape below can consume
chunks of different series that interleave, so an Interleaved ordering is refused once, at the
entry, and [NativeNode::validate] refuses the same ordering for a tree built some other way.
Structs§
- Compiled
Input - One compiled selector occurrence and the scan layout bound for it.
- Compiled
Tree - Context 🔒
- Shared statement, per-selector binding callback, occurrence records, ordering and current grid.
- Info
Input 🔒 - The info series
info(v[, {selector}])reads, as a selector, and the selector its kernel combines by. - Selector
Requirement - Subquery
Grid 🔒 - The grid a subquery’s inner expression is evaluated on, and the window that reads it.
Enums§
- Math
Signature 🔒 - What a call to a math function has to be: the constants it reads after its vector, and how to bind them.
- Range
Signature 🔒 - What a call to a range function has to be: where its range selector sits, how many constants it reads, and how they bind.
- Selector
Timing
Constants§
- ABSENT_
OVER_ 🔒TIME - The PromQL name of
absent_over_time, which is not a range function here: it reads a matrix but answers withabsent’s vector operation rather than with one value per series. - DEFAULT_
SUBQUERY_ 🔒STEP_ MS - The step a subquery names when it names none, as in
x[5m:]. - PRESENCE_
FOLD 🔒 - The range function
absent_over_timefolds a window with, to learn whether a series has a sample in it. - SUPPORTED 🔒
- The shapes this compiler accepts, quoted in every rejection.
- VECTOR_
FIELD 🔒 - The field
vector(s)reports its value under: a scalar names no column, so it is the planner’s default value column.
Functions§
- absent_
labels_ 🔒of - The labels
absent(expr)andabsent_over_time(m[d])answer with, as PrometheuscreateLabelsForAbsentFunctionderives them: from the matchers of a vector or range selector, and none for any other expression. - absent_
over_ 🔒time_ call - Binds
absent_over_time(m[d])intoabsent’s own vector operation, over a presence read of the window. - aggregate_
expr 🔒 - Compiles an aggregation into an aggregate node over the expression it groups.
- argument_
position 🔒 - How a rejection names the argument a parameter was read from.
- arity_
error 🔒 - The rejection of a math call that wrote
givenarguments wheresignaturetakes another count. - binary_
op 🔒 - The binary operator a PromQL token denotes, or
Nonewhen there is no node for it. - binary_
operation 🔒 - Compiles a binary operation, preserving written operand order and vector matching modifiers.
- bind_
evaluated_ 🔒binary - Uses the ordinary binary binder for two evaluated frame inputs.
- bind_
selector 🔒 - bind_
vector_ 🔒binary - Binds the same vector operation over compiled or already evaluated operands.
- broadcast 🔒
- Wraps two compiled operands in the binary node that applies
opto them. - compile
- Compiles
stmtinto a tree that readsinputand evaluates the statement’s expression over it. - compile_
call 🔒 - Compiles a range, pointwise, histogram or label function call.
- compile_
expr 🔒 - Compiles
exprinto the node it denotes, recursing into whatever it is made of. - compile_
selectors - Compiles
stmtinto a tree and returns a binding record for each selector occurrence. - compile_
selectors_ 🔒with_ grid - Compiles
stmtwith an explicit evaluation grid. Seecompile_statement_with_grid. - compile_
selectors_ 🔒with_ grid_ and_ value_ field - compile_
selectors_ 🔒with_ value_ field - compile_
statement - Compiles a statement that reads no data — a constant,
time(),vector(time()),hour()— to a tree without inputs, on the statement’s grid. - compile_
statement_ 🔒with_ grid - Compiles
stmtwith an explicit evaluation grid, instead of the one its bounds derive. - constant_
argument 🔒 - The constant
argumentfolds to, or a rejection naming the argument and where it was read. - constant_
value 🔒 - The number
exprfolds to, orNonewhen its value is not the statement’s to know. - data_
free_ 🔒value_ field - grid_
endpoints 🔒 - grid_
error 🔒 - grouping 🔒
- How a
byorwithoutmodifier identifies a group, in the tree’s own terms. - histogram_
function 🔒 - The native histogram function
namedenotes, orNonewhen it is not one. - info_
call 🔒 - Compiles
info(v[, {selector}])as its basevand the instant selection of the info series it reads. - info_
input 🔒 - Splits the arguments of
infointo the selector of its info series and the kernel’s selector. - info_
reference 🔒 - The
@anchor and offset the info series are selected at: those of the first selector ofexprwhen every vector-producing path of it reaches a selector with the same effective reference time, and none otherwise, as PrometheusinfoSeriesSelectTimestampAndOffsetdecides. - instant_
selection 🔒 - Compiles a bare selector into an instant selection over the raw input.
- instant_
selection_ 🔒of - label_
names 🔒 - label_
rewrite 🔒 - The label change
label_replaceorlabel_joinmakes, and the vector it is made to, orNonewhennameis neither. - lookback_
millis 🔒 - The statement’s lookback in milliseconds, which is how far back an instant selection reads.
- map_
call 🔒 - Wraps
opover the value ofvector, which is the first argument of a math, native histogram or label function. - match_
op 🔒 - matching_
labels 🔒 - The labels
on(...)orignoring(...)names; with neither, every label but__name__. - math_op 🔒
- The pointwise kernel
namedenotes over its vector, and that vector, orNonewhen this compiler has no math kernel for that name. - math_
signature 🔒 - The signature the PromQL name calls, or
Nonewhen this compiler has no math kernel for it. - millis 🔒
- A millisecond magnitude as an
i64, orNonewhen it does not fit. - negate 🔒
- Compiles a negation as the multiplication it is:
-xis-1 * x. - not_
implemented 🔒 - A rejection of a shape outside
SUPPORTED. - parameterized_
call 🔒 - Binds
round,clamp,clamp_minorclamp_maxwhose parameters are not all constants, or returnsNonefor any other call. Prometheus reads each parameter at every step, so each one is bound as the scalar node it compiles to; with constant parameters the call is a [MathOp]. - parameterized_
histogram_ 🔒call - Compiles a classic histogram call with at least one varying scalar argument.
- range_
function 🔒 - The range function a PromQL name denotes, or
Nonewhen this compiler has no kernel for it. - range_
millis 🔒 - The window length of a range selector, which has to be longer than zero to select anything.
- range_
signature 🔒 - The signature the PromQL name calls, or
Nonewhen this compiler has no range kernel for it. - refuses_
modifier 🔒 - Whether a binary operation carries a modifier this compiler refuses to bind.
- scalar_
call 🔒 - Binds
time(),vector(s)and a date function called without an argument, or returnsNonewhen the call is none of them. - scalar_
function_ 🔒call - Binds
max_of(a, b)ormin_of(a, b), whose arguments are both scalars, or returnsNonefor any other call. - scalar_
matching 🔒 - The matching a vector-and-constant operation records.
- scalar_
node 🔒 - Binds
valueas a constant scalar on the grid of the expression being compiled. - selector_
anchor_ 🔒ms - The selector’s
@anchor in milliseconds since the Unix epoch, orNonewhen it has none. - selector_
offset_ 🔒ms - The selector’s offset in milliseconds; positive reads earlier samples.
- set_op 🔒
- The set operator a PromQL token denotes, or
Nonewhen it is not one. - set_
operation 🔒 - Compiles
and,ororunlessbetween two vectors, matched by the labels the query wrote. - statement_
grid 🔒 - The grid the statement is evaluated on:
(end - start) / step + 1points, the last of which does not passend. - subquery_
child_ 🔒grid - Derives the
SubqueryGridofsubquery, whose outer statement is evaluated onouter. - subquery_
window 🔒 - Binds
functionoversubquery, whose inner expression is evaluated on a grid of its own. - timestamp_
call 🔒 - Compiles
timestamp, projecting selected source times for a selector and evaluation times for an already evaluated vector. - unparen 🔒
- The expression a parenthesised one wraps, however many parentheses were written.
- validate_
inputs 🔒 - vector_
aggregate 🔒 - Binds
topk,bottomk,limitk,limit_ratioorcount_valuesas a vector operation over the expression it aggregates. - vector_
call 🔒 - Wraps the vector operation
opover the value ofvector, which keeps its type as a map does. - vector_
matching 🔒 - The matching a vector–vector operation records.
- vector_
of 🔒 vector(s): the scalarinputas one series without labels, under one value field.- vector_
op 🔒 - The vector operation a sort,
absentor a classic histogram function binds, and the vector it reads, orNonewhennameis none of them. - window_
call 🔒 - Binds a range function over the matrix it is given, and the window that matrix opens.
- window_
selection 🔒 - Binds
functionover the range selectorvs[range]into a window over the raw input.