Crate flow

This crate manage dataflow in Greptime, including adapter, expr, plan, repr and utils. It can transform substrait plan into it’s own plan and execute it. It also contains definition of expression, adapter and plan, and internal state management.

Re-exports

pub use error::Error;

pub use error::Result;

Modules

adapter 🔒

for getting data from source and sending results to sink and communicating with other parts of the database

batching_mode 🔒

Run flow as batching mode which is time-window-aware normal query triggered when new data arrives

compute 🔒

Build and Compute the dataflow

df_optimizer 🔒

Datafusion optimizer for flow plan

engine 🔒

Define a trait for flow engine, which is used by both streaming engine and batch engine

error

Error definition for flow module

expr 🔒

for declare Expression in dataflow, including map, reduce, id and join(TODO!) etc.

heartbeat

Send heartbeat from flownode to metasrv

metrics 🔒

Some of the metrics used in the flow module.

plan 🔒

This module contain basic definition for dataflow’s plan that can be translate to hydro dataflow

repr 🔒

basically a wrapper around the datatype crate for basic Data Representation

server 🔒

Implementation of grpc service for flow node

transform 🔒

Transform Substrait into execution plan

utils 🔒

utilities for managing state of dataflow execution

Structs

FlowAuthHeader

FlowConfig

Flow config that exists both in standalone&distributed mode

FlownodeBuilder

FlownodeInstance Builder

FlownodeInstance

The flownode server instance.

FlownodeOptions

Options for flow node

FlownodeServer

FlownodeServiceBuilder

Useful in distributed mode

FrontendInvoker

Basically a tiny frontend that communicates with datanode, different from FrontendClient which connect to a real frontend instead, this is used for flow’s streaming engine. And is for simple query.

StreamingEngine

FlowNodeManager manages the state of all tasks in the flow node, which should be run on the same thread

Enums

FrontendClient

A simple frontend client able to execute sql using grpc protocol

Traits

GrpcQueryHandlerWithBoxedError

Just like GrpcQueryHandler but use BoxedError

Functions

get_flow_auth_options

Type Aliases

FlowStreamingEngineRef

Arc-ed FlowNodeManager, cheaper to clone