Module key

Expand description

This mod defines all the keys used in the metadata store (Metasrv). Specifically, there are these kinds of keys:

Datanode table key: __dn_table/{datanode_id}/{table_id}
- The value is a DatanodeTableValue struct; it contains table_id and the regions that belong to this Datanode.
- This key is primary used in the startup of Datanode, to let Datanode know which tables and regions it should open.
Table info key: __table_info/{table_id}
- The value is a TableInfoValue struct; it contains the whole table info (like column schemas).
- This key is mainly used in constructing the table in Datanode and Frontend.
Catalog name key: __catalog_name/{catalog_name}
- Indices all catalog names
Schema name key: __schema_name/{catalog_name}/{schema_name}
- Indices all schema names belong to the {catalog_name}
Table name key: __table_name/{catalog_name}/{schema_name}/{table_name}
- The value is a TableNameValue struct; it contains the table id.
- Used in the table name to table id lookup.
Flow info key: __flow/info/{flow_id}
- Stores metadata of the flow.
Flow route key: __flow/route/{flow_id}/{partition_id}
- Stores route of the flow.
Flow name key: __flow/name/{catalog}/{flow_name}
- Mapping {catalog}/{flow_name} to {flow_id}
Flownode flow key: __flow/flownode/{flownode_id}/{flow_id}/{partition_id}
- Mapping {flownode_id} to {flow_id}
Table flow key: __flow/source_table/{table_id}/{flownode_id}/{flow_id}/{partition_id}
- Mapping source table’s {table_id} to {flownode_id}
- Used in Flownode booting.
View info key: __view_info/{view_id}
- The value is a ViewInfoValue struct; it contains the encoded logical plan.
- This key is mainly used in constructing the view in Datanode and Frontend.
Kafka topic key: __topic_name/kafka/{topic_name}
- The key is used to track existing topics in Kafka.
- The value is a TopicNameValue struct; it contains the pruned_entry_id which represents the highest entry id that has been pruned from the remote WAL.
- When a region uses this topic, it should start replaying entries from pruned_entry_id + 1 (minimum available entry id).
Topic name to region map key __topic_region/{topic_name}/{region_id}
- Mapping {topic_name} to {region_id}

All keys have related managers. The managers take care of the serialization and deserialization of keys and values, and the interaction with the underlying KV store backend.

To simplify the managers used in struct fields and function parameters, we define “unify” table metadata manager: TableMetadataManager and flow metadata manager: FlowMetadataManager. It contains all the managers defined above. It’s recommended to just use this manager only.

The whole picture of flow keys will be like this:

__flow/ info/ {flow_id} route/ {flow_id}/ {partition_id}

name/ {catalog_name} {flow_name}

flownode/ {flownode_id}/ {flow_id}/ {partition_id}

source_table/ {table_id}/ {flownode_id}/ {flow_id}/ {partition_id}