Module utils

some utils for helping with batching mode

Structs

AddFilterRewriter

Find out the Filter Node corresponding to innermost(deepest) WHERE and add a new filter expr to it

ColumnMatcherRewriter

Optionally add to the final select columns like update_at if the sink table has such column (which doesn’t necessary need to have exact name just need to be a extra timestamp column) and __ts_placeholder(this column need to have exact this name and be a timestamp) with values like now() and 0

FindGroupByFinalName

Helper to find the innermost group by expr in schema, return None if no group by expr

Functions

df_plan_to_sql

gen_plan_with_matching_schema 🔒

Generate a plan that matches the schema of the sink table from given sql by alias and adding auto columns

get_table_info_df_schema

sql_to_df_plan

Convert sql to datafusion logical plan Also support TQL (but only Eval not Explain or Analyze)