Struct DataflowState

pub struct DataflowState {
    schedule_subgraph: Rc<RefCell<BTreeMap<i64, VecDeque<SubgraphId>>>>,
    as_of: Rc<RefCell<i64>>,
    err_collector: ErrCollector,
    arrange_used: Vec<ArrangeHandler>,
    expire_after: Option<i64>,
    last_exec_time: Option<i64>,
}

input/output of a dataflow One ComputeState manage the input/output/schedule of one Hydroflow

Fields

schedule_subgraph: Rc<RefCell<BTreeMap<i64, VecDeque<SubgraphId>>>>

it is important to use a deque to maintain the order of subgraph here TODO(discord9): consider dedup? Also not necessary for hydroflow itself also do dedup when schedule

as_of: Rc<RefCell<i64>>

Frontier (in sys time) before which updates should not be emitted.

We must apply it to sinks, to ensure correct outputs. We should apply it to sources and imported shared state, because it improves performance. Which means it’s also the current time in temporal filter to get current correct result

err_collector: ErrCollector

error collector local to this ComputeState, useful for distinguishing errors from different Hydroflow

arrange_used: Vec<ArrangeHandler>

save all used arrange in this dataflow, since usually there is no delete operation we can just keep track of all used arrange and schedule subgraph when they need to be updated

expire_after: Option<i64>

the time arrangement need to be expired after a certain time in milliseconds

last_exec_time: Option<i64>

the last time each subgraph executed

Implementations

impl DataflowState

pub fn new_arrange(&mut self, name: Option<Vec<String>>) -> ArrangeHandler

pub fn run_available_with_schedule(&mut self, df: &mut Hydroflow<'_>) -> bool

schedule all subgraph that need to run with time <= as_of and run_available()

return true if any subgraph actually executed

pub fn get_scheduler(&self) -> Scheduler

pub fn current_time_ref(&self) -> Rc<RefCell<i64>>

return a handle to the current time, will update when as_of is updated

so it can keep track of the current time even in a closure that is called later

pub fn current_ts(&self) -> i64

pub fn set_current_ts(&mut self, ts: i64)

pub fn get_err_collector(&self) -> ErrCollector

pub fn set_expire_after(&mut self, after: Option<i64>)

pub fn expire_after(&self) -> Option<i64>

pub fn get_state_size(&self) -> usize

pub fn set_last_exec_time(&mut self, time: i64)

pub fn last_exec_time(&self) -> Option<i64>

Trait Implementations

impl Debug for DataflowState

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for DataflowState

fn default() -> DataflowState

Returns the “default value” for a type.