Lazy Evaluation and Streaming v2

The Runtime defers consumption of Streams and Arrays until the Step that requires the data executes. This approach allows a simple Workflow design to scale to large data volumes.

Deferred values

A Step can return a Stream or an asynchronous collection. The Runtime registers the concrete deferred value after the Step produces it.

Deferred consumption does not necessarily mean that the external operation has not started. Some Node Methods establish a Connection and fetch the first page or process the first batch before returning a deferred result. For example, the MySQL Node's Run Stored Procedure Method opens the database Connection, executes the statement, and attempts to read the first row before returning the remaining row stream.

For a Method with this behavior, execution has three stages:

  1. The producing Step starts the operation. A failure while connecting or obtaining the first result is reported at this stage.
  2. A downstream Step consumes the deferred result. Further reads or writes can occur during consumption, and a later failure can surface while that consumer is running.
  3. The Runtime finishes or closes the deferred source and releases its resources when the owning scope ends.

Check the selected Node Method's behavior when planning side effects and error handling. A write may have processed its first batch before a downstream Step starts, and stopping consumption does not undo work already performed.

The first downstream read begins consumption. If no branch reads the value, the Runtime drains it when the owning execution scope closes so that the producing Node can complete and release its resources.

A ForEach iteration has its own deferred-value scope. Values owned by a completed iteration are finalized before the Runtime moves beyond that scope, while values that escape the iteration remain owned by the surrounding Workflow scope.

Replay

Some designs require the same deferred value more than once. The Runtime analyses references in the compiled Workflow and adds replay when a later consumer must read data that an earlier consumer has already consumed.

Replay can add memory, disk, and processing overhead. In particular, the current asynchronous-enumerable replay buffer holds up to 10,000 records in memory. This limit applies when the Runtime must buffer a deferred asynchronous collection for replay; it is not a limit on every array or streamed integration.

Streams can use temporary spill files for replay. Flowgear removes these files when the Workflow releases them and retries cleanup when an immediate delete fails.

For best performance, avoid triggering replay support by allowing only one downstream Step to consume a Stream or Array.

Materialization

Most Nodes are able to operate by materializing small batches of data.

For example, the Salesforce Node will only materialize (create demand for and receive) a set of 200 records at a time for a write action since that is the limit for its underlying API. Once a batch has been processed, its memory is released and the next batch of records is demanded. This is memory efficient and is the mechanism that allows Workflows to process large numbers of records without having to adjust the Workflow design when it needs to scale for large numbers of records.

By contrast, some operations require a full materialization. Examples include:

  • A collection nested inside an object mapping.
  • A function that must inspect the complete collection (for example, aggregator functions)
  • A value that must be replayed in a form that cannot remain single-pass.

Materialization is sometimes required for correct results, but it removes the memory advantage of incremental consumption for that part of the Workflow.

Mapping collections

An array mapping normally contains a collection Expression and Properties that define the output item shape.

Items:
  Type: array
  Expression: "{readSource.Rows}"
  Properties:
    customerId:
      Type: string
      Expression: "{readSource.Rows.id}"
    displayName:
      Type: string
      Expression: "{readSource.Rows.name}"

The Runtime applies the child mappings as it enumerates the source. A scalar or Object can also be promoted to a one-item collection when the receiving contract expects an Array.

Logging deferred data

Workflow Logging wraps Streams and asynchronous collections so it can capture progress and a preview without changing the value received by the consumer.

Deferred sources like streams and collections show record or bytes processed in the Workflow logs while consumption is in progress. Preview limits, large-value storage rules, and redaction determine how much content remains available after the run.

If a root deferred Property uses Logging.Redact, its lifecycle can still be tracked but its preview content and counters are not retained in Workflow logs.

Design guidance

To preserve incremental processing:

  • Connect a deferred source directly to the Step that consumes it where practical.
  • Avoid referencing the same single-pass result from several branches unless replay is necessary.
  • Avoid nesting a large collection inside an object unless the receiving contract requires a materialized object.
  • Map target fields on the consuming array rather than creating intermediate copies only to rename or select fields.
  • Inspect Workflow logs to see whether rows or bytes continue to advance during a long run.
  • Confirm the behaviour and limits of the selected source and destination Nodes. The Runtime cannot make a Node stream data if its Method materializes the complete value.

See also