Handle Large Data v2

Keep large data deferred for as long as the source and target contracts allow. Map only required fields and avoid creating extra consumers that force replay or materialization.

Choose the appropriate shape

  • Use a stream for opaque document or binary content when both Nodes support it.
  • Use an array when the Workflow must address item fields or use collection functions.
  • Parse a document only when a later mapping needs its structured contents.
  • Do not convert a Stream to text or an Array solely to inspect it in the designer.

Design one clear consumption path

The Runtime pulls deferred sources when a downstream consumer asks for data. The producing Node may already have opened its Connection and fetched the first page or written the first batch before returning the deferred result. Later pages or batches can then be processed as the consumer requests them.

Multiple consumers can require replay. Replayed Streams may spill to temporary files, while collection replay is limited to 10,000 items to avoid excessive memory allocation. Avoid branching the same large source into several consumers unless reuse is deliberate and tested.

A direct Stream source cannot be exposed through more than one top-level Workflow Return. Return one stream or consume it inside the Workflow.

Control tests and logs

  1. Start with a small page, date range, or file.
  2. Inspect Step duration, deferred counters, and completion state in Logs.
  3. Confirm whether the source and target overlap in time as expected.
  4. Increase volume gradually while monitoring the selected Cluster.

Workflow logging captures bounded previews and can retain large values separately. Redaction hides selected Property content, but logging, multiple consumers, and downstream Node behavior can still affect throughput.

See Lazy Evaluation and Streaming, Stream Data Type, and Map Arrays.