Parquet Convert v2
Apache Parquet is a columnar file format used to exchange structured datasets efficiently.
Use the Node to create Parquet documents from object rows, parse documents into rows and inspect schemas without reading their data pages.
Revision History
0.0.0.8 - Initial release.
Setup Notes
This Node supports Cloud and Local Runtimes.
No Connection is required. Supply object rows for Create or a readable Parquet stream for Parse and Inspect Schema. Use templates to configure write or read options.
Methods
Create
Creates a Parquet document from streamed object rows.
| Parameter | Type | Description |
|---|---|---|
Items |
Array | Object rows to write. Each item must be a JSON object. |
Options |
Object | Options that control schema, row groups, compression, and metadata. |
| Return | Type | Description |
|---|---|---|
Content |
Stream | Seekable Parquet document positioned at its beginning. |
Parse
Parses a Parquet document into streamed object rows.
| Parameter | Type | Description |
|---|---|---|
Document |
Stream | Readable Parquet document stream. |
Options |
Object | Options that select fields, row groups, offset, and limit. |
| Return | Type | Description |
|---|---|---|
Rows |
Array | Parsed object rows. |
Inspect Schema
Inspects Parquet schema and file metadata.
| Parameter | Type | Description |
|---|---|---|
Document |
Stream | Readable Parquet document stream. |
| Return | Type | Description |
|---|---|---|
Schema |
Object | Schema and file metadata. |
Usage Notes
- Create returns a seekable stream positioned at its beginning. Dispose the output stream when finished. Input rows must be JSON objects.
- SchemaInferenceMode FirstRowGroup infers from the first bounded batch; FullInput inspects the complete input through temporary storage; Explicit uses ordered Columns and supports empty datasets.
- Write Options include compression, row-group size, dictionary encoding, unknown-field handling and footer Metadata. Decimal, timestamp and nested-field contracts can be specified explicitly.
- Parse Options select Columns, RowGroups, Offset and Limit. Parsing validates the document and selections before returning its row stream.
- Inspect Schema reads footer metadata, including schema, row counts, row groups, compression and custom metadata.
Examples
Create and read a document
- Connect an upstream object stream to Create Items. For example, supply rows with OrderId and Amount:
{
"OrderId": 1001,
"Amount": 12.5
}
- Select FirstRowGroup schema inference or define explicit Columns when stable types, decimal precision or an empty dataset are required.
- Connect the returned Content stream to Parse Document, or store it with a file or storage Node.
- Use Inspect Schema to read column and row-group metadata without parsing all data pages.