The storage adapter interface.
Each adapter implements these methods to provide table management for a specific storage backend (e.g., Hive/S3, Iceberg, ClickHouse).
Type Parameters
| Parameter | Constraint | Default |
|---|---|---|
TConfig | AdapterConfig | AdapterConfig |
Properties
| Name | Type |
|---|---|
| typeRequired | TConfig["type"] |
The adapter type identifier. | |
| createTableRequired | (definition: TableDefinition) => Promise<void> |
Creates a table in the storage backend. Uses IF NOT EXISTS by default. | |
| dropTableRequired | (catalog: string, schema: string, table: string) => Promise<void> |
Drops a table from the storage backend. Uses IF EXISTS — does not throw if the table doesn’t exist. | |
| replaceTableRequired | (definition: TableDefinition) => Promise<void> |
Drops and recreates a table. Useful for replacing external tables with updated schemas. | |
Base configuration shared by all adapters.
Properties
| Name | Type |
|---|---|
| typeRequired | string |
Unique identifier for this adapter type. | |
A table definition describing what to create in the storage backend.
Properties
| Name | Type |
|---|---|
| catalogRequired | string |
The catalog name. | |
| schemaRequired | string |
The schema name. | |
| tableRequired | string |
The table name. | |
| columnsRequired | ColumnDefinition[] |
Column definitions (name + backend-specific type). | |
Creates storage operations backed by files-sdk. Adapter selection is based on config.type:
- “s3”: reads credentials from AWS_* environment variables
- ”minio”: reads credentials from MINIO_* environment variables
Parameters
| Name | Type |
|---|---|
| configRequired | StorageConfig |
Returns
StorageOperationsError class for storage operation failures, providing path and operation context.
Constructor
| Name | Type |
|---|---|
| messageRequired | string |
| pathRequired | string |
| operationRequired | "upload" | "delete" |
| options? | ErrorOptions |
Properties
| Name | Type |
|---|---|
| pathRequired | string |
| operationRequired | "upload" | "delete" |
Storage configuration for S3 or MinIO adapters.
Credentials are read internally from environment variables:
- S3: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_DEFAULT_REGION, AWS_ENDPOINT_URL
- MinIO: MINIO_ACCESS_KEY_ID, MINIO_SECRET_ACCESS_KEY, requires explicit endpoint
Properties
| Name | Type |
|---|---|
| typeRequired | StorageType |
Storage adapter type. "s3" | |
| bucketRequired | string |
Bucket name. | |
| region? | string |
Region override. Falls back to AWS_DEFAULT_REGION env var for S3. | |
| endpoint? | string |
Custom endpoint. Required for MinIO, optional for S3 (falls back to AWS_ENDPOINT_URL env var). | |
Supported storage adapter types.
Definition
"s3" | "minio"Storage operations interface for S3-compatible object stores.
Properties
| Name | Type |
|---|---|
| uploadRequired | (buffer: Uint8Array, targetPath: string) => Promise<void> |
Uploads a file buffer to the specified S3 path. | |
| deletePrefixRequired | (prefix: string) => Promise<void> |
Deletes all objects under the given prefix. | |
Manages Hive external table DDL operations (DROP + CREATE) for the mutation write pipeline.
Properties
| Name | Type |
|---|---|
| recreateTableRequired | (definition: HiveTableDefinition) => Promise<void> |
Drops and recreates a single Hive external table. Executes DROP TABLE IF EXISTS followed by CREATE TABLE. | |
| recreateTablePairRequired | (
latestDefinition: HiveTableDefinition,
allDefinition: HiveTableDefinition
) => Promise<void> |
For full_load_append: manages both _latest and _all tables. Creates both tables, and attempts rollback (best-effort drop both) if either creation fails. | |
| buildExternalLocationRequired | (path: string) => string |
Builds a properly-formatted external location URI for Hive tables. Uses the s3a:// scheme required by Hive’s Hadoop FileSystem. | |
Configuration for the Hive Table Manager.
Properties
| Name | Type |
|---|---|
| clientRequired | TrinoClient |
The Trino client instance to use for DDL operations. | |
| bucketRequired | string |
S3 bucket name for external table locations. | |
Definition of a Hive external table to create.
Properties
| Name | Type |
|---|---|
| catalogRequired | string |
The catalog name. | |
| schemaRequired | string |
The schema name. | |
| tableNameRequired | string |
The table name. | |
| externalLocationRequired | string |
S3 location for the external table. | |
| columnsRequired | { name: string; type: string }[] |
SQL column definitions from JSON Schema. | |
Creates a HiveTableManager that executes DDL via the Trino client.
Parameters
| Name | Type |
|---|---|
| configRequired | HiveTableManagerConfig |
Returns
HiveTableManagerExecutes the write pipeline:
- Convert records to Parquet via
Parameters
| Name | Type |
|---|---|
| inputRequired | WritePipelineInput |
Returns
Promise<void>Generates a Hive-style partition path based on date and format. Format options:
- “year”: year=YYYY/ <uuid> .parquet
- ”year/month”: year=YYYY/month=MM/ <uuid> .parquet
- ”year/month/day”: year=YYYY/month=MM/day=DD/ <uuid> .parquet
Parameters
| Name | Type |
|---|---|
| dateRequired | Date |
| format? | PartitioningFormat |
Default: "year/month/day" | |
Returns
stringGenerates a flat file path (no partitioning).
Returns
stringNormalizes the raw partitioning config into a resolved structure.
Parameters
| Name | Type |
|---|---|
| partitioning? | PartitioningValue |
Default: true | |
| partitioningFormat? | PartitioningFormat |
Default: "year/month/day" | |
Returns
ResolvedPartitioningAdds load_timestamp, load_timestamp_year, and load_timestamp_month to JSON Schema for consistent Parquet + Hive DDL derivation. Returns a new schema object (does not mutate the input).
Parameters
| Name | Type |
|---|---|
| jsonSchemaRequired | JsonSchema |
Returns
JsonSchemaInjects load_timestamp, load_timestamp_year, and load_timestamp_month into each record. Returns new record array (does not mutate input).
Parameters
| Name | Type |
|---|---|
| recordsRequired | Record<string, unknown>[] |
| timestampRequired | Date |
Returns
Record<string, unknown>[]Error thrown when a record’s partition field is missing, null, or invalid.
Constructor
| Name | Type |
|---|---|
| fieldNameRequired | string |
| reasonRequired | "missing" | "null" | "invalid_date" |
| recordIndexRequired | number |
| value? | unknown |
Properties
| Name | Type |
|---|---|
| fieldNameRequired | string |
| reasonRequired | "missing" | "null" | "invalid_date" |
| recordIndexRequired | number |
| value? | unknown |
Parses a custom partition format string into an array of segments.
Format: “segment/segment/…“ where each segment is either:
- A plain field name: “customer_id” → extracts raw value
- A field with date component: “event_date:year” → extracts year from ISO date
Parameters
| Name | Type |
|---|---|
| formatRequired | string |
Returns
PartitionSegment[]Generates a custom partition path from a record and parsed segments. For each segment:
- Without component: formats as fieldName=<value>
- With component: parses the field as ISO date, extracts the component, formats as component=<value> Appends /<uuid>.parquet at the end.
Parameters
| Name | Type |
|---|---|
| recordRequired | Record<string, unknown> |
| segmentsRequired | PartitionSegment[] |
| recordIndex? | number |
Default: 0 | |
Returns
stringGroups records by their custom partition path. Records with the same partition key go to the same file (one UUID per unique partition key).
Parameters
| Name | Type |
|---|---|
| recordsRequired | Record<string, unknown>[] |
| segmentsRequired | PartitionSegment[] |
Returns
Map<string, Record<string, unknown>[]>Load strategy determines how data is stored and how Hive tables are managed.
Definition
"full_load" | "full_load_append" | "append"Resolved partitioning mode after normalizing the raw config.
Definition
"disabled" | "timestamp" | "field" | "custom"The normalized partitioning configuration used internally by the pipeline.
Properties
| Name | Type |
|---|---|
| modeRequired | PartitionMode |
| formatRequired | PartitioningFormat |
| fieldName? | string |
| formatString? | string |
Configuration for the write pipeline.
Properties
| Name | Type |
|---|---|
| loadStrategy? | LoadStrategy |
The load strategy for this endpoint. "full_load" | |
| type? | StorageType |
Storage adapter type. "s3" | |
| bucketRequired | string |
Bucket name for storing Parquet files. | |
| basePathRequired | string |
The base path for storing Parquet files. | |
| region? | string |
Optional region override. | |
| endpoint? | string |
Optional custom endpoint for S3-compatible storage. | |
| tableRequired | {
catalog: string
schema: string
tableName: string
} |
Hive table definition for DDL management. | |
| trinoClientRequired | TrinoClient |
The Trino client instance for DDL operations. | |
| partitioning? | PartitioningValue |
Partitioning mode. true partitions by write timestamp, false disables, or a string for field-based/custom. true | |
| partitioningFormat? | PartitioningFormat |
Partition format granularity. "year/month/day" | |
Input for the write pipeline execution.
Properties
| Name | Type |
|---|---|
| recordsRequired | Record<string, any>[] |
The records to persist. Accepts any array of objects with string keys. | |
| jsonSchemaRequired | JsonSchema |
The JSON Schema describing the record structure. | |
| configRequired | WritePipelineConfig |
Pipeline configuration. | |
A parsed segment from a custom partition format string.
Properties
| Name | Type |
|---|---|
| fieldNameRequired | string |
The field name to extract from the record. | |
| component? | PartitioningComponent |
If set, extract this date component from the field’s ISO date value. | |
Configuration for the Trino + Hive + S3 adapter.
Properties
| Name | Type |
|---|---|
| typeRequired | "trino-hive-s3" |
| clientRequired | TrinoClient |
The Trino client instance to use for DDL operations. | |
| bucketRequired | string |
S3 bucket name for external table locations. | |
| prefix? | string |
Optional base prefix within the bucket (default: “”). | |
| format? | "PARQUET" | "ORC" | "AVRO" | "JSON" |
Storage format (default: “PARQUET”). | |
Adapter for Hive external tables stored on S3.
Generates CREATE TABLE statements with:
- external_location pointing to s3://<bucket>/<prefix>/<schema>/<table>
- format set to the configured format (default: PARQUET)
Parameters
| Name | Type |
|---|---|
| configRequired | Omit<TrinoHiveS3Config, "type"> |
Returns
StorageAdapter<TrinoHiveS3Config>