docId: adapters

source: adapters

slug: adapters

title: Adapters

documentType: unknown

contentOrigin: static-doc

content:
---
title: Adapters
navTitle: Adapters
description: Storage adapters for the LakeQL write pipeline — Parquet generation, S3/MinIO uploads, and Hive table management.
entrypoint: /docs/adapters/overview/introduction
---

navTitle: Adapters

keywords: adapters storage lakeql write pipeline


docId: adapters/overview/hive-table-manager

slug: overview/hive-table-manager

title: Hive Table Manager

text: How it works

text: External table locations

text: Table strategies

text: Usage

text: Generated DDL

text: Configuration

text: Table definition

content:
---
title: Hive Table Manager
navTitle: Hive Table Manager
description: How the write pipeline manages Hive external tables in Trino.
---

The Hive Table Manager is responsible for creating and maintaining external tables in Trino's Hive catalog. After the write pipeline uploads Parquet files to object storage, the table manager ensures a corresponding Hive table exists that points to that data — making it immediately queryable.

## How it works

Every mutation write goes through this flow:

1. Data is written as Parquet to S3/MinIO
2. The existing Hive table is dropped (if present)
3. A new external table is created pointing to the uploaded data

This "drop + create" approach ensures the table schema always matches the current data, even if fields were added or removed.

## External table locations

Hive external tables point to a **directory** in object storage, not a single file. The table manager uses the `s3a://` URI scheme, which is required by Hive's underlying Hadoop FileSystem.

```
s3a://<bucket>/<basePath>/latest.parquet/
         │         │              │
         │         │              └─ Directory containing Parquet file(s)
         │         └─ Configured basePath for the endpoint
         └─ Configured bucket name
```

<Note>
  The `s3a://` scheme is always used — regardless of whether the storage is AWS
  S3 or MinIO. The standard `s3://` scheme is not recognized by Hadoop's
  FileSystem.
</Note>

## Table strategies

Depending on the [load strategy](/docs/adapters/write-pipeline/load-strategies), different tables are managed:

| Strategy           | Tables created                                                               |
| ------------------ | ---------------------------------------------------------------------------- |
| `full_load`        | `<tableName>` → `latest.parquet/`                                            |
| `full_load_append` | `<tableName>_latest` → `latest.parquet/`, `<tableName>_all` → `all.parquet/` |
| `append`           | `<tableName>` → `all.parquet/`                                               |

For `full_load_append`, both tables are managed atomically — if creating one fails, the manager attempts a best-effort rollback by dropping both.

## Usage

The table manager is typically used internally by `executeWritePipeline`. For direct usage:

```ts
import { createHiveTableManager } from "@lakeql/adapters"
import { TrinoClient } from "@lakeql/trino-client"

const trinoClient = new TrinoClient({
  host: "https://trino.example.com",
  port: 8443,
  auth: { type: "basic", username: "admin", password: "secret" },
  catalog: "hive",
})

const hiveManager = createHiveTableManager({
  client: trinoClient,
  bucket: "my-datalake",
})

// Build a properly formatted location URI
const location = hiveManager.buildExternalLocation(
  "analytics/events/latest.parquet/"
)
// => "s3a://my-datalake/analytics/events/latest.parquet/"

// Create a table pointing to that location
await hiveManager.recreateTable({
  catalog: "hive",
  schema: "analytics",
  tableName: "events",
  externalLocation: location,
  columns: [
    { name: "event_id", type: "VARCHAR" },
    { name: "message", type: "VARCHAR" },
    { name: "created_at", type: "TIMESTAMP(3)" },
  ],
})
```

## Generated DDL

The above produces:

```sql
DROP TABLE IF EXISTS hive.analytics.events;

CREATE TABLE hive.analytics.events (
  event_id VARCHAR,
  message VARCHAR,
  created_at TIMESTAMP(3)
)
WITH (
  external_location = 's3a://my-datalake/analytics/events/latest.parquet/',
  format = 'PARQUET'
);
```

## Configuration

<InterfaceReference
  file="adapters/src/hive-table-manager"
  name="HiveTableManagerConfig"
  mode="declaration"
/>

## Table definition

<InterfaceReference
  file="adapters/src/hive-table-manager"
  name="HiveTableDefinition"
  mode="declaration"
/>

navTitle: Hive Table Manager

keywords: table external manager write pipeline


docId: adapters/overview/introduction

slug: overview/introduction

title: Introduction

text: What it does

text: Package exports

text: Architecture

text: Storage types

text: Installation

content:
---
title: Introduction
navTitle: Introduction
description: Overview of the @lakeql/adapters package and how it powers the mutation write pipeline.
---

The `@lakeql/adapters` package provides the storage layer for LakeQL mutations. When a GraphQL mutation is executed, the adapters package handles everything from converting records to Parquet, uploading to object storage, and managing Hive external table DDL in Trino.

## What it does

The package is responsible for three core tasks:

1. **Parquet generation** — Converts input records to columnar Parquet format using `@lakeql/parquet`
2. **Object storage** — Uploads Parquet files to S3 or S3-compatible storage (MinIO, etc.)
3. **Hive DDL management** — Creates and manages external tables in Trino's Hive catalog so the data is immediately queryable

## Package exports

The main entry point (`@lakeql/adapters`) exports storage operations, the write pipeline, and the Hive table manager. See the [API Reference](/docs/adapters/api-reference) for the full list of exports.

## Architecture

```mermaid
graph TD
    A[GraphQL Mutation] --> B[executeWritePipeline]
    B --> C[writeParquet]
    B --> D[Storage Operations]
    B --> E[Hive Table Manager]
    C --> D
    D --> F[S3 / MinIO]
    E --> G[Trino DDL]
```

The `executeWritePipeline` function orchestrates the full flow. It accepts records, a JSON Schema describing their shape, and a pipeline configuration that determines the load strategy, storage target, and table definition.

## Storage types

The adapters support two storage backends:

| Type    | Description                 | Credentials                                      |
| ------- | --------------------------- | ------------------------------------------------ |
| `s3`    | AWS S3                      | `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`     |
| `minio` | S3-compatible (MinIO, etc.) | `MINIO_ACCESS_KEY_ID`, `MINIO_SECRET_ACCESS_KEY` |

Both use the same S3 protocol under the hood. The `minio` type requires an explicit `endpoint` configuration.

## Installation

<Command variant="install">@lakeql/adapters</Command>

Peer dependencies for S3/MinIO storage:

<Command variant="install">
  @aws-sdk/client-s3 @aws-sdk/s3-presigned-post @aws-sdk/s3-request-presigner
</Command>

navTitle: Introduction

keywords: package introduction overview lakeqladapters powers


docId: adapters/storage/storage-operations

slug: storage/storage-operations

title: Storage Operations

text: createStorageOperations

text: Configuration

text: Operations

text: Environment variables

text: Error handling

content:
---
title: Storage Operations
navTitle: Storage Operations
description: Low-level S3/MinIO storage operations for uploading and managing Parquet files.
---

Storage operations provide the low-level interface for interacting with S3 or S3-compatible object storage. The write pipeline uses these internally, but they are exported for advanced use cases.

## createStorageOperations

Creates a storage operations instance backed by `files-sdk`. The adapter selection is based on `config.type`:

- `"s3"` — reads credentials from `AWS_*` environment variables
- `"minio"` — reads credentials from `MINIO_*` environment variables

```ts
import { createStorageOperations } from "@lakeql/adapters"

const storage = createStorageOperations({
  type: "minio",
  bucket: "my-datalake",
  endpoint: "http://localhost:9000",
})
```

## Configuration

<InterfaceReference
  file="adapters/src/storage-operations"
  name="StorageConfig"
  mode="declaration"
/>

## Operations

The returned object provides these methods:

| Method                 | Description                                           |
| ---------------------- | ----------------------------------------------------- |
| `upload(buffer, path)` | Upload a `Uint8Array` to the given path in the bucket |
| `deletePrefix(prefix)` | Delete all objects under the given prefix             |

## Environment variables

### S3

| Variable                | Description              |
| ----------------------- | ------------------------ |
| `AWS_ACCESS_KEY_ID`     | AWS access key           |
| `AWS_SECRET_ACCESS_KEY` | AWS secret key           |
| `AWS_DEFAULT_REGION`    | Default region           |
| `AWS_ENDPOINT_URL`      | Optional custom endpoint |

### MinIO

| Variable                  | Description      |
| ------------------------- | ---------------- |
| `MINIO_ACCESS_KEY_ID`     | MinIO access key |
| `MINIO_SECRET_ACCESS_KEY` | MinIO secret key |

<Note>
  The `endpoint` field in the config is required for MinIO. For S3, it's
  optional and defaults to the standard AWS endpoint for the region.
</Note>

## Error handling

Storage errors are wrapped in a `StorageError` with a descriptive message including the operation and path:

```ts
import { StorageError } from "@lakeql/adapters"

try {
  await storage.upload(buffer, "path/to/file.parquet")
} catch (error) {
  if (error instanceof StorageError) {
    console.error(error.message)
    // "Storage upload failed for "path/to/file.parquet": The specified bucket does not exist"
  }
}
```

<InlineReference
  file="adapters/src/storage-operations"
  include={["StorageError"]}
/>

navTitle: Storage Operations

keywords: operations storage low-level s3minio uploading


slug: write-pipeline/execute-write-pipeline

title: executeWritePipeline

text: Signature

text: Input

text: Configuration

text: Usage

text: Pipeline steps

content:
---
title: executeWritePipeline
navTitle: executeWritePipeline
description: The main entry point for persisting mutation data through the write pipeline.
---

`executeWritePipeline` is the primary function for persisting GraphQL mutation input. It converts records to Parquet, uploads them to object storage, and manages the corresponding Hive external table in Trino.

## Signature

```ts
function executeWritePipeline(input: WritePipelineInput): Promise<void>
```

## Input

<InterfaceReference
  file="adapters/src/write-pipeline"
  name="WritePipelineInput"
  mode="declaration"
/>

## Configuration

<InterfaceReference
  file="adapters/src/write-pipeline"
  name="WritePipelineConfig"
  mode="declaration"
/>

## Usage

```ts
import { executeWritePipeline } from "@lakeql/adapters"
import { TrinoClient } from "@lakeql/trino-client"

const trinoClient = new TrinoClient({
  host: "https://trino.example.com",
  port: 8443,
  auth: { type: "basic", username: "admin", password: "secret" },
  catalog: "hive",
})

await executeWritePipeline({
  records: [
    {
      event_id: "abc-123",
      message: "Hello",
      timestamp: "2025-01-15T10:30:00Z",
    },
  ],
  jsonSchema: {
    type: "object",
    properties: {
      event_id: { type: "string" },
      message: { type: "string" },
      timestamp: { type: "string", format: "date-time" },
    },
  },
  config: {
    loadStrategy: "full_load",
    type: "minio",
    bucket: "my-datalake",
    basePath: "analytics/events",
    endpoint: "http://localhost:9000",
    table: {
      catalog: "hive",
      schema: "analytics",
      tableName: "events",
    },
    trinoClient,
  },
})
```

## Pipeline steps

The pipeline executes these steps in order:

1. **Convert to Parquet** — Records are serialized to a Parquet buffer using the provided JSON Schema
2. **Upload to storage** — The Parquet file is uploaded to the configured bucket/path
3. **Manage Hive DDL** — The external table is dropped and recreated pointing to the new data location

If any step fails, the pipeline stops immediately and throws the error. There is no automatic rollback of earlier steps.

<Note>
  The `jsonSchema` property is typically generated by the CLI as
  `json-schema.json` in each endpoint directory. You don't need to write it by
  hand.
</Note>

navTitle: executeWritePipeline

keywords: pipeline executewritepipeline entry point persisting


docId: adapters/write-pipeline/load-strategies

slug: write-pipeline/load-strategies

title: Load Strategies

text: Available strategies

text: Partitioning interaction

content:
---
title: Load Strategies
navTitle: Load Strategies
description: How full_load, full_load_append, and append strategies control data storage and table management.
---

The `loadStrategy` configuration determines how new data is stored and how Hive tables are managed. Each strategy provides a different tradeoff between data freshness, history retention, and storage costs.

## Available strategies

### full_load

The default strategy. Replaces all existing data with the new records on every write.

**Steps:**

1. Delete all existing files at the base path
2. Upload new Parquet file to `<basePath>/latest.parquet/<uuid>.parquet`
3. Drop and recreate the Hive table pointing to the `latest.parquet/` directory

**Use when:** You always want the table to reflect the most recent payload. No history is kept.

**Table structure:**
| Table | Location |
|-------|----------|
| `<tableName>` | `s3a://<bucket>/<basePath>/latest.parquet/` |

### full_load_append

Combines `full_load` with historical archiving. The latest data is always queryable, and all historical writes are preserved in an append-only directory.

**Steps:**

1. Upload Parquet file to `<basePath>/latest.parquet/<uuid>.parquet` (replaces previous)
2. Upload the same Parquet file to `<basePath>/all.parquet/<partition_path>`
3. Drop and recreate both `_latest` and `_all` Hive tables

**Use when:** You need both a "current state" view and a historical log of all writes.

**Table structure:**
| Table | Location |
|-------|----------|
| `<tableName>_latest` | `s3a://<bucket>/<basePath>/latest.parquet/` |
| `<tableName>_all` | `s3a://<bucket>/<basePath>/all.parquet/` |

### append

Only appends data. No "latest" snapshot is maintained.

**Steps:**

1. Upload Parquet file to `<basePath>/all.parquet/<partition_path>`
2. Drop and recreate the Hive table pointing to the `all.parquet/` directory

**Use when:** You're building a log or event stream where every write adds to the history.

**Table structure:**
| Table | Location |
|-------|----------|
| `<tableName>` | `s3a://<bucket>/<basePath>/all.parquet/` |

## Partitioning interaction

For `full_load`, partitioning configuration is ignored — data is always written as a single file in `latest.parquet/`.

For `full_load_append` and `append`, the partition path within `all.parquet/` is determined by the [partitioning configuration](/docs/adapters/write-pipeline/partitioning).

<Note>
  The `full_load` strategy deletes the entire base path before uploading. Make
  sure you don't share the `basePath` between multiple endpoints.
</Note>

navTitle: Load Strategies

keywords: strategies fullload fullloadappend append control


docId: adapters/write-pipeline/partitioning

slug: write-pipeline/partitioning

title: Partitioning

text: Partitioning modes

text: Timestamp mode

text: Disabled mode

text: Field mode

text: Custom mode

text: Related types

content:
---
title: Partitioning
navTitle: Partitioning
description: How data is partitioned into Hive-style paths within the all.parquet/ directory.
---

Partitioning controls how data is organized within the `all.parquet/` directory for `full_load_append` and `append` strategies. It determines the subdirectory structure using Hive-style `key=value` paths.

<Note>
  Partitioning only applies to `full_load_append` and `append` strategies. For
  `full_load`, partitioning is ignored.
</Note>

## Partitioning modes

The `partitioning` config value determines the mode:

| Value                   | Mode      | Description                                               |
| ----------------------- | --------- | --------------------------------------------------------- |
| `true` (default)        | Timestamp | Partitions by write timestamp                             |
| `false`                 | Disabled  | Flat file layout (UUID-only paths)                        |
| `"fieldName"`           | Field     | Partitions by a record field's date value                 |
| `"field:component/..."` | Custom    | Multi-segment partitioning with date component extraction |

## Timestamp mode

When `partitioning: true` (the default), each write is partitioned by the current timestamp. The pipeline automatically injects `load_timestamp`, `load_timestamp_year`, and `load_timestamp_month` columns into the records and schema.

```
all.parquet/year=2025/month=01/day=15/<uuid>.parquet
```

The `partitioningFormat` controls granularity:

| Format             | Example path                               |
| ------------------ | ------------------------------------------ |
| `"year"`           | `year=2025/<uuid>.parquet`                 |
| `"year/month"`     | `year=2025/month=01/<uuid>.parquet`        |
| `"year/month/day"` | `year=2025/month=01/day=15/<uuid>.parquet` |

## Disabled mode

When `partitioning: false`, files are placed directly in `all.parquet/` with a flat UUID-based path:

```
all.parquet/<uuid>.parquet
```

## Field mode

When `partitioning` is a simple field name (e.g. `"event_date"`), the pipeline extracts the date value from each record's field and partitions accordingly:

```json
{ "partitioning": "event_date", "partitioningFormat": "year/month" }
```

For a record with `event_date: "2025-03-20"`:

```
all.parquet/year=2025/month=03/<uuid>.parquet
```

Records are grouped by their partition key — records with the same partition path are written to the same Parquet file.

<Warning>
  If a record is missing the partition field, has a null value, or contains an
  unparseable date, the pipeline throws a `PartitionFieldError`.
</Warning>

## Custom mode

Custom partitioning allows multi-segment paths with field extraction and date component parsing. The format uses `/` to separate segments and `:` to specify a date component:

```json
{ "partitioning": "region/event_date:year/event_date:month" }
```

For a record with `region: "eu-west-1"` and `event_date: "2025-03-20"`:

```
all.parquet/region=eu-west-1/year=2025/month=03/<uuid>.parquet
```

### Segment types

| Segment            | Description              | Example output     |
| ------------------ | ------------------------ | ------------------ |
| `fieldName`        | Raw field value          | `customer_id=acme` |
| `fieldName:year`   | Year from ISO date       | `year=2025`        |
| `fieldName:month`  | Month from ISO date      | `month=03`         |
| `fieldName:day`    | Day from ISO date        | `day=20`           |
| `fieldName:hour`   | Hour from ISO datetime   | `hour=14`          |
| `fieldName:minute` | Minute from ISO datetime | `minute=30`        |
| `fieldName:second` | Second from ISO datetime | `second=45`        |

## Related types

<InterfaceReference
  file="adapters/src/write-pipeline"
  name="ResolvedPartitioning"
  mode="declaration"
/>

<InlineReference
  file="adapters/src/write-pipeline"
  include={["PartitionMode", "PartitionSegment", "PartitionFieldError"]}
/>

navTitle: Partitioning

keywords: partitioning partitioned hive-style paths within