Mutations

Working with Mutations

The Mutation Pipeline

Enabling Mutations

Load Strategies

Mutation Configuration Reference

Partitioning

Storage Configuration

Generated Resolver

Input Validation

Write Permission Model

System User Impersonation

Error Handling

Custom Mutations

---
title: Mutations
description: Enable write operations with the mutation pipeline, configure load strategies, and use system user impersonation.
---

## Working with Mutations

LakeQL provides a complete mutation pipeline that handles data persistence end-to-end. When you configure mutation support in your endpoint definition, the generated resolver automatically handles validation, Parquet conversion, S3 upload, and Hive table management — no manual stub implementation needed.

## The Mutation Pipeline

When a mutation is executed, the generated resolver runs through this pipeline:

1. **Input arrives** via GraphQL mutation
2. **Validation** — Input is checked against a generated Zod schema (if field validations are configured)
3. **Parquet conversion** — Records are serialized into Parquet format via `@lakeql/parquet`
4. **S3 upload** — The Parquet file is uploaded to the configured base path
5. **Hive DDL** — The external table is recreated to point at the new data

If any step fails, the pipeline stops immediately and propagates the error to the GraphQL error layer.

## Enabling Mutations

Add a `mutation` configuration to your endpoint definition:

```json path="my-endpoint.json"
{
  "version": "1.0",
  "tableName": "user_events",
  "catalog": "hive",
  "schema": "analytics",
  "fields": [
    {
      "name": "email",
      "type": "String",
      "options": {
        "required": true,
        "validations": [{ "type": "email" }]
      }
    },
    {
      "name": "age",
      "type": "Integer",
      "options": {
        "required": false,
        "validations": [
          { "type": "min", "value": 0 },
          { "type": "max", "value": 150 }
        ]
      }
    },
    { "name": "event_type", "type": "String" },
    { "name": "timestamp", "type": "DateTime" }
  ],
  "mutation": {
    "loadStrategy": "full_load",
    "type": "s3",
    "bucket": "my-datalake",
    "basePath": "warehouse/analytics/user_events",
    "region": "eu-central-1",
    "endpoint": "https://s3.eu-central-1.amazonaws.com"
  }
}
```

Then generate the endpoint:

```bash
lakeql-cli create-endpoint --from-file ./my-endpoint.json
```

This produces a `mutation-schema.ts` with a fully wired resolver that invokes the write pipeline, and a `validations.ts` with the Zod schema for input validation.

## Load Strategies

The `loadStrategy` field controls how data is persisted. Choose based on your use case:

| Strategy           | Best for                           | Behavior                                     |
| ------------------ | ---------------------------------- | -------------------------------------------- |
| `full_load`        | Dimension tables, reference data   | Replaces all data on every write             |
| `full_load_append` | Datasets needing current + history | Maintains latest snapshot and historical log |
| `append`           | Event streams, audit logs          | Only adds data, never replaces               |

See the [Load Strategies](/lakeql/guides/load-strategies) guide for detailed explanations and file layout diagrams.

## Mutation Configuration Reference

<InterfaceReference
  file="schema-generator/src/endpoint-schema"
  name="MutationConfig"
/>

## Partitioning

The `partitioning` and `partitioningFormat` fields control how files are organized in storage. These settings apply to `full_load_append` and `append` strategies only — `full_load` ignores partitioning entirely.

### Write-timestamp partitioning (default)

When `partitioning` is `true` (the default), files are partitioned by the current UTC timestamp. A `load_timestamp` field is added to the endpoint definition with `readOnly: true` — it appears in the query schema and Hive DDL but is excluded from the mutation input. The write pipeline populates it automatically at runtime. Additionally, `load_timestamp_year` (Integer) and `load_timestamp_month` (Integer) fields are injected as materialized partition columns, enabling direct filtering in tools that read Parquet files without Hive metastore awareness (e.g. Jupyter notebooks with PyArrow, Pandas, or DuckDB).

```json
{
  "fields": [
    { "name": "event_type", "type": "String" },
    {
      "name": "load_timestamp",
      "type": "DateTime",
      "options": { "readOnly": true }
    },
    {
      "name": "load_timestamp_year",
      "type": "Integer",
      "options": { "readOnly": true }
    },
    {
      "name": "load_timestamp_month",
      "type": "Integer",
      "options": { "readOnly": true }
    }
  ],
  "mutation": {
    "loadStrategy": "append",
    "type": "s3",
    "bucket": "my-datalake",
    "basePath": "warehouse/raw/events",
    "partitioning": true,
    "partitioningFormat": "year/month/day"
  }
}
```

<Note>
  The Endpoint Builder adds the `load_timestamp`, `load_timestamp_year`, and
  `load_timestamp_month` fields automatically when timestamp partitioning is
  active. Fields with `readOnly: true` are queryable via GraphQL but cannot be
  provided as mutation input.
</Note>

### Disabled partitioning (flat layout)

When `partitioning` is `false`, files are written to a flat directory without date-based directories. No `load_timestamp` field is added.

```json
{
  "mutation": {
    "loadStrategy": "append",
    "type": "s3",
    "bucket": "my-datalake",
    "basePath": "warehouse/raw/events",
    "partitioning": false
  }
}
```

File layout with flat partitioning:

```
<basePath>/
  all.parquet/
    <uuid-1>.parquet
    <uuid-2>.parquet
```

### Field-based partitioning

When `partitioning` is set to a field name string, records are grouped by the value of that date/datetime field and written to separate partition paths. No `load_timestamp` field is added.

```json
{
  "mutation": {
    "loadStrategy": "append",
    "type": "s3",
    "bucket": "my-datalake",
    "basePath": "warehouse/raw/events",
    "partitioning": "event_date",
    "partitioningFormat": "year/month"
  }
}
```

The field must exist in every record and contain a valid ISO 8601 date or datetime value. Field names must start with a letter or underscore, contain only alphanumeric characters and underscores, and be 1–64 characters long.

### Custom partition format

For advanced use cases you can define a custom partition format string that combines multiple fields and date components. The format uses `/` to separate path segments and `:` to extract date components from a field.

```json
{
  "mutation": {
    "loadStrategy": "append",
    "type": "s3",
    "bucket": "my-datalake",
    "basePath": "warehouse/raw/events",
    "partitioning": "customer_id/event_date:year/event_date:month"
  }
}
```

This produces paths like:

```
all.parquet/
  customer_id=42/year=2024/month=06/<uuid>.parquet
  customer_id=99/year=2024/month=06/<uuid>.parquet
```

#### Format syntax

Each segment is either a plain field name or a field with a date component:

| Segment            | Behavior                                                  |
| ------------------ | --------------------------------------------------------- |
| `customer_id`      | Extracts the raw field value → `customer_id=<value>`      |
| `event_date:year`  | Parses the field as ISO date, extracts year → `year=2024` |
| `event_date:month` | Extracts month (zero-padded) → `month=06`                 |
| `event_date:day`   | Extracts day (zero-padded) → `day=15`                     |
| `ts:hour`          | Extracts hour (zero-padded) → `hour=08`                   |
| `ts:minute`        | Extracts minute (zero-padded) → `minute=30`               |
| `ts:second`        | Extracts second (zero-padded) → `second=07`               |

Supported date components: `year`, `month`, `day`, `hour`, `minute`, `second`.

Records are grouped by their resolved partition key — records with the same key are written to the same Parquet file.

When using a custom format, the `partitioningFormat` field is ignored (the format is fully determined by the custom string). No `load_timestamp` field is added.

<Note>
  Fields referenced in the custom format must exist in every record. Date
  component segments require valid ISO 8601 date or datetime values. A
  `PartitionFieldError` is thrown if any record has a missing, null, or
  unparseable field value.
</Note>

<Note>
  The `partitioning` and `partitioningFormat` fields are not included in the
  generated config when `loadStrategy` is `full_load`, since full-load always
  overwrites the entire dataset as a single file.
</Note>

## Storage Configuration

The `type` field in the mutation configuration selects the storage adapter used by the write pipeline.

### S3 (default)

S3 is the default storage backend. Credentials and region are read from standard AWS environment variables:

| Environment Variable    | Description                   |
| ----------------------- | ----------------------------- |
| `AWS_ACCESS_KEY_ID`     | AWS access key                |
| `AWS_SECRET_ACCESS_KEY` | AWS secret key                |
| `AWS_DEFAULT_REGION`    | Default region (fallback)     |
| `AWS_ENDPOINT_URL`      | Custom S3 endpoint (fallback) |

```json
{
  "mutation": {
    "loadStrategy": "full_load",
    "type": "s3",
    "bucket": "my-datalake",
    "basePath": "warehouse/analytics/user_events"
  }
}
```

### MinIO (local development)

MinIO is supported for local development and self-hosted S3-compatible storage. The `endpoint` field is required when using MinIO. Credentials are read from MinIO-specific environment variables:

| Environment Variable      | Description                 |
| ------------------------- | --------------------------- |
| `MINIO_ACCESS_KEY_ID`     | MinIO access key            |
| `MINIO_SECRET_ACCESS_KEY` | MinIO secret key            |
| `MINIO_ENDPOINT`          | MinIO server URL (fallback) |

```json
{
  "mutation": {
    "loadStrategy": "full_load",
    "type": "minio",
    "bucket": "local-datalake",
    "basePath": "warehouse/analytics/user_events",
    "endpoint": "http://localhost:9000"
  }
}
```

<Note>
  When `type` is `"minio"`, the `endpoint` field is required either in the
  configuration or via the `MINIO_ENDPOINT` environment variable.
</Note>

## Generated Resolver

The generated `mutation-schema.ts` integrates with `@lakeql/adapters` to execute the full write pipeline. Here's what the generated code looks like conceptually:

```ts path="schemas/custom/hive/analytics/user_events/mutation-schema.ts"
import { builder } from "@lakeql/api"
import { executeWritePipeline } from "@lakeql/adapters"
import { validationSchema } from "./validations"
import jsonSchema from "./json-schema.json"

// GraphQL input type generated from field definitions
const UserEventsInput = builder.inputType("UserEventsInput", {
  fields: (t) => ({
    email: t.string({ required: true }),
    age: t.int({ required: false }),
    event_type: t.string({ required: false }),
    timestamp: t.string({ required: false }),
  }),
})

builder.mutationField("writeUserEvents", (t) =>
  t.field({
    type: "Boolean",
    args: {
      input: t.arg({ type: [UserEventsInput], required: true }),
    },
    resolve: async (_root, args, context) => {
      // 1. Validate input against Zod schema
      const validated = validationSchema.parse(args.input)

      // 2. Execute the write pipeline
      await executeWritePipeline({
        records: validated,
        jsonSchema,
        config: {
          loadStrategy: "full_load",
          basePath: "warehouse/analytics/user_events",
          s3: context.s3Config,
          table: {
            catalog: "hive",
            schema: "analytics",
            tableName: "user_events",
          },
        },
      })

      return true
    },
  })
)
```

<Note>
  You don't write this code manually — it's generated by `create-endpoint`. The
  example above shows what the generated resolver does under the hood.
</Note>

## Input Validation

When fields have `options.validations` configured, a `validations.ts` file is generated with a Zod schema. The mutation resolver validates input before invoking the pipeline:

- If validation passes, the pipeline proceeds
- If validation fails, Zod errors are returned as GraphQL field errors without any data being written

See the [create-endpoint](/cli/commands/create-endpoint) reference for the full list of available validations.

## Write Permission Model

LakeQL's write permission model is **default-deny**. Users must have explicit `Mutation` rules to perform any write operation.

This design is intentional:

- Write operations are more sensitive than reads
- Writes often execute via a shared system user in Trino
- Each table typically has a defined data owner (source system)
- Explicit rules enforce data ownership boundaries

### Configuring Write Permissions

```ts
import { defineConfig } from "@lakeql/api"
import { allConfigs } from "./config-registry"

export default defineConfig({
  allConfigs,
  permissions: [
    {
      name: "ingestion-service",
      useSystemUser: true,
      permissions: {
        Query: [{ catalog: "hive", schema: "raw", tables: ["*"] }],
        Mutation: [
          {
            catalog: "hive",
            schema: "raw",
            tables: ["events", "user_actions"],
          },
        ],
      },
    },
    {
      name: "admin-user",
      useSystemUser: false,
      permissions: {
        Query: [{ catalog: "hive", schema: "*", tables: ["*"] }],
        Mutation: [{ catalog: "hive", schema: "config", tables: ["*"] }],
      },
    },
  ],
})
```

### Permission Resolution

For write operations, the permission check follows this logic:

1. **No authenticated user** → Denied
2. **No `Mutation` rules for this user** → Denied
3. **Rules exist but don't match catalog/schema/table** → Denied
4. **Matching rule found** → Allowed

## System User Impersonation

Write statements in Trino often need to execute as a system user with broad permissions, even when the request originates from a regular user.

When `useSystemUser: true` is set for a permission entry, the Trino client uses system credentials for that user's write operations instead of their own identity. This means:

- Trino sees the system user as the executor
- The application layer controls which tables a given client can write to
- Audit trails should track both the requesting user and the executing identity

```ts
{
  name: "etl-pipeline",
  useSystemUser: true,  // Execute writes as system user in Trino
  permissions: {
    Query: [],
    Mutation: [
      { catalog: "hive", schema: "staging", tables: ["*"] }
    ]
  }
}
```

<Note>
  When `useSystemUser` is `false`, the write is executed with the authenticated
  user's Trino credentials. This requires that user to exist in Trino with
  appropriate table privileges.
</Note>

## Error Handling

The mutation pipeline uses fail-fast error propagation:

| Failure point      | Behavior                                                        |
| ------------------ | --------------------------------------------------------------- |
| Zod validation     | Returns field errors to GraphQL, no data written                |
| Parquet conversion | Error propagated, no S3 or DDL operations attempted             |
| S3 upload          | Error propagated, no DDL operations attempted                   |
| Hive DDL           | Error propagated (with rollback attempt for `full_load_append`) |

For `full_load_append`, if one of the two Hive table creations fails, the system attempts a best-effort rollback of both tables before propagating the original error.

<Warning>
  The pipeline does not provide transactional guarantees across S3 and Hive. If
  the DDL step fails after a successful S3 upload, the Parquet file remains in
  S3. Re-running the mutation will overwrite it on the next successful
  execution.
</Warning>

## Custom Mutations

For use cases not covered by the pipeline (e.g., custom SQL, cross-table operations), you can still create manual mutation resolvers. See the [Custom Resolvers](/lakeql/guides/custom-resolvers) guide.

mutations mutation write pipeline strategies


Key Concepts

Data Lakehouse

GraphQL-over-Trino

Schema Introspection

Code Generation

---
title: Key Concepts
description: Core concepts behind LakeQL — data lakehouses, GraphQL-over-Trino, schema introspection, and code generation.
---

## Data Lakehouse

A data lakehouse combines the flexibility of data lakes with the structured query capabilities of data warehouses. In a typical LakeQL setup:

- **Storage** — Data lives in object storage (S3, MinIO) in open formats like Parquet or ORC
- **Metastore** — Apache Hive Metastore tracks table metadata (schemas, partitions, locations)
- **Query Engine** — Trino provides SQL access across all catalogs and schemas

LakeQL sits on top of this stack, exposing lakehouse data through a GraphQL API without requiring you to build a separate application layer.

## GraphQL-over-Trino

LakeQL translates incoming GraphQL queries into Trino SQL. This translation uses Kysely as a type-safe query builder:

```ts
// A GraphQL query like this:
query {
  orders(filter: { status: { eq: "shipped" } }, paging: { limit: 10 }) {
    nodes { id, status, total }
    pageInfo { hasNext, currentPage }
  }
}

// Becomes a Trino SQL query like this:
WITH total_count AS (
  SELECT COUNT(*) AS total_records
  FROM hive.sales.orders
  WHERE status = 'shipped'
),
records AS (
  SELECT id, status, total
  FROM hive.sales.orders
  WHERE status = 'shipped'
  ORDER BY id ASC
  FETCH FIRST 10 ROWS ONLY
)
SELECT * FROM total_count FULL JOIN records ON TRUE
```

The query builder handles field selection, WHERE clause generation, pagination (FETCH/OFFSET), and sorting — all derived from the GraphQL resolve info and input arguments.

## Schema Introspection

LakeQL discovers table structures by querying Trino metadata. When you run `lakeql-cli pull`, the CLI:

1. Connects to your Trino instance via the REST API
2. Executes `SHOW COLUMNS FROM catalog.schema.table` for each table
3. Parses column type strings (including complex types like `array(row(...))`)
4. Produces structured column definitions with names, types, and nullability

This means your GraphQL schema always reflects the actual state of your lakehouse tables.

## Code Generation

From introspected metadata, LakeQL generates four artifacts per table:

| File                 | Purpose                                                           |
| -------------------- | ----------------------------------------------------------------- |
| `config.ts`          | Table metadata — catalog, schema, table name, column mappings     |
| `interface.ts`       | TypeScript interface matching the table's column types            |
| `query-schema.ts`    | Pothos query schema with filtering, sorting, and pagination       |
| `mutation-schema.ts` | Pothos mutation schema with input types and resolver stub         |
| `json-schema.json`   | JSON Schema used by the response transformer at runtime           |
| `endpoint.json`      | Endpoint definition for re-generation via CLI or Endpoint Builder |

These files are committed to your repository and imported by the API server at startup. When your table schema changes, re-run `pull` to regenerate them.

```bash
# Generated file structure after pulling the "orders" table
src/schemas/generated/
└── sales/
    └── orders/
        ├── config.ts
        ├── interface.ts
        ├── query-schema.ts
        ├── mutation-schema.ts
        ├── json-schema.json
        └── endpoint.json
```

concepts graphql-over-trino schema introspection generation


Overview

What is LakeQL?

The Problem

The Solution

Key Benefits

How It Works

---
title: Overview
description: LakeQL is a toolkit for generating type-safe GraphQL APIs on top of Trino data lakehouses.
---

## What is LakeQL?

LakeQL is a toolkit that automatically generates a type-safe GraphQL API on top of your Trino data lakehouse. It introspects your Trino schemas, generates TypeScript interfaces and query schemas, and serves them through a production-ready GraphQL API with built-in pagination, filtering, and sorting.

## The Problem

Manually writing GraphQL resolvers for data lakehouse tables is repetitive and error-prone. Every time a table changes, you update interfaces, resolvers, and type definitions by hand. Schema drift, missing fields, and inconsistent filtering logic creep in fast — especially when your lakehouse has dozens of tables across multiple schemas.

## The Solution

LakeQL flips the workflow. Instead of writing resolvers manually, you introspect your Trino catalog and generate everything:

- **TypeScript interfaces** from column metadata
- **Pothos query schemas** with filtering, sorting, and pagination baked in
- **JSON schemas** for response transformation
- **A config registry** that ties it all together at runtime

The generated code is fully typed end-to-end, from GraphQL input to SQL output.

Pull a schema from Trino and generate all artifacts:

<Command variant="exec">lakeql-cli pull --catalog hive --schema sales</Command>

## Key Benefits

- **Type safety** — Generated TypeScript interfaces ensure compile-time correctness from GraphQL layer to SQL queries
- **Automatic schema generation** — Run one CLI command to generate resolvers, types, and configs for entire schemas
- **Pothos-based schema builder** — Leverage the Pothos GraphQL schema builder for type-safe, declarative schema construction
- **Built-in pagination** — Connection-based pagination with `pageInfo` metadata out of the box
- **Flexible filtering** — AND/OR combinable filters with operators like `eq`, `neq`, `like`, `in`, `gt`, `lt`, and more
- **Sorting** — Multi-field sorting with ASC/DESC direction
- **Permission model** — Read/write permission checks per table with support for user-level and system-user flows
- **Kysely SQL generation** — SQL queries are built with Kysely, giving you type-safe query construction without raw strings

## How It Works

```mermaid preview
flowchart LR
    trino["Trino Catalog"] -->|introspect| cli["LakeQL CLI<br/>(pull)"]
    cli -->|generate| ts["Generated TypeScript"]
    ts -->|load| api["LakeQL API<br/>(serve)"]
    api --> gql["GraphQL Endpoint"]
```

1. **Introspect** — The CLI connects to Trino and discovers table structures
2. **Generate** — TypeScript files are written with interfaces, query schemas, and configs
3. **Serve** — The API server loads generated schemas and exposes a GraphQL endpoint
4. **Query** — Clients send GraphQL queries that are translated into optimized Trino SQL

lakeql overview toolkit generating type-safe


Package Map

Packages

Dependency Flow

---
title: Package Map
description: All packages in the LakeQL monorepo and how they relate to each other.
---

## Packages

LakeQL is organized as a monorepo with focused, single-responsibility packages.

### @lakeql/api

GraphQL API server built on Hono, GraphQL Yoga, and Pothos. Handles HTTP serving, schema loading, authentication, and permission checks.

### @lakeql/cli

Command-line tool for schema introspection and code generation. Connects to Trino, discovers tables, and writes TypeScript source files.

### @lakeql/query-builder

Kysely-based SQL query generation. Translates GraphQL resolve info (selected fields, filters, sorting, pagination) into Trino-compatible SQL with CTEs.

### @lakeql/trino-client

HTTP client for the Trino REST API. Handles statement submission, polling for results, and authentication (Basic and Bearer).

### @lakeql/schema-generator

Generates GraphQL model definitions, Hive table DDL, and JSON schemas from parsed column metadata.

### @lakeql/column-parser

Parses Trino column type strings (e.g. `varchar`, `array(row(id bigint, name varchar))`) into structured type objects.

### @lakeql/response-transformer

Transforms Trino's array-based responses into typed JavaScript objects using JSON Schema definitions.

### @lakeql/file-generator

Generates TypeScript source files — config, interfaces, and query schemas — from schema-generator output.

### @lakeql/helpers

Shared utilities for pagination calculation, code formatting, and object manipulation.

### @lakeql/logger

Structured logging with loglayer and winston. Includes sensitive field redaction.

### @lakeql/create-app

Project scaffolding tool (`@lakeql/create-app`). Downloads a starter template via giget and configures dependencies.

## Dependency Flow

The packages connect through two main flows:

### CLI Flow (Introspection + Generation)

```mermaid preview
flowchart LR
    cli["@lakeql/cli"] --> tc["@lakeql/trino-client"]
    tc --> cp["@lakeql/column-parser"]
    cp --> sg["@lakeql/schema-generator"]
    sg --> fg["@lakeql/file-generator"]
    cli --> fg
```

### API Flow (Serving + Querying)

```mermaid preview
flowchart LR
    api["@lakeql/api"] --> qb["@lakeql/query-builder"]
    qb --> tc["@lakeql/trino-client"]
    tc --> rt["@lakeql/response-transformer"]
    api --> rt
```

### Shared Packages

Both flows use these shared packages:

- `@lakeql/helpers` — Pagination math, formatting utilities
- `@lakeql/logger` — Structured logging with redaction