Key Concepts

This page introduces the concepts the rest of the documentation assumes. Each one links to the section that covers it in depth.

Data Lakehouse #

LakeQL is a universal data access layer. It reaches a query engine through an adapter, so the engine underneath is a choice, not a fixture. A data lakehouse is one common deployment. It combines open storage with structured query capabilities and has three parts:

• Storage — data lives in object storage (S3, MinIO, GCS, Azure, and the like) in open formats such as Parquet.
• Metastore — a catalog service tracks table metadata, partitions, and file locations.
• Query engine — a SQL engine provides access across catalogs and schemas. Adapters for Trino and Microsoft Fabric are available today. ClickHouse and DuckDB are on the roadmap.

LakeQL sits on top of the query engine and exposes the data through GraphQL, with no separate hand-written application layer.

Data Sources and Adapters #

A data source is a named connection to a query engine, declared in lakeql.config.ts. An adapter teaches LakeQL how to talk to a particular engine. It owns connection handling, the SQL dialect, authentication forwarding, and response decoding.

Register the adapters a project may use, then declare named data sources that each pick one:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20import { defineConfig } from "@lakeql/config"
import { trinoAdapterDefinition } from "@lakeql/adapter-trino"
import { fabricAdapterDefinition } from "@lakeql/adapter-fabric"

export default defineConfig({
  sourcePath: "src",
  adapters: [trinoAdapterDefinition, fabricAdapterDefinition],
  dataSources: {
    warehouse: {
      adapter: "trino",
      connection: {
        host: "trino.internal",
        port: 8080,
        catalog: "hive",
        auth: { type: "basic", username: "svc", password: "..." },
      },
    },
  },
})

A single project can register multiple adapters and route each named data source to a different engine. See Data Sources for adapter details.

Schema Introspection and Generation #

LakeQL discovers table structures by querying the data source through its adapter. Run the CLI introspection command, and it reads each table's columns, types, and nullability, parses complex type strings (such as array(row(...))) into structured definitions, and writes typed files per table:

mutation-schema.ts is generated only when the endpoint defines a mutation. Its presence is what makes the API register a Mutation root type.

Generated files are organized by data source, catalog, and schema (here warehouse is the configured data-source name), so tables from different data sources never collide. Commit them to your repository and regenerate them when a table changes. See the CLI commands.

The Runtime #

LakeQL separates configuration from the live objects that use it. Loading the config never opens a connection on its own. An application composes a runtime explicitly:

1
2
3
4
5
6
7
8
9
10
import { createLakeQLRuntime, loadConfig } from "@lakeql/config"

const project = await loadConfig()

const runtime = await createLakeQLRuntime({
  config: project.config,
  adapters: project.adapters,
  resolveAuth,
})

The runtime owns the data-source instances and their read and write strategies, and exposes a close() method for graceful shutdown. Nothing is global, so an application can create more than one runtime. That helps with reloads and tests. See Runtime Lifecycle.

GraphQL-to-SQL Translation #

The API server composes a GraphQL schema from the generated endpoint files. When a query arrives, LakeQL derives the SQL from the GraphQL resolve info. It reads the selected fields, filters, sorting, and pagination, then runs the query through the data source's adapter.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
query {
  orders(
    filter: { status: { eq: "shipped" } }
    sorting: [{ field: id, direction: ASC }]
    paging: { page: 1, perPage: 10 }
  ) {
    nodes {
      id
      status
      total
    }
    pageInfo {
      hasNext
      currentPage
    }
  }
}

The query builder constructs the field selection, the WHERE clause, ordering, and pagination. The adapter then renders them in its own SQL dialect. See the Schema Builder and query-builder.

Authentication and Permissions #

Before any query runs, the API server validates an incoming token, resolves the authenticated user, and checks per-table read and write permissions. A data source can also forward the caller's identity to the engine. Trino user impersonation is one example. See Authentication.

Where to Next #