Storage Operations

Storage operations provide the low-level interface for interacting with S3 or S3-compatible object storage. The write pipeline uses these internally, but they are exported for advanced use cases.

createStorageOperations #

Creates a storage operations instance backed by files-sdk. The adapter selection is based on config.type:

• "s3" — reads credentials from AWS_* environment variables
• "minio" — reads credentials from MINIO_* environment variables

1
2
3
4
5
6
7
8
import { createStorageOperations } from "@lakeql/adapters"

const storage = createStorageOperations({
  type: "minio",
  bucket: "my-datalake",
  endpoint: "http://localhost:9000",
})

Configuration #

Operations #

The returned object provides these methods:

Environment variables #

S3 #

MinIO #

Error handling #

Storage errors are wrapped in a StorageError with a descriptive message including the operation and path:

1
2
3
4
5
6
7
8
9
10
11
import { StorageError } from "@lakeql/adapters"

try {
  await storage.upload(buffer, "path/to/file.parquet")
} catch (error) {
  if (error instanceof StorageError) {
    console.error(error.message)
    // "Storage upload failed for "path/to/file.parquet": The specified bucket does not exist"
  }
}