Data Overview

Connect databases and files, build analytic models with SQL, and publish snapshots that agents and apps read

Data turns the tables in your databases, object storage and workspace files into reusable analytic models. You connect a source once, write (or ask the AI Data Builder to write) the SQL that shapes it, and publish the result. Agents and apps read the published model, never the raw connection, so credentials stay in DataGen and every consumer sees the same, documented numbers.

Open Data from the More menu in the navigation rail (pin it to keep it on the rail). The page has three tabs: Datasets, Connections and Settings.

The lifecycle

Connection ──> Source ──> Analytic model ──> Publish ──> Version + snapshot ──> Agents / Apps
                                                 ^                |
                                                 └── Refresh ─────┘

1. Connect

   An admin saves a read-only connection to PostgreSQL, MySQL / MariaDB, Oracle or Parquet files on S3. DataGen checks that it can reach the database before it saves. See Connect a data source.

2. Register sources

   Pick the tables to expose with Manage tables. Each selected table becomes a source with a stable SQL name. A CSV, TSV or Parquet file from Files can also be a source, and on Oracle a saved query can be one.

3. Build a model

   An analytic model is a SQL query over sources and other models. Write it yourself or describe what you need to the Data Builder agent, preview the result, and document the columns. See Build analytic models.

4. Publish

   Publish dataset freezes the SQL as a new version. A materialized model is built into a Parquet snapshot at publish time.

5. Refresh and roll back

   Refresh reruns the published SQL by hand or on a schedule and stores a new snapshot. Every published version stays in the history, and you can roll back to one.

6. Use

   Connect the published model to an agent. The agent queries its snapshot with the datagen-data CLI in its sandbox. See Use data in agents and apps.

Key concepts

Concept	What it is
Connection	Saved host, database and credentials for one database or S3 bucket. Credentials are encrypted and never sent to the browser or to a sandbox.
Source	A table (or file, or Oracle query) registered from a connection. Sources are read live: a query reads the current rows.
Analytic model	A SQL definition over sources and other models, with a name, description, column descriptions and a guide (model.md).
Mode	Analytic SQL runs DuckDB SQL across any sources. Native SQL runs one database’s own SQL unchanged on that connection.
Execution mode	Live query runs the SQL each time the model is read. Materialized stores the result as a snapshot. Agents can only read materialized models.
Version	An immutable copy of the model definition, created by each publish. The Live version is the one consumers read.
Snapshot	The Parquet result of building a materialized version. Refreshing creates a new snapshot of the same version.

SQL names

Every dataset has a SQL name that stays the same when you rename it. Use Copy SQL reference in a dataset’s actions menu to copy it.

Dataset	SQL name
Database table	data.<connection alias>.<schema>.<table>
Oracle query source	data.<connection alias>.query.<name>
Workspace file	data.files.raw.<file name> or data.files.outputs.<file name>
Analytic model, Parquet source	data.<id>, for example data.dataset_3f2a...

Who can do what

Action	Role
Add, edit and delete connections; manage tables; add file and query sources; change Data Builder settings	Admin
Create, edit, publish and roll back analytic models; edit descriptions; set refresh schedules	Admin, Builder
Preview sources and published models; refresh published data	Admin, Builder, Operator
Browse datasets and published models	Every member

Unpublished drafts are visible to Admins and Builders only.

Limits

Limit	Value
Rows in one materialized model snapshot	100,000
Snapshot storage per workspace	512 MiB
Rows shown in a source or result preview	500 (50 per page)
Query time in the SQL editor	30 seconds
Workspace file read as a source	512 MiB by default
Tables added in one save	100

Data is the governed path: admins hold the credentials and agents read published snapshots only. When an agent needs to run its own queries against a live database, give it a read-only credential through a Connection Vault instead. See Query a database from code.

Next

Connect a data source

Add a PostgreSQL, MySQL, Oracle or Parquet connection and choose its tables.

Build analytic models

Write or generate model SQL, preview it, publish it and keep it fresh.