> ## Documentation Index
> Fetch the complete documentation index at: https://datagen.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Data Overview

> Connect databases and files, build analytic models with SQL, and publish snapshots that agents and apps read

**Data** turns the tables in your databases, object storage and workspace files into
reusable **analytic models**. You connect a source once, write (or ask the AI Data
Builder to write) the SQL that shapes it, and publish the result. Agents and apps read
the published model, never the raw connection, so credentials stay in DataGen and every
consumer sees the same, documented numbers.

Open **Data** from the **More** menu in the navigation rail (pin it to keep it on the
rail). The page has three tabs: **Datasets**, **Connections** and **Settings**.

## The lifecycle

```text theme={null}
Connection ──> Source ──> Analytic model ──> Publish ──> Version + snapshot ──> Agents / Apps
                                                 ^                |
                                                 └── Refresh ─────┘
```

<Steps>
  <Step title="Connect">
    An admin saves a read-only **connection** to PostgreSQL, MySQL / MariaDB, Oracle
    or Parquet files on S3. DataGen checks that it can reach the database before it
    saves. See [Connect a data source](/data/connect).
  </Step>

  <Step title="Register sources">
    Pick the tables to expose with **Manage tables**. Each selected table becomes a
    **source** with a stable SQL name. A CSV, TSV or Parquet file from **Files** can also
    be a source, and on Oracle a saved query can be one.
  </Step>

  <Step title="Build a model">
    An **analytic model** is a SQL query over sources and other models. Write it yourself
    or describe what you need to the **Data Builder** agent, preview the result, and
    document the columns. See [Build analytic models](/data/build-models).
  </Step>

  <Step title="Publish">
    **Publish dataset** freezes the SQL as a new **version**. A materialized model is
    built into a Parquet **snapshot** at publish time.
  </Step>

  <Step title="Refresh and roll back">
    Refresh reruns the published SQL by hand or on a schedule and stores a new snapshot.
    Every published version stays in the history, and you can roll back to one.
  </Step>

  <Step title="Use">
    Connect the published model to an agent. The agent queries its snapshot with the
    `datagen-data` CLI in its sandbox. See [Use data in agents and apps](/data/use).
  </Step>
</Steps>

## Key concepts

| Concept | What it is |
| - | - |
| **Connection** | Saved host, database and credentials for one database or S3 bucket. Credentials are encrypted and never sent to the browser or to a sandbox. |
| **Source** | A table (or file, or Oracle query) registered from a connection. Sources are read live: a query reads the current rows. |
| **Analytic model** | A SQL definition over sources and other models, with a name, description, column descriptions and a guide (`model.md`). |
| **Mode** | **Analytic SQL** runs DuckDB SQL across any sources. **Native SQL** runs one database's own SQL unchanged on that connection. |
| **Execution mode** | **Live query** runs the SQL each time the model is read. **Materialized** stores the result as a snapshot. Agents can only read materialized models. |
| **Version** | An immutable copy of the model definition, created by each publish. The **Live** version is the one consumers read. |
| **Snapshot** | The Parquet result of building a materialized version. Refreshing creates a new snapshot of the same version. |

## SQL names

Every dataset has a SQL name that stays the same when you rename it. Use **Copy SQL
reference** in a dataset's actions menu to copy it.

| Dataset | SQL name |
| - | - |
| Database table | `data.<connection alias>.<schema>.<table>` |
| Oracle query source | `data.<connection alias>.query.<name>` |
| Workspace file | `data.files.raw.<file name>` or `data.files.outputs.<file name>` |
| Analytic model, Parquet source | `data.<id>`, for example `data.dataset_3f2a...` |

## Who can do what

| Action | Role |
| - | - |
| Add, edit and delete connections; manage tables; add file and query sources; change Data Builder settings | Admin |
| Create, edit, publish and roll back analytic models; edit descriptions; set refresh schedules | Admin, Builder |
| Preview sources and published models; refresh published data | Admin, Builder, Operator |
| Browse datasets and published models | Every member |

Unpublished drafts are visible to Admins and Builders only.

## Limits

| Limit | Value |
| - | - |
| Rows in one materialized model snapshot | 100,000 |
| Snapshot storage per workspace | 512 MiB |
| Rows shown in a source or result preview | 500 (50 per page) |
| Query time in the SQL editor | 30 seconds |
| Workspace file read as a source | 512 MiB by default |
| Tables added in one save | 100 |

<Note>
  Data is the governed path: admins hold the credentials and agents read published
  snapshots only. When an agent needs to run its own queries against a live database,
  give it a read-only credential through a Connection Vault instead. See
  [Query a database from code](/data/query-from-code).
</Note>

## Next

<CardGroup cols={2}>
  <Card title="Connect a data source" icon="plug" href="/data/connect">
    Add a PostgreSQL, MySQL, Oracle or Parquet connection and choose its tables.
  </Card>

  <Card title="Build analytic models" icon="hammer" href="/data/build-models">
    Write or generate model SQL, preview it, publish it and keep it fresh.
  </Card>
</CardGroup>


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.