AI Processing Guide

Overview

When you upload a file, SDCStudio uses a language model to decide what each column means, and deterministic code to turn those decisions into components of a data model. This guide explains which model is used, what it is shown, what it decides for each column, and what the pipeline never does. The upload steps, statuses and costs are in Uploading Data.

The model and its inputs

All language-model calls go to Google Vertex AI. The generation model is Gemini, gemini-3.7-flash unless the operator sets another through the VERTEX_AI_GENERATION_MODEL environment variable, run at low temperature so the same input tends to produce the same answer. Embeddings come from text-embedding-004. There is no other provider in the production configuration.

For each upload the model is shown:

  • the column names and, for CSV, the first five values of each column;
  • for CSV, the project name and description, labeled in the prompt as a strong hint about the domain that the model should still verify against the data;
  • your Preferred Ontologies from Settings, labeled as a weak prior to be used only if the data supports it. Standard namespaces such as rdf, rdfs, owl, skos and dc carry no domain signal and are left out;
  • for a Markdown template, the descriptions and types you wrote in the template.

The project description is the largest lever you control: one that names the domain, the source and the intended use lets the model resolve ambiguous headers, and a misleading one steers every column the wrong way.

Stage 1: reading the file

CSV

The csv2pd parser reads the file and keeps the first five values of each column as samples. It then runs four agents, each a Python class whose name ends in Agent:

  1. ProfilerAgent sees every column name and a few samples and returns a dataset description and a domain classification.
  2. DispatcherAgent routes each column to one of three specialists: quantified, temporal or string. If the model's answer cannot be parsed, rules route by whether the samples are numeric and by words in the header.
  3. QuantifiedTypeAgent, TemporalAgent and StringTypeAgent run in parallel, one call per column. Each returns a description, a type, business rules and semantic keywords. Inside each specialist the type is chosen by embeddings: the column name and samples are compared against the descriptions of the SDC4 types and the closest wins, re-run against meaning when the model's description is substantive, which is how an eight-digit date column becomes XdTemporal instead of XdCount. Where the model returned no keywords, a further call asks for three to five ontology search terms.
  4. SynthesizerAgent returns confidence scores, then places all columns in one flat cluster named after the dataset.

What reaches the second stage from a CSV is each column's type, description, confidence, semantic keywords and business rules, and, for a quantified column, its unit (since 4.17.12). The unit is the one the language model reads from the column's name and samples (weight_kg gives kg); when the model is not available, a fallback reads the name word by word (price and unit_cost are USD, age is years, completion_pct is %, order_count is transactions; page_views is not an age and lab_value is not money). No unit, or an answer such as "N/A", leaves the placeholder (below). Value ranges and distinct values are not carried, by design: the specialists see the first five values of each column, and a reference range or an allowed-value list made from five values would reject valid data. Declare them in a Markdown template, or set them in the editor.

Every step that calls the model has a deterministic fallback, so a malformed or empty response, including one blocked by Vertex AI's content filters, degrades the description rather than stopping the run.

Markdown templates

The md2pd parser needs no model to read a template: types, descriptions, enumerations, reuse references and the cluster hierarchy come from what you wrote, as defined in Template Format. Enrichment runs afterward unless the front matter sets enrichment.enable_llm: false. It records a suggestion when the embedding match disagrees with a declared type, generates semantic keywords for columns that have none, and, for a description shorter than 20 characters, appends the type in parentheses, Weight (XdQuantity), without a model call. It never changes a type you provided.

Stage 2: building the model

The second task hands the parsed description to DataModelAgent, which delegates the data cluster to ClusterAgent, which runs one component agent per column. There is one agent per SDC4 type, named for it: XdStringAgent, XdCountAgent, XdTemporalAgent and so on, through the list types and UnitsAgent. For templates that declare them, DataModelAgent also builds the model-level parts: subject and provider parties, participations, audit, attestation, protocol, a workflow cluster, access control and links.

Most of this stage is ordinary code, not model calls. For each column the component agent:

  • Reuses before creating. A template reference of the form @Project:Label resolves to the existing component whatever its type, and an existing ct_id resolves within the declared type. Otherwise a component in the same project with the same label and type is reused. If one with the same label but a different type exists, the new component gets the suffix (duplicate label).
  • Assembles the description from the model's description, the business rules and the relationships.
  • Sets the type's constraints. XdString: a regular expression inferred from the samples when at least three share a pattern, from a catalog covering UUID, MAC address, SSN, IPv4, hex color, ISO and US dates, US ZIP, US phone, email and URL, or a fixed-width pattern derived position by position, plus length bounds from the samples or defaults keyed on words in the label such as email, phone, name or code. XdTemporal: the allowed forms, date, time, datetime, duration, year and month, inferred from the samples and the column name, defaulting to date and datetime. XdBoolean: the true and false value lists from the template, otherwise defaults chosen by words such as compliant, present or active. XdToken: the enumeration values from the template's enumeration table. XdLink: a default relation of isRelatedTo.
  • Attaches units to the quantified types. A unit named by the template is looked up in your project, then in the Default library, then among public units elsewhere, and created only if none match. From a CSV, the unit implied by the column name is looked up the same way. A column with no stated or implied unit gets a new Units component labeled <column>_units, described as count.
  • Creates reference ranges on quantified types from a template range written as min-max, >n, <n, >=n or <=n (with an optional name: prefix). A range written as a two-value list is not converted yet; add that reference range in the editor.
  • Links the component to ontologies. For the non-quantified types a fixed set of type-level predicates and documentation links is attached first, without a model call. Then two sources are searched: the Fuseki triplestore, which holds the ontologies loaded into SDCStudio including the ones you upload, weighted toward your Preferred Ontologies; and the language model, asked for five candidate IRIs given the column's description and samples. Model-proposed IRIs are checked over HTTP with a ten-second timeout and dropped if they do not resolve. The candidates are scored against the column's context and filtered for quality, and the first surviving candidate is stored as the component's semantic link. A column whose linking finds nothing still gets its component, without external links.

When all columns are done the cluster is created, the components are attached to it, and the data model is created with the cluster as its data section and the template's dataset metadata where present. A final pass checks the model's components, reused ones included, for missing length constraints, boolean values, numeric bounds, token descriptions, link relations, ordinal values and semantic links. Its findings go to the server log, not the app, so those are the fields to review first in the draft.

What the pipeline does not do

  • It does not read your data rows into the model. The first five values of each CSV column are used for inference; they are kept with the parsed record of the upload in the database, and the parse cache is deleted.
  • It does not change a component you named. A component referenced by @Project:Label or ct_id is used as it is. A component matched by label and type in the same project is reused too, and the run may add or replace its semantic link and add reference ranges.
  • An upload does not publish or share anything. The data model and every new component are created in draft, and every new component is private until you make it public. (An assembly through the API publishes and generates when it asks to; see the assembly reference.)
  • It does not nest clusters from a CSV. Hierarchy comes only from a Markdown template.
  • It does not charge for a failed run or for a reused component, whether the column named it by ReuseComponent reference or ct_id or it was matched by label, project and type during the run. The charge is for the components the run created, never more than the estimate the balance check quoted.
  • It does not learn from previous uploads or from your edits. Each run starts from the file, the project description and your Preferred Ontologies.

Getting better results

Write the project description before you upload. Use column headers a stranger could read; the header is the first ontology search term and the main input to type selection. Keep each column to one format, since a mixed column is typed from five values that may not show the mix. Name units in the header, weight_kg rather than weight, so the model reads the column right and names the Units component for you; to be sure of a unit, declare it in a Markdown template or set it in the editor. Set Preferred Ontologies in Settings once; they travel with every upload. When the draft is wrong, correct it in the editor rather than re-uploading.