Uploading Data and AI Processing

Reading time: ~7 minutes | Level: Beginner Prerequisites: Walkthrough 2: SDCStudio Overview

What You'll Learn

• Which file formats SDCStudio accepts
• How to upload a file and monitor its progress
• What happens during Stage 1 (structural parsing) and Stage 2 (AI enhancement)
• How automatic string format inference works
• Tips for getting the best results from AI processing

Supported File Formats

SDCStudio currently accepts two file formats:

CSV (recommended for tabular data): - Must have a header row with column names - UTF-8 encoding required - Comma-separated values - Best for: spreadsheets, database exports, structured datasets

Markdown dataset template (for documentation-driven modeling): - An MD2PD dataset template: a Markdown file with YAML front matter declaring template_version: "4.0.0" - Can be written by hand, or produced from a PDF form with Form2SDCTemplate - Best for: specifications, data dictionaries, domain models

Other formats (JSON, XML, PDF, DOCX, Excel) are not accepted for upload. For Excel files, export to CSV first.

How to Upload

1. Open the user menu and click Upload Data; this opens the Data Sources page. Click Upload New Data (this is /app/upload)
2. Select Project
3. Choose the File Type: CSV, or Markdown dataset template
4. Choose a file, or drag it onto the drop zone (.csv or .md only, maximum 50 MB, recommended under 10 MB). The page checks the file content before it is sent
5. Click Upload File

Cost: 100 credits for each new component and each new cluster the upload creates, taken when processing completes; columns matched to existing components are free, and nothing is charged if processing fails.

Your file appears in the Data Sources list (/app/data-sources) with a status badge. The page does not auto-refresh, so reload it to see a new status.

The Two-Stage Pipeline

SDCStudio processes your file in two distinct stages. This design means that if AI processing fails, you can retry without re-uploading or re-parsing the file.

Stage 1: Structural Parsing

Duration: 30 seconds to 2 minutes Status flow: Uploaded → Parsing → Parsed

This stage is fast and deterministic, with no AI or LLM calls. The system:

1. Detects the file format (CSV or Markdown)
2. Reads the structure: headers, columns, row counts
3. Infers basic data types: integer, decimal, string, date, boolean
4. Detects string format patterns automatically (see below)
5. Creates a ParsedData record: a standardized representation stored in the database

For a CSV file, the output at this stage is a list of columns with detected types and sample values:

customer_id     → integer
first_name      → string
email           → string (email pattern detected)
signup_date     → date (ISO 8601)
status          → string (categorical: 2 unique values)
total_purchases → integer
price           → decimal

Automatic String Format Inference

During parsing, the system scans sample values in each string column and tests them against a pattern catalog, without using AI. Detected formats include:

• UUIDs
• MAC addresses
• US Social Security numbers
• IPv4 addresses
• Hex color codes
• ISO dates (YYYY-MM-DD) and US dates (MM/DD/YYYY)
• US ZIP codes (5-digit and ZIP+4)
• US phone numbers
• Email addresses
• URLs

When a pattern matches the sample values, the system pre-populates the column's String Format with an XML Schema-compatible regular expression. This saves manual configuration and gives the AI in Stage 2 a validation rule to build on.

Stage 2: AI Enhancement

Duration: 1 to 5 minutes Status flow: Parsed → Agent Processing → Modeling → Completed

This is where the intelligence happens. Several AI agents work together:

1. A profiler agent analyzes the dataset as a whole
2. A dispatcher agent routes each column to a specialist agent
3. Specialist agents (string, quantified, temporal) create an SDC4 component for each column:
4. Semantic analysis: understands what each column represents
5. Pattern recognition: refines format detection from Stage 1
6. Ontology matching: links to the ontologies you uploaded
7. Validation rules: suggests constraints (min/max, patterns, enumerations, required/optional)
8. Labels and descriptions: generates human-readable names and documentation
9. A synthesizer agent assembles the results into a data model with one Data cluster

The project's name and description, and the ontologies you have uploaded, are given to the agents as context. A column named mrn in a project described as hospital records is treated differently from the same column in a retail project. This is why uploading ontologies and describing the project before uploading data matters.

Status Reference

Status values as shown on the Data Sources page:

The badges show the labels of the API's status values (uploaded, parsing, parsed, agent_processing, modeling, completed, error, agent_error).

Tips for Best Results

Use descriptive column names: - Good: customer_email, order_total_amount, signup_date - Bad: col1, data, field_x

Keep data consistent: - Same date format throughout a column (don't mix 2024-01-15 with 1/15/24) - Same type per column (don't mix numbers and text)

Include enough rows: - Minimum 10 rows for reliable type inference - 20-100 rows is ideal for pattern detection - Include edge cases (min values, max values, empty cells)

Describe the project and upload ontologies first: - Give the project a description that says what domain the data comes from - Open Settings, then Preferences, then the Modeler Information section, and click Manage Ontologies (/app/ontologies) to upload your organization's vocabularies before your first data upload - SDCStudio ships no ontologies; the list shows the ones you upload, the ones your teammates shared at Team or Corporate level, and any uploaded as Public

Start small: - Upload a sample (50-100 rows) first - Validate the generated model - Upload the full dataset after you are satisfied

What to Do After Processing

Once the row shows Completed:

1. Click View Details on the row, then View Data Model, or open your project and find the model in its Data Models (n) list
2. Review the structure, types, and validation rules
3. Edit any component that needs refinement
4. When satisfied, publish and generate outputs

Summary

• SDCStudio accepts CSV and Markdown dataset template files
• Processing happens in two stages: fast structural parsing, then AI enhancement
• The AI creates a complete model with types, validation, labels, and semantic links
• Describing the project and uploading ontologies before data files improves AI quality
• If AI processing fails, click Retry Processing; no re-upload needed

Next walkthrough

Understanding Components and Data Types covers the SDC4 type system and how to interpret the components the AI created for your data.