Onboard Source Data and Measure Quality in EBX | Part 4

An import that ends with “success” may still have loaded unusable data. In an AI-ready program, onboarding produces two outputs: records and evidence about their fitness.

Audience, prerequisites and outcomes

For integration engineers, data stewards and quality leads. Complete Parts 1–3, prepare the lab dataset, and obtain import permission. The base platform includes CSV/XML import services; richer mapping behavior may depend on the separately licensed Data Exchange add-on.

You will preserve raw evidence, map source values, select an error strategy, calculate quality dimensions and build a remediation queue.

Version note: import dialogs and options differ between base EBX and Data Exchange add-on releases. Confirm options such as force import, null handling and invalid-record export in your installed documentation. Never enable force import against a trusted master zone merely to make a workshop pass.

Source files

customers-crm.csv:

source_id,name,email,country,loyalty_tier,updated_at
C-1007,Amina Rahman,amina@example.test,US,Gold,2026-09-01T12:00:00Z
C-1008,Jon Bell,jon.example.test,US,Silver,2024-01-01T00:00:00Z
C-1009,,lee@example.test,XX,Bronze,2026-09-02T08:00:00Z

customers-commerce.csv:

source_id,name,email,country,loyalty_tier,updated_at
WEB-882,Amina R.,AMINA@example.test,USA,,2026-09-03T15:00:00Z
WEB-883,Jonathan Bell,jon@example.test,United States,Silver,2026-09-03T16:00:00Z

These include invalid email syntax, a missing name, unknown country, stale data, duplicates and inconsistent representations.

A controlled onboarding sequence

Gate and register the batch

Action: work only in the lab landing dataset; confirm import permission and installed import capability. Compute a checksum outside EBX, count data rows excluding the header, and create an IngestionEvent containing exact filename, source, receipt time, row count, checksum, schema version, operator, and unique event ID. Expected result: one immutable batch record exists before data loading. Validate: recompute checksum/count and compare. Stop/recover: if either differs, quarantine the file and create no replacement event under the same ID.

Land and profile without altering evidence

Action: load source columns as received into landing, retaining source key and event ID; store normalized derivatives separately. Profile nulls, distinct counts, patterns, minimum/maximum lengths, duplicate source IDs, and reference coverage. Expected result: landing preserves USA, United States, malformed email, missing name, and XX exactly while the profile exposes them. Validate: compare every source row and checksum-derived count with landing. Stop/recover: if import coerces or truncates values, delete only the lab load after exporting its history, correct field widths/types, and replay with a new attempt record linked to the same event.

Map with versioned rules

Action: create a mapping version that maps headers, trims comparison names, normalizes email comparison case, and converts USA/United States to US while retaining original display/evidence values. Expected result: standardized rows contain both raw lineage and governed values. Validate: known mappings succeed and unknown XX remains an explicit exception rather than becoming a default. Stop/recover: if an unknown value is silently coerced, disable the mapping version and reprocess from immutable landing after correction.

Validate, route, and reconcile

Action: run model and business rules; write each failure to a remediation queue with rule ID/version, source, source ID, observed value, severity, owner, batch, and disposition. Route structurally valid accepted rows forward and keep blocking failures out of mastering. Expected result: malformed email, missing name, unknown country, stale row, and duplicate candidates have visible dispositions. Validate: accepted plus quarantined plus intentionally skipped counts equal the registered row count. Stop/recover: any count mismatch or untraceable row blocks the batch; preserve evidence and rerun from landing only after reconciliation.

Promote with an idempotency gate

Action: approve only records with no blocking issues, using event ID plus source key as the replay key. Record mapping/rule versions and promotion approval. Expected result: a retry of the same event updates no duplicate master record and yields the same quality counts. Validate: run a controlled replay and compare record identifiers, counts, and histories. Stop/recover: if replay duplicates or overwrites evidence, halt promotion, restore the lab snapshot, and correct the idempotency key before proceeding.

Hands-on import

Preflight the installed import service

Action: in the lab dataset, open the CSV import exposed under the installed perspective's Actions/Services menu. Confirm whether it is the base built-in service or Data Exchange and record the version. Inspect delimiter, encoding, header, update mode, null/empty, error, tracking, and invalid-row options without running. Expected result: a written import configuration matched to the installed documentation. Validate: another operator can reproduce the selections. Stop/recover: if an option's semantics are unclear, cancel the dialog and consult the exact installed-version documentation; do not experiment against master data.

Import the CRM landing file

Action: select customers-crm.csv, map every source header by name, set tracking to batch-20260904-01, and choose the preapproved structural-error stop behavior. For a new immutable landing load, do not use an update mode that merges blanks into prior records. Run the import. Expected result: three landing rows are associated with the batch and preserve original values. Validate: compare row count, source IDs, raw strings, history/tracking entry, and checksum evidence. Stop/recover: on wrong mapping, encoding, truncation, or partial unaccounted load, stop downstream processing, export the import log, remove only the isolated lab attempt, and retry with a new attempt identifier.

Capture validation outcomes

Action: execute validation on the standardized candidates, export or record invalid rows using the installed service's supported mechanism, and link each issue to the event. Expected result: jon.example.test, missing name, XX, and stale timestamp trigger their planned rules/severities. Validate: reconcile issue counts with the quality scorecard and confirm accepted/quarantined totals equal three. Stop/recover: if force import bypassed validation or invalid rows cannot be identified, do not promote; mark the attempt failed and replay safely from landing.

Import and reconcile commerce

Action: register a distinct commerce event/checksum, import customers-commerce.csv with explicit header mapping, and standardize country/email representations using the same versioned rules. Keep USA, United States, and original email casing in landing. Expected result: two commerce rows, normalized comparison values, and candidate relationships to CRM records without automatic identity claims. Validate: trace both source IDs to their events and review candidate matches. Stop/recover: if source evidence is overwritten or candidate matching auto-merges records, undo the lab promotion/merge using recorded history and route candidates to stewardship.

Quality scorecard

Calculate at record and batch level:

Report both percentage and count. “99% valid” conceals whether the remainder is one row or one million.

AI context: quality must follow the record

An embedding pipeline should not infer fitness from table location alone. Export quality_status, rule results, batch ID and effective timestamp. Prevent records with blocking issues from indexing. For warnings, decide whether the AI can retrieve them with an uncertainty label or must wait for review. When a record is remediated, invalidate the old embedding and retain the relationship between versions.

Exercise: diagnose before correcting

Exercise: diagnose, remediate, and remeasure

Action: run both registered synthetic batches and export a defect ledger with rule ID/version, batch, source, source ID, observed raw value, expected policy, severity, disposition, and owner. Expected result: every deliberately bad value appears once and every input row is accepted or accounted for. Validate: reconcile ledger and batch counts; investigate duplicates or omissions. Stop/recover: freeze promotion if the ledger cannot be reconciled to source rows.

Action: correct USA and United States in the versioned central mapping; send the missing-name defect to its source owner without editing landing; keep XX quarantined until an approved mapping/source correction exists. Reprocess from immutable landing under a recorded attempt. Expected result: mapping defects clear consistently while record defects retain ownership and evidence. Validate: compare rule-by-rule counts, numerators, denominators, and percentages before and after; confirm a replay does not duplicate records. Stop/recover: if the total improves by hiding, defaulting, or dropping failures, reject the remediation, restore the prior mapping version, and reopen the defects.

Validation checklist

• Not completed: Batch checksum, schema version and row count are recorded.
• Not completed: Raw source values remain available.
• Not completed: Mapping rules are versioned and repeatable.
• Not completed: Null/empty update semantics are explicitly selected.
• Not completed: Invalid rows have rule, severity, owner and disposition.
• Not completed: Quality is measured by dimension and count.
• Not completed: Blocking records cannot enter the AI index.
• Not completed: Reprocessing is idempotent for the same event ID.

Failure modes

Force-importing into master. Data Exchange documentation describes force import for architectures with landing/staging zones; it bypasses model validation, so using it in master defeats the control.

Silent coercion. Converting unknown values to a default produces clean-looking falsehoods.

Overwriting with blanks. A source-system omission is not necessarily a deletion instruction.

A single quality score. It lets high completeness hide broken identity uniqueness.

No replay key. Retrying a batch creates duplicates and irreproducible metrics.

Takeaway and next

Onboarding is a governed experiment: receive, profile, transform, validate, explain and only then promote. Part 5 uses the standardized records to detect candidate duplicates, apply survivorship and establish a traceable golden record.

Sources

• TIBCO EBX: Built-in CSV/XML import services
• TIBCO EBX Data Exchange 6.2.2: Import behavior
• TIBCO EBX 6.2.2: Constraints