Integration succeeds when contracts survive change
A pipeline is not successful because data moved once. It succeeds when producers and consumers agree on meaning, identity, schema, freshness, quality, access, failure handling, and ownership—and when those agreements continue to hold through change and recovery.
Begin with a data-flow inventory: producer, transport, transformation, store, consumer, accountable owner, data classification, contract version, service objective, and replay path. Then test the eight failure modes below.
1. Quality rules do not match the use case
“Clean data” is not a sufficient requirement. Define measurable rules for a named dataset and use: required fields, valid domains, uniqueness scope, referential integrity, timeliness, reconciliation tolerances, and known exceptions. W3C’s Data Quality Vocabulary deliberately avoids prescribing one universal list of dimensions; it supports publishing metrics and their provenance so consumers can judge fitness for use.
Acceptance tests: measure every rule on representative history and current loads; quarantine failures; record denominator, threshold, owner, and disposition; confirm that retries do not create duplicates; reconcile source and destination counts, totals, and sampled records.
2. The same fields mean different things
Matching names do not establish matching semantics. “Customer,” “revenue,” “active,” currency, time zone, effective date, and identifier scope can differ across domains. Preserve source meaning and provenance; document transformations; assign owners to definitions; and version mappings.
Use industry standards where they fit, but test conformance at the implementation level. For example, FHIR standardizes exchange structures for health information, while profiles, terminology bindings, extensions, version choice, and local rules still determine whether two systems interoperate.
Acceptance tests: domain owners approve definitions and examples; units and time zones are explicit; code systems are versioned; unknown values are distinguishable from false or zero; round-trip and cross-system reconciliation pass.
3. Schemas and interfaces change without coordination
Formats and protocols are only the surface. A producer can preserve valid JSON while changing a field’s meaning. Establish versioned data contracts covering schema, semantics, keys, nullability, compatibility, deprecation, ownership, and notice periods. Validate contracts in producer and consumer delivery pipelines.
Acceptance tests: backward/forward compatibility is tested for the declared policy; unknown fields and missing fields behave as documented; consumers receive deprecation notice; contract violations stop or quarantine data instead of silently coercing it.
4. Legacy constraints are hidden by a wrapper
An API or middleware layer can isolate a legacy interface, but it cannot remove source capacity, transaction, batch-window, security, licensing, or recovery limits. Document the system of record, supported read/write path, consistency model, maintenance window, and retirement plan. Treat screen scraping as an exceptional, reviewed bridge with monitoring and a replacement deadline.
Acceptance tests: load remains within source limits; reconciliation proves no missed or duplicated changes; authentication and authorization are enforced end to end; recovery from partial writes is tested; a rollback and decommission path exists.
5. Capacity and latency goals are vague
Measure volume, event size, burst, concurrency, transformation cost, state, retention, recovery backlog, and downstream capacity. Define service-level indicators such as p95 end-to-end freshness, error rate, backlog age, and recovery time. Choose vertical scaling, partitioning, parallelism, buffering, caching, or elastic capacity from evidence rather than slogans.
Acceptance tests: normal, peak, burst, downstream-slow, and recovery loads meet declared objectives; partition skew is measured; backpressure protects dependencies; caches have tested invalidation and authorization behavior.
6. Event delivery semantics are assumed
“Real time” should be replaced by an explicit freshness objective. Select batch, micro-batch, request/response, messaging, or streaming based on that objective and the cost of staleness. For change-data capture, record snapshot boundaries, source offsets, schema evolution, deletes, and replay behavior.
At-least-once systems can redeliver events after failures. Debezium documents this behavior and advises designs that tolerate duplicates; exactly-once behavior requires specific supported configurations and does not automatically extend through every external side effect.
Acceptance tests: kill and restart each component; replay a window; inject duplicates, late events, out-of-order events, and poison records; verify idempotency, ordering scope, dead-letter handling, reconciliation, and recovery.
7. Security and privacy stop at the connector
Map sensitive data across source, transport, staging, logs, backups, dead-letter queues, analytics stores, and support access. Apply least privilege, authentication, authorization, encryption, key management, audit logging, retention, deletion, and incident response according to risk. NIST distinguishes data integrity—the protection against unauthorized or accidental alteration or loss—from broader data-quality properties.
Acceptance tests: unauthorized reads and writes fail; credentials rotate; sensitive fields do not leak into logs; deletion and retention propagate to every in-scope copy; integrity monitoring detects unauthorized modification; recovery uses verified data.
8. Nobody owns the end-to-end outcome
Producer and consumer teams can each meet local targets while the integrated product fails. Assign an accountable owner for the end-to-end data product, plus named owners for sources, contracts, quality rules, access, incidents, and consumer communication. Governance should resolve decisions and fund fixes, not merely create meetings.
Acceptance tests: every alert has an owner and response objective; lineage reaches the originating source and affected consumers; a breaking-change drill reaches all owners; quality and reliability trends are reviewed against business outcomes.
A production-readiness checklist
| Control | Evidence | Decision |
|---|---|---|
| Data contract | Versioned schema, semantics, compatibility and owner | Pass / gap / not applicable |
| Quality | Metrics, thresholds, provenance, quarantine and reconciliation | Pass / gap / not applicable |
| Delivery | Freshness objective, ordering scope, retry and replay tests | Pass / gap / not applicable |
| Security/privacy | Flow map, classification, controls, retention and deletion tests | Pass / gap / not applicable |
| Reliability | Peak, dependency failure, recovery and backlog tests | Pass / gap / not applicable |
| Operations | Lineage, monitoring, runbook, owner, rollback and retirement plan | Pass / gap / not applicable |
The strongest integration architecture is not the one with the most products or the lowest nominal latency. It is the one whose contracts, controls, evidence, and ownership remain understandable when data and systems change.
Historical comments from Datanizant
No public comments on this article
No approved public comments were included in the WordPress export for this article.