From Big Data to AI Data Systems: A Retrospective

A decade of changing data questions

When I began writing about big data in 2013, the practical questions were dominated by distributed storage, MapReduce, Hadoop ecosystem tools, and how to process datasets that no longer fit comfortably on one machine. The archive's “four Vs”—volume, velocity, variety, and veracity—were a useful way to organize that moment.

The questions did not disappear. They became more specific: Which events need streaming? Which transformations belong in batch? What must be retained at row level? Where is lineage recorded? Who owns a data product? Which data may legally and safely train or ground a model? How do we test freshness, quality, representativeness, and failure behavior?

The archive, organized by learning path

1. Distributed-data foundations

• What's So Big About Big Data? — an early Hadoop and MapReduce exercise
• Installing Hadoop, Hive, and HBase — a historical setup guide that requires a current-version warning
• Scala and Spark for Data Processing — a bridge from MapReduce-era workflows to Spark

2. Databases, search, and analytical serving

• Introduction to MongoDB
• Apache Pinot series summary
• Apache Druid series summary

3. Event streaming

• Advanced Kafka configurations and integrations
• Kafka monitoring, streams, and cloud deployment
• Kafka Streams and production monitoring
• Kafka security and multi-cluster architecture

4. Architecture and ownership

• Data fabric and data mesh
• Cloud computing
• Data science with R and Python

What changed—and what did not

Layer	Earlier emphasis	Current engineering question
Storage	Distributed file systems and local clusters	Object/file/table layout, transactions, retention, locality, sovereignty, and total cost
Processing	Batch MapReduce jobs	Batch, incremental, and streaming semantics selected per workload
Serving	General-purpose databases and search	Purpose-fit operational, search, vector, graph, and real-time OLAP stores
Ownership	Central platform teams	Explicit domain accountability balanced with interoperable platform controls
Metadata	Documentation after delivery	Lineage, contracts, quality signals, policy, and discoverability as operational data
AI	Downstream model training	Training, evaluation, retrieval, features, feedback, permissions, and monitoring across the lifecycle

Data is necessary for AI, but volume is not the objective

AI systems depend on data, but “more” is not a quality criterion. A defensible data program records origin and permitted use; tests relevance, coverage and representativeness; controls sensitive data; detects leakage and contamination; versions transformations; and preserves evaluation sets appropriate to the deployment context.

Streaming is equally contextual. Apache Kafka describes event streaming as capture, durable storage, processing, and routing of event streams. Apache Spark's Structured Streaming incrementally processes an unbounded input table. Those capabilities can support features, inference, monitoring, and some learning workflows, but they do not by themselves make a decision accurate, fair, safe, or timely.

A modern architecture is a set of contracts

1. Source contract: schema, meaning, owner, rights, expected volume, and change policy.
2. Processing contract: batch/stream semantics, event time, late data, retries, idempotency, and correctness tests.
3. Serving contract: freshness, latency, concurrency, retention, and permitted query patterns.
4. AI-use contract: allowed training/retrieval uses, versioned datasets, evaluation coverage, and human oversight.
5. Evidence contract: lineage, quality measurements, incidents, approvals, and reproducible releases.

The real story

The big-data era was not replaced by the AI era. Its distributed-systems lessons became part of the AI stack. The useful shift is from celebrating scale to engineering evidence: the right data, under the right controls, delivered with known semantics to a measured workload.