Large-scale search is an information-retrieval system, not simply Hadoop plus a search box. It ingests governed sources, analyzes text, builds indexes, retrieves candidates, ranks results, enforces permissions, and operates under freshness, latency, availability, and cost objectives.

Core pipeline

  1. Acquire: authorize sources; track provenance, deletion, and change events.
  2. Parse and enrich: extract fields, language, entities, and permissions while retaining failures.
  3. Analyze: apply versioned tokenization, normalization, stop-word, synonym, and stemming rules by language/domain.
  4. Index: build inverted and other purpose-fit structures; define shards, replicas, refresh, and retention.
  5. Retrieve and rank: combine lexical, structured, vector, or learned ranking only when validated.
  6. Serve and observe: enforce access at query time and monitor quality, freshness, latency, errors, and capacity.

Evaluate relevance

Build representative queries and graded judgments with documented sampling and rater guidance. Use precision, recall, MRR, MAP, or NDCG only where their definitions match the task. Pair offline measures with task success, reformulation, abandonment, latency, and severe-failure review. Clicks are biased behavioral signals, not ground truth.

See learning to rank.

Scale deliberately

Shard design depends on corpus, fields, routing, query mix, update rate, and failure tolerance. More shards can increase coordination and resource cost. Benchmark representative indexing and queries under expected concurrency; test node loss, rebalance, recovery, schema changes, and rolling upgrades.

Protect content

Apply data-access governance. Indexes can expose secrets and personal data even when sources are protected. Enforce document/field permissions, deletion propagation, encryption, audit, injection-safe query handling, and tenant isolation.

Monitor ingestion and freshness with pipeline monitoring. Hadoop-era batch concepts remain historical context in the MapReduce project, not a universal modern architecture.

Reviewed and substantially updated September 4, 2026. Original publication date preserved.