Apache Pinot with Iceberg: Hot and Historical Analytics

Version note: Rewritten September 4, 2026. Verify Pinot, Iceberg, catalog, storage, compute-engine, and connector versions before implementation.

Apache Pinot and Apache Iceberg solve different problems. Pinot serves low-latency analytical queries from Pinot segments. Iceberg is a table format whose deployment also needs a catalog, storage, and compatible compute engines. Pinot deep store holds Pinot segments for distribution and recovery; it does not turn them into an Iceberg table.

Use an authoritative source pattern

Do not treat querying Pinot after retention as a default archive method. Pinot ingestion may transform, aggregate, deduplicate, or omit fields. Instead, retain the authoritative event stream or batch source and write independently to Pinot for hot serving and Iceberg for governed history. Where dual writing is unavoidable, define delivery, replay, idempotency, late-data, schema, and failure semantics.

Define each system boundary

Layer	Responsibility
Authoritative source	Durable event or batch record, provenance, replay, and contract
Pinot	Hot indexed serving, ingestion retention, segment lifecycle, low-latency queries
Iceberg	Historical table metadata, snapshots, schema/partition evolution, analytical access
Catalog/storage	Table discovery, metadata concurrency, object/file durability, identity and keys
Federated engine	Explicit catalogs/schemas, join/union plan, access controls, resource limits

Manage retention correctly

Pinot retention removes eligible segments according to configured policy; coordinate it with source durability and reconciliation. Iceberg snapshot expiration removes old table versions and files no longer referenced by retained snapshots. It is not a row-level “delete everything older than 90 days” policy. Row retention requires an engine-supported delete/rewrite plus snapshot and file maintenance planned around legal holds and recovery.

Use documented Iceberg maintenance procedures: compact data files, rewrite manifests where justified, expire snapshots deliberately, and delete orphan files only with safe retention intervals and exclusive knowledge of writers.

Query across hot and historical ranges

Configure the federation engine with explicit Pinot and Iceberg catalogs, schemas, identities, time zones, and resource controls. Define one cutoff convention such as hot timestamps greater than or equal to the boundary and history strictly below it so no row is lost or duplicated. Reconcile counts, sums, keys, and late corrections around the boundary.

Production readiness

1. Capacity-test Pinot controllers, brokers, servers, stream ingestion, ZooKeeper, storage, and queries.
2. Make metadata/catalog and object storage highly available and protected.
3. Configure authentication, TLS, least privilege, secrets, logs, backup, and restore.
4. Test replay, partial write, schema change, late data, cutoff movement, connector outage, and rollback.
5. Monitor ingestion lag, segment health, table maintenance, reconciliation, query cost, and freshness.

Review the Pinot series summary, learn federation concepts in Presto basics, and compare systems in Druid versus Pinot.