What this series covers

Apache Druid is an analytics database designed for fast queries over event-oriented data. A realistic solution still spans several boundaries: sources, ingestion, Druid storage and query services, a visualization client, optional external machine-learning services, and production security and operations.

This page is a guide to the series, not evidence that a production system or benchmark has been completed.

  1. Apache Druid basics and setup — architecture, local evaluation, ingestion, and a first query.
  2. Druid and Pinot comparison — workload definition, datasource design, and measurable requirements.
  3. Performance and ingestion tuning — segments, partitioning, filters, query plans, and capacity tests.
  4. Machine-learning integration — exporting or querying features and operating a separate model workflow.
  5. Dashboards and visualization — connecting supported SQL clients and measuring freshness and latency.
  6. Production security and verification — authentication, authorization, TLS, least privilege, and acceptance tests.

Reference architecture

  1. Sources: transaction or activity events arrive through Kafka; historical files reside in durable object storage.
  2. Ingestion: a Kafka supervisor manages streaming ingestion tasks. Batch data uses a native index_parallel task or SQL INSERT/REPLACE, depending on the workload.
  3. Druid: datasources store time-partitioned segments and expose Druid SQL and native queries.
  4. Consumption: applications and dashboards query the Broker/Router through authenticated, authorized, encrypted endpoints.
  5. Optional ML: a separately operated service trains and evaluates models from governed data, then writes scored events or serves predictions through its own interface.

Minimal Kafka supervisor shape

The following is a structural starting point, not a production-ready secret or capacity configuration:

{
  "type": "kafka",
  "spec": {
    "dataSchema": {
      "dataSource": "ecommerce_sales",
      "timestampSpec": {"column": "timestamp", "format": "iso"},
      "dimensionsSpec": {
        "dimensions": ["order_id", "product_id", "category"]
      },
      "metricsSpec": [
        {"type": "doubleSum", "name": "revenue", "fieldName": "amount"},
        {"type": "longSum", "name": "units", "fieldName": "quantity"}
      ]
    },
    "ioConfig": {
      "type": "kafka",
      "consumerProperties": {"bootstrap.servers": "kafka.example.internal:9092"},
      "topic": "sales_stream",
      "inputFormat": {"type": "json"},
      "useEarliestOffset": false
    },
    "tuningConfig": {"type": "kafka"}
  }
}

Do not place credentials directly in the supervisor spec. Druid's security guidance recommends a dynamic configuration provider for sensitive Kafka properties. Choose the first-run offset behavior deliberately and test replay, late data, handoff, and failure recovery.

Queries and dashboards

Druid supports SQL and native queries. Druid SQL is parsed and planned by Apache Calcite and translated into native queries. Use EXPLAIN PLAN FOR, request logging, and workload-level measurements when tuning. Filter the primary __time column so the planner can restrict intervals.

SELECT
  TIME_FLOOR(__time, 'PT1H') AS hour,
  category,
  SUM(revenue) AS revenue
FROM ecommerce_sales
WHERE __time >= CURRENT_TIMESTAMP - INTERVAL '24' HOUR
GROUP BY 1, 2
ORDER BY 1, 2;

Define dashboard freshness separately from query latency. Measure source-to-queryable delay, query p50/p95/p99, concurrency, error rate, and dashboard render time. Do not promise sub-second behavior without a reproducible workload, data volume, cluster shape, and acceptance test.

Machine learning is an external responsibility

Druid can provide time-windowed aggregates or event features, but training, validation, serving, drift detection, and model governance belong to the ML system around it. For forecasting or anomaly detection, document the feature query, leakage controls, train/test split, baseline, error metric, alert threshold, false-positive cost, model version, and write-back or serving path.

An anomaly score is not proof of fraud. Route consequential alerts through a documented review and response process.

Production security baseline

Druid's security features are disabled by default. A production deployment must explicitly configure them.

  • Run Druid as an unprivileged operating-system user.
  • Enable authentication and authorization; do not expose the web console without authorization.
  • Use least-privilege READ and WRITE permissions on the required Druid resource types.
  • Enable TLS for client-to-cluster and service-to-service traffic.
  • Restrict network access, ideally through an API gateway or equivalent control.
  • Keep passwords and Kafka secrets out of plain-text specifications.
  • Disable JavaScript unless a reviewed requirement justifies it.

If using druid-basic-security, load the extension and configure its authenticator, escalator, and authorizer in runtime properties. Manage users, roles, and permissions through the documented security APIs; do not invent a standalone auth_config.json schema.

Acceptance checklist

  • Ingestion: schema, rejected rows, duplicates, late events, replay, and supervisor recovery are tested.
  • Correctness: aggregates reconcile with a source-of-record sample; approximate query behavior is explicitly accepted or disabled where supported.
  • Performance: freshness and latency targets are measured under expected volume and concurrency.
  • ML: predictions are evaluated against a baseline and monitored independently of Druid availability.
  • Security: unauthenticated requests fail, roles are tested positively and negatively, TLS is verified, and secrets are absent from specs and logs.
  • Operations: deep storage, metadata storage, backups, alerting, capacity, and recovery procedures are documented and exercised.

Primary documentation