> ## Documentation Index
> Fetch the complete documentation index at: https://private-7c7dfe99-mintlify-8a08bda2.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Monitoring Kafka metrics with ClickStack

> Monitoring Kafka Metrics with ClickStack

export const TrackedLink = ({href, eventName, children, ...rest}) => {
  const handleClick = () => {
    try {
      if (typeof window !== "undefined" && window.galaxy && eventName) {
        window.galaxy.track(eventName, {
          interaction: "click"
        });
      }
    } catch (e) {}
  };
  return <a href={href} onClick={handleClick} {...rest}>
      {children}
    </a>;
};

export const Image = ({img, alt, size}) => {
  return <Frame>
      <img src={img} alt={alt} />
    </Frame>;
};

<Info>
  **TL;DR**

  Monitor Apache Kafka performance metrics in ClickStack using the OTel JMX Metric Gatherer. Includes a demo dataset and pre-built dashboard.
</Info>

<h2 id="existing-kafka">
  Integration with an existing Kafka deployment
</h2>

Monitor your existing Kafka deployment by running the OpenTelemetry JMX Metric Gatherer container to collect metrics and send them to ClickStack via OTLP.

If you want to test this integration first without modifying your existing setup, skip to the [demo dataset section](#demo-dataset).

<h5 id="prerequisites">
  Prerequisites
</h5>

* ClickStack instance running
* Existing Kafka installation (version 2.0 or newer) with JMX enabled
* Network access between ClickStack and Kafka (JMX port 9999, Kafka port 9092)
* OpenTelemetry JMX Metric Gatherer JAR (download instructions below)

<Steps>
  <Step>
    <h4 id="get-api-key">
      Get ClickStack API key
    </h4>

    The JMX Metric Gatherer sends data to ClickStack's OTLP endpoint, which requires authentication.

    1. Open HyperDX at your ClickStack URL (e.g., [http://localhost:8080](http://localhost:8080))
    2. Create an account or log in if needed
    3. Navigate to **Team Settings → API Keys**
    4. Copy your **Ingestion API Key**

    <Image img="https://mintcdn.com/private-7c7dfe99-mintlify-8a08bda2/vQMVv9Ng6hRPWQ0d/images/clickstack/api-key.png?fit=max&auto=format&n=vQMVv9Ng6hRPWQ0d&q=85&s=d3451b04e25288d1df17aa2eb35670b7" alt="ClickStack API Key" width="3810" height="1924" data-path="images/clickstack/api-key.png" />

    5. Set it as an environment variable:

    ```bash theme={null}
    export CLICKSTACK_API_KEY=your-api-key-here
    ```
  </Step>

  <Step>
    <h4 id="download-jmx">
      Download the OpenTelemetry JMX Metric Gatherer
    </h4>

    Download the JMX Metric Gatherer JAR:

    ```bash theme={null}
    curl -L -o opentelemetry-jmx-metrics.jar \
      https://github.com/open-telemetry/opentelemetry-java-contrib/releases/download/v1.32.0/opentelemetry-jmx-metrics.jar
    ```
  </Step>

  <Step>
    <h4 id="verify-jmx">
      Verify Kafka JMX is enabled
    </h4>

    Ensure JMX is enabled on your Kafka brokers. For Docker deployments:

    ```yaml theme={null}
    services:
      kafka:
        image: confluentinc/cp-kafka:latest
        environment:
          JMX_PORT: 9999
          KAFKA_JMX_HOSTNAME: kafka
          # ... other Kafka configuration
        ports:
          - "9092:9092"
          - "9999:9999"
    ```

    For non-Docker deployments, set these in your Kafka startup:

    ```bash theme={null}
    export JMX_PORT=9999
    ```

    Verify JMX is accessible:

    ```bash theme={null}
    netstat -an | grep 9999
    ```
  </Step>

  <Step>
    <h4 id="deploy-jmx">
      Deploy JMX Metric Gatherer with Docker Compose
    </h4>

    This example shows a complete setup with Kafka, the JMX Metric Gatherer, and ClickStack. Adjust service names and endpoints to match your existing deployment:

    ```yaml theme={null}
    services:
      clickstack:
        image: clickhouse/clickstack-all-in-one:latest
        ports:
          - "8080:8080"
          - "4317:4317"
          - "4318:4318"
        networks:
          - monitoring

      kafka:
        image: confluentinc/cp-kafka:latest
        hostname: kafka
        container_name: kafka
        environment:
          KAFKA_NODE_ID: 1
          KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT'
          KAFKA_ADVERTISED_LISTENERS: 'PLAINTEXT://kafka:9092'
          KAFKA_PROCESS_ROLES: 'broker,controller'
          KAFKA_CONTROLLER_QUORUM_VOTERS: '1@kafka:29093'
          KAFKA_LISTENERS: 'PLAINTEXT://kafka:9092,CONTROLLER://kafka:29093'
          KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
          KAFKA_LOG_DIRS: '/tmp/kraft-combined-logs'
          KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
          KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
          KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
          CLUSTER_ID: 'MkU3OEVBNTcwNTJENDM2Qk'
          JMX_PORT: 9999
          KAFKA_JMX_HOSTNAME: kafka
          KAFKA_JMX_OPTS: '-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=kafka -Dcom.sun.management.jmxremote.rmi.port=9999'
        ports:
          - "9092:9092"
          - "9999:9999"
        networks:
          - monitoring

      kafka-jmx-exporter:
        image: eclipse-temurin:11-jre
        depends_on:
          - kafka
          - clickstack
        environment:
          - CLICKSTACK_API_KEY=${CLICKSTACK_API_KEY}
        volumes:
          - ./opentelemetry-jmx-metrics.jar:/app/opentelemetry-jmx-metrics.jar
        command: >
          sh -c "java
          -Dotel.jmx.service.url=service:jmx:rmi:///jndi/rmi://kafka:9999/jmxrmi
          -Dotel.jmx.target.system=kafka
          -Dotel.metrics.exporter=otlp
          -Dotel.exporter.otlp.protocol=http/protobuf
          -Dotel.exporter.otlp.endpoint=http://clickstack:4318
          -Dotel.exporter.otlp.headers=authorization=\${CLICKSTACK_API_KEY}
          -Dotel.resource.attributes=service.name=kafka,kafka.broker.id=broker-0
          -Dotel.jmx.interval.milliseconds=10000
          -jar /app/opentelemetry-jmx-metrics.jar"
        networks:
          - monitoring

    networks:
      monitoring:
        driver: bridge
    ```

    **Key configuration parameters:**

    * `service:jmx:rmi:///jndi/rmi://kafka:9999/jmxrmi` - JMX connection URL (use your Kafka hostname)
    * `otel.jmx.target.system=kafka` - Enables Kafka-specific metrics
    * `http://clickstack:4318` - OTLP HTTP endpoint (use your ClickStack hostname)
    * `authorization=\${CLICKSTACK_API_KEY}` - API key for authentication (required)
    * `service.name=kafka,kafka.broker.id=broker-0` - Resource attributes for filtering
    * `10000` - Collection interval in milliseconds (10 seconds)
  </Step>

  <Step>
    <h4 id="verify-metrics">
      Verify metrics in HyperDX
    </h4>

    Log into HyperDX and confirm metrics are flowing:

    1. Navigate to the Chart Explorer
    2. Search for `kafka.message.count` or `kafka.partition.count`
    3. Metrics should appear at 10-second intervals

    **Key metrics to verify:**

    * `kafka.message.count` - Total messages processed
    * `kafka.partition.count` - Total partitions
    * `kafka.partition.under_replicated` - Should be 0 in a healthy cluster
    * `kafka.network.io` - Network throughput
    * `kafka.request.time.*` - Request latency percentiles

    To generate activity and populate more metrics:

    ```bash theme={null}
    # Create a test topic
    docker exec kafka bash -c "unset JMX_PORT && kafka-topics --create --topic test-topic --bootstrap-server kafka:9092 --partitions 3 --replication-factor 1"

    # Send test messages
    echo -e "Message 1\nMessage 2\nMessage 3" | docker exec -i kafka bash -c "unset JMX_PORT && kafka-console-producer --topic test-topic --bootstrap-server kafka:9092"
    ```

    <Note>
      When running Kafka client commands (kafka-topics, kafka-console-producer, etc.) from within the Kafka container, prefix with `unset JMX_PORT &&` to prevent JMX port conflicts.
    </Note>
  </Step>
</Steps>

<h2 id="demo-dataset">
  Demo dataset
</h2>

For users who want to test the Kafka Metrics integration before configuring their production systems, we provide a pre-generated dataset with realistic Kafka metrics patterns.

<Steps>
  <Step>
    <h4 id="download-sample">
      Download the sample metrics dataset
    </h4>

    Download the pre-generated metrics files (29 hours of Kafka metrics with realistic patterns):

    ```bash theme={null}
    # Download gauge metrics (partition counts, queue sizes, latencies, consumer lag)
    curl -O https://datasets-documentation.s3.eu-west-3.amazonaws.com/clickstack-integrations/kafka/kafka-metrics-gauge.csv

    # Download sum metrics (message rates, byte rates, request counts)
    curl -O https://datasets-documentation.s3.eu-west-3.amazonaws.com/clickstack-integrations/kafka/kafka-metrics-sum.csv
    ```

    The dataset includes realistic patterns for a single-broker e-commerce Kafka cluster:

    * **06:00-08:00: Morning surge** - Sharp traffic ramp from overnight baseline
    * **10:00-10:15: Flash sale** - Dramatic spike to 3.5x normal traffic
    * **11:30: Deployment event** - 12x consumer lag spike with under-replicated partitions
    * **14:00-15:30: Peak shopping** - Sustained high traffic at 2.8x baseline
    * **17:00-17:30: After-work surge** - Secondary traffic peak
    * **18:45: Consumer rebalance** - 6x lag spike during rebalancing
    * **20:00-22:00: Evening drop** - Steep decline to overnight levels
  </Step>

  <Step>
    <h4 id="start-clickstack">
      Start ClickStack
    </h4>

    Start a ClickStack instance:

    ```bash theme={null}
    docker run -d --name clickstack-demo \
      -p 8080:8080 -p 4317:4317 -p 4318:4318 \
      clickhouse/clickstack-all-in-one:latest
    ```
  </Step>

  <Step>
    <h4 id="load-metrics">
      Load metrics into ClickStack
    </h4>

    Load the metrics directly into ClickHouse:

    ```bash theme={null}
    # Load gauge metrics (partition counts, queue sizes, latencies, consumer lag)
    cat kafka-metrics-gauge.csv | docker exec -i clickstack-demo \
      clickhouse-client --query "INSERT INTO otel_metrics_gauge FORMAT CSVWithNames"

    # Load sum metrics (message rates, byte rates, request counts)
    cat kafka-metrics-sum.csv | docker exec -i clickstack-demo \
      clickhouse-client --query "INSERT INTO otel_metrics_sum FORMAT CSVWithNames"
    ```
  </Step>

  <Step>
    <h4 id="verify-demo-metrics">
      Verify metrics in HyperDX
    </h4>

    Once loaded, the quickest way to see your metrics is through the pre-built dashboard.

    Proceed to the [Dashboards and visualization](#dashboards) section to import the dashboard and view all Kafka metrics at once.

    <Info>
      **Timezone Display**

      HyperDX displays timestamps in your browser's local timezone. The demo data spans **2025-11-05 16:00:00 - 2025-11-06 16:00:00 (UTC)**. Set your time range to **2025-11-04 16:00:00 - 2025-11-07 16:00:00** to ensure you see the demo metrics regardless of your location. Once you see the metrics, you can narrow the range to a 24-hour period for clearer visualizations.
    </Info>
  </Step>
</Steps>

<h2 id="dashboards">
  Dashboards and visualization
</h2>

To help you get started monitoring Kafka with ClickStack, we provide essential visualizations for Kafka metrics.

<Steps>
  <Step>
    <h4 id="download">
      <TrackedLink href={'/examples/kafka-metrics-dashboard.json'} download="kafka-metrics-dashboard.json" eventName="docs.kafka_metrics_monitoring.dashboard_download">Download</TrackedLink> the dashboard configuration
    </h4>
  </Step>

  <Step>
    <h4 id="import-dashboard">
      Import the pre-built dashboard
    </h4>

    1. Open HyperDX and navigate to the Dashboards section
    2. Click **Import Dashboard** in the upper right corner under the ellipses

    <Image img="https://mintcdn.com/private-7c7dfe99-mintlify-8a08bda2/OwB6o9ddvLojEP8N/images/clickstack/import-dashboard.png?fit=max&auto=format&n=OwB6o9ddvLojEP8N&q=85&s=cdfe26f160c1c080b995c8451311241d" alt="Import dashboard button" width="3024" height="556" data-path="images/clickstack/import-dashboard.png" />

    3. Upload the `kafka-metrics-dashboard.json` file and click **Finish Import**

    <Image img="https://mintcdn.com/private-7c7dfe99-mintlify-8a08bda2/OwB6o9ddvLojEP8N/images/clickstack/kafka/import-kafka-dashboard.png?fit=max&auto=format&n=OwB6o9ddvLojEP8N&q=85&s=277a0844972135198649bda66599aad4" alt="Finish import dialog" width="1898" height="966" data-path="images/clickstack/kafka/import-kafka-dashboard.png" />
  </Step>

  <Step>
    <h4 id="created-dashboard">
      View the dashboard
    </h4>

    The dashboard will be created with all visualizations pre-configured:

    <Image img="https://mintcdn.com/private-7c7dfe99-mintlify-8a08bda2/OwB6o9ddvLojEP8N/images/clickstack/kafka/kafka-metrics-dashboard.png?fit=max&auto=format&n=OwB6o9ddvLojEP8N&q=85&s=91981766e83ba9f510bfa1c7bef6d8d8" alt="Kafka Metrics dashboard" width="1905" height="968" data-path="images/clickstack/kafka/kafka-metrics-dashboard.png" />

    <Note>
      For the demo dataset, set the time range to **2025-11-05 16:00:00 - 2025-11-06 16:00:00 (UTC)** (adjust based on your local timezone). The imported dashboard won't have a time range specified by default.
    </Note>
  </Step>
</Steps>

<h2 id="troubleshooting">
  Troubleshooting
</h2>

<h3 id="no-metrics">
  No metrics appearing in HyperDX
</h3>

**Verify API key is set and passed to the container:**

```bash theme={null}
# Check environment variable
echo $CLICKSTACK_API_KEY

# Verify it's in the container
docker exec <jmx-exporter-container> env | grep CLICKSTACK_API_KEY
```

If missing, set it and restart:

```bash theme={null}
export CLICKSTACK_API_KEY=your-api-key-here
docker compose up -d kafka-jmx-exporter
```

**Check if metrics are reaching ClickHouse:**

```bash theme={null}
docker exec <clickstack-container> clickhouse-client --query "
SELECT DISTINCT MetricName 
FROM otel_metrics_sum 
WHERE ServiceName = 'kafka' 
LIMIT 10
"
```

If you don't see any results, check the JMX exporter logs:

```bash theme={null}
docker compose logs kafka-jmx-exporter | grep -i "error\|connection" | tail -10
```

**Generate Kafka activity to populate metrics:**

```bash theme={null}
# Create a test topic
docker exec kafka bash -c "unset JMX_PORT && kafka-topics --create --topic test-topic --bootstrap-server kafka:9092 --partitions 3 --replication-factor 1"

# Send test messages
echo -e "Message 1\nMessage 2\nMessage 3" | docker exec -i kafka bash -c "unset JMX_PORT && kafka-console-producer --topic test-topic --bootstrap-server kafka:9092"
```

<h3 id="auth-errors">
  Authentication errors
</h3>

If you see `Authorization failed` or `401 Unauthorized`:

1. Verify the API key in HyperDX UI (Settings → API Keys → Ingestion API Key)
2. Re-export and restart:

```bash theme={null}
export CLICKSTACK_API_KEY=your-correct-api-key
docker compose down
docker compose up -d
```

<h3 id="port-conflicts">
  Port conflicts with Kafka client commands
</h3>

When running Kafka commands from within the Kafka container, you may see:

```bash theme={null}
Error: Port already in use: 9999
```

Prefix commands with `unset JMX_PORT &&`:

```bash theme={null}
docker exec kafka bash -c "unset JMX_PORT && kafka-topics --list --bootstrap-server kafka:9092"
```

<h3 id="network-issues">
  Network connectivity issues
</h3>

If the JMX exporter logs show `Connection refused`:

Verify all containers are on the same Docker network:

```bash theme={null}
docker compose ps
docker network inspect <network-name>
```

Test connectivity:

```bash theme={null}
# From JMX exporter to ClickStack
docker exec <jmx-exporter-container> sh -c "timeout 2 bash -c 'cat < /dev/null > /dev/tcp/clickstack/4318' && echo 'Connected' || echo 'Failed'"
```

<h2 id="next-steps">
  Next steps
</h2>

* Set up [alerts](/clickstack/features/alerts) for critical metrics (under-replicated partitions, consumer lag growth, request latency spikes)
* Create additional dashboards for specific use cases (per-topic throughput, consumer group monitoring)
* Monitor multiple Kafka brokers by adding additional JMX Metric Gatherer instances with unique `kafka.broker.id` resource attributes

<h2 id="going-to-production">
  Going to production
</h2>

This guide sends metrics directly from the JMX Metric Gatherer to ClickStack's OTLP endpoint, which works well for testing and small deployments.

For production environments, deploy your own OpenTelemetry Collector as an agent to receive metrics from the JMX Exporter and forward them to ClickStack. This provides batching, resilience, and centralized configuration management.

See [Ingesting with OpenTelemetry](/clickstack/ingesting-data/opentelemetry) for production deployment patterns and collector configuration examples.