> ## Documentation Index
> Fetch the complete documentation index at: https://private-7c7dfe99-mintlify-8a08bda2.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Monitoring EC2 host logs with ClickStack

> Monitoring EC2 Host Logs with ClickStack

export const TrackedLink = ({href, eventName, children, ...rest}) => {
  const handleClick = () => {
    try {
      if (typeof window !== "undefined" && window.galaxy && eventName) {
        window.galaxy.track(eventName, {
          interaction: "click"
        });
      }
    } catch (e) {}
  };
  return <a href={href} onClick={handleClick} {...rest}>
      {children}
    </a>;
};

export const Image = ({img, alt, size}) => {
  return <Frame>
      <img src={img} alt={alt} />
    </Frame>;
};

<Info>
  **TL;DR**

  Collect and visualize EC2 system logs in ClickStack using the OpenTelemetry Collector with automatic EC2 metadata enrichment (instance ID, region, AZ, instance type). Includes a demo dataset and pre-built dashboard.
</Info>

<h2 id="existing-ec2">
  Integration with existing EC2 instance
</h2>

This section covers installing OpenTelemetry Collector on your EC2 instances to collect system logs and send them to ClickStack with automatic EC2 metadata enrichment. This distributed architecture is production-ready and scales to multiple instances.

<Info>
  **Running ClickStack on the same EC2 instance?**

  If ClickStack is running on the same EC2 instance whose logs you want to monitor, you can use the all-in-one approach similar to the [Generic Host Logs guide](/clickstack/integration-examples/host-logs). Mount `/var/log` into the ClickStack container and add the `resourcedetection` processor to your custom config to automatically capture EC2 metadata. This guide focuses on the more common distributed architecture for production deployments.
</Info>

If you would like to test the EC2 host logs integration before configuring your production instance, you can test with our preconfigured setup and sample data in the ["Demo dataset"](/clickstack/integration-examples/host-logs/ec2#demo-dataset) section.

<h5 id="prerequisites">
  Prerequisites
</h5>

* ClickStack instance running (can be on-premises, cloud, or local)
* EC2 instance running (Ubuntu, Amazon Linux, or other Linux distribution)
* Network connectivity from EC2 instance to ClickStack's OTLP endpoint (port 4318 for HTTP or 4317 for gRPC)
* EC2 instance metadata service accessible (enabled by default)

<Steps>
  <Step>
    <h4 id="verify-metadata">
      Verify EC2 metadata is accessible
    </h4>

    From your EC2 instance, verify the metadata service is accessible:

    ```bash theme={null}
    # Get metadata token (IMDSv2)
    TOKEN=$(curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")

    # Verify instance metadata
    curl -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/instance-id
    curl -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/placement/region
    curl -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/instance-type
    ```

    You should see your instance ID, region, and instance type. If these commands fail, verify:

    * The instance metadata service is enabled
    * IMDSv2 isn't blocked by security groups or network ACLs
    * You're running these commands from the EC2 instance itself

    <Note>
      EC2 metadata is available at `http://169.254.169.254` from within the instance. The OpenTelemetry `resourcedetection` processor uses this endpoint to automatically enrich logs with cloud context.
    </Note>
  </Step>

  <Step>
    <h4 id="verify-syslog">
      Verify syslog files exist
    </h4>

    Verify that your EC2 instance is writing syslog files:

    ```bash theme={null}
    # Ubuntu instances
    ls -la /var/log/syslog

    # Amazon Linux / RHEL instances
    ls -la /var/log/messages

    # View recent entries
    tail -20 /var/log/syslog
    # or
    tail -20 /var/log/messages
    ```
  </Step>

  <Step>
    <h4 id="install-collector">
      Install OpenTelemetry Collector
    </h4>

    Install the OpenTelemetry Collector Contrib distribution on your EC2 instance:

    ```bash theme={null}
    # Download the latest release
    wget https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.114.0/otelcol-contrib_0.114.0_linux_amd64.tar.gz

    # Extract and install
    tar -xvf otelcol-contrib_0.114.0_linux_amd64.tar.gz
    sudo mv otelcol-contrib /usr/local/bin/

    # Verify installation
    otelcol-contrib --version
    ```
  </Step>

  <Step>
    <h4 id="create-config">
      Create collector configuration
    </h4>

    Create a configuration file for the OpenTelemetry Collector at `/etc/otelcol-contrib/config.yaml`:

    ```bash theme={null}
    sudo mkdir -p /etc/otelcol-contrib
    ```

    Choose the configuration based on your Linux distribution:

    <Tabs>
      <Tab title="Modern Linux (Ubuntu 24.04+)">
        ```yaml theme={null}
        sudo tee /etc/otelcol-contrib/config.yaml > /dev/null << 'EOF'
        receivers:
          filelog/syslog:
            include:
              - /var/log/syslog
              - /var/log/**/*.log
            start_at: end
            operators:
              - type: regex_parser
                regex: '^(?P<timestamp>\S+) (?P<hostname>\S+) (?P<unit>\S+?)(?:\[(?P<pid>\d+)\])?: (?P<message>.*)$'
                parse_from: body
                parse_to: attributes
              
              - type: time_parser
                parse_from: attributes.timestamp
                layout_type: gotime
                layout: '2006-01-02T15:04:05.999999-07:00'
              
              - type: add
                field: attributes.source
                value: "ec2-host-logs"

        processors:
          resourcedetection:
            detectors: [ec2, system]
            timeout: 5s
            override: false
            ec2:
              tags:
                - ^Name
                - ^Environment
                - ^Team
          
          batch:
            timeout: 10s
            send_batch_size: 10000

        exporters:
          otlphttp:
            endpoint: "http://YOUR_CLICKSTACK_HOST:4318"
            headers:
              authorization: "${env:CLICKSTACK_API_KEY}"

        service:
          pipelines:
            logs:
              receivers: [filelog/syslog]
              processors: [resourcedetection, batch]
              exporters: [otlphttp]
        EOF
        ```
      </Tab>

      <Tab title="Legacy Linux (Amazon Linux 2, RHEL, older Ubuntu)">
        ```yaml theme={null}
        sudo tee /etc/otelcol-contrib/config.yaml > /dev/null << 'EOF'
        receivers:
          filelog/syslog:
            include:
              - /var/log/messages
              - /var/log/**/*.log
            start_at: end
            operators:
              - type: regex_parser
                regex: '^(?P<timestamp>\w+ \d+ \d{2}:\d{2}:\d{2}) (?P<hostname>\S+) (?P<unit>\S+?)(?:\[(?P<pid>\d+)\])?: (?P<message>.*)$'
                parse_from: body
                parse_to: attributes
              
              - type: time_parser
                parse_from: attributes.timestamp
                layout: '%b %d %H:%M:%S'
              
              - type: add
                field: attributes.source
                value: "ec2-host-logs"

        processors:
          resourcedetection:
            detectors: [ec2, system]
            timeout: 5s
            override: false
            ec2:
              tags:
                - ^Name
                - ^Environment
                - ^Team
          
          batch:
            timeout: 10s
            send_batch_size: 10000

        exporters:
          otlphttp:
            endpoint: "http://YOUR_CLICKSTACK_HOST:4318"
            headers:
              authorization: "${env:CLICKSTACK_API_KEY}"

        service:
          pipelines:
            logs:
              receivers: [filelog/syslog]
              processors: [resourcedetection, batch]
              exporters: [otlphttp]
        EOF
        ```
      </Tab>
    </Tabs>

    <br />

    **Replace the following in the configuration:**

    * `YOUR_CLICKSTACK_HOST`: The hostname or IP address where ClickStack is running
    * For local testing, you can use an SSH tunnel (see the [Troubleshooting section](#troubleshooting))

    This configuration:

    * Reads system log files from standard locations (`/var/log/syslog` for Ubuntu, `/var/log/messages` for Amazon Linux/RHEL)
    * Parses syslog format to extract structured fields (timestamp, hostname, unit/service, PID, message)
    * **Automatically detects and adds EC2 metadata** using the `resourcedetection` processor
    * Optionally includes EC2 tags (Name, Environment, Team) if present
    * Sends logs to ClickStack via OTLP HTTP

    <Info>
      **EC2 Metadata Enrichment**

      The `resourcedetection` processor automatically adds these attributes to every log:

      * `cloud.provider`: "aws"
      * `cloud.platform`: "aws\_ec2"
      * `cloud.region`: AWS region (e.g., "us-east-1")
      * `cloud.availability_zone`: AZ (e.g., "us-east-1a")
      * `cloud.account.id`: AWS account ID
      * `host.id`: EC2 instance ID (e.g., "i-1234567890abcdef0")
      * `host.type`: Instance type (e.g., "t3.medium")
      * `host.name`: Instance hostname
    </Info>
  </Step>

  <Step>
    <h4 id="set-api-key">
      Set ClickStack API key
    </h4>

    Export your ClickStack API key as an environment variable:

    ```bash theme={null}
    export CLICKSTACK_API_KEY="your-api-key-here"
    ```

    To make this persistent across reboots, add it to your shell profile:

    ```bash theme={null}
    echo 'export CLICKSTACK_API_KEY="your-api-key-here"' >> ~/.bashrc
    source ~/.bashrc
    ```
  </Step>

  <Step>
    <h4 id="run-collector">
      Run the collector
    </h4>

    Start the OpenTelemetry Collector:

    ```bash theme={null}
    CLICKSTACK_API_KEY="your-api-key-here" /usr/local/bin/otelcol-contrib --config /etc/otelcol-contrib/config.yaml
    ```

    <Info>
      **For production use**

      Configure the collector to run as a systemd service so it starts automatically on boot and restarts on failure. See the [OpenTelemetry Collector documentation](https://opentelemetry.io/docs/collector/deployment/) for details.
    </Info>
  </Step>

  <Step>
    <h4 id="verifying-logs">
      Verifying Logs in HyperDX
    </h4>

    Once the collector is running, log into HyperDX and verify logs are flowing with EC2 metadata:

    1. Navigate to the search view
    2. Set source to `Logs`
    3. Filter by `source:ec2-host-logs`
    4. Click on a log entry to expand it
    5. Verify you see EC2 metadata in the resource attributes:
       * `cloud.provider`
       * `cloud.region`
       * `host.id` (instance ID)
       * `host.type` (instance type)
       * `cloud.availability_zone`

    <Image img="https://mintcdn.com/private-7c7dfe99-mintlify-8a08bda2/OwB6o9ddvLojEP8N/images/clickstack/host-logs/ec2/search-view.png?fit=max&auto=format&n=OwB6o9ddvLojEP8N&q=85&s=5028c29f33660843b864fda94c9868ce" alt="EC2 logs search view" width="1920" height="920" data-path="images/clickstack/host-logs/ec2/search-view.png" />

    <Image img="https://mintcdn.com/private-7c7dfe99-mintlify-8a08bda2/OwB6o9ddvLojEP8N/images/clickstack/host-logs/ec2/log-view.png?fit=max&auto=format&n=OwB6o9ddvLojEP8N&q=85&s=6fcb2e2f5f51b464fe1b78ad0f8a5d06" alt="EC2 log detail showing metadata" width="3840" height="1840" data-path="images/clickstack/host-logs/ec2/log-view.png" />
  </Step>
</Steps>

<h2 id="demo-dataset">
  Demo dataset
</h2>

For users who want to test the EC2 host logs integration before configuring their production instances, we provide a sample dataset with simulated EC2 metadata.

<Steps>
  <Step>
    <h4 id="download-sample">
      Download the sample dataset
    </h4>

    Download the sample log file:

    ```bash theme={null}
    curl -O https://datasets-documentation.s3.eu-west-3.amazonaws.com/clickstack-integrations/host-logs/journal.log
    ```

    The dataset includes:

    * System boot sequence
    * SSH login activity (successful and failed attempts)
    * Security incident (brute force attack with fail2ban response)
    * Scheduled maintenance (cron jobs, anacron)
    * Service restarts (rsyslog)
    * Kernel messages and firewall activity
    * Mix of normal operations and notable events
  </Step>

  <Step>
    <h4 id="test-config">
      Create test collector configuration
    </h4>

    Create a file named `ec2-host-logs-demo.yaml` with the following configuration:

    ```yaml theme={null}
    cat > ec2-host-logs-demo.yaml << 'EOF'
    receivers:
      filelog/journal:
        include:
          - /tmp/host-demo/journal.log
        start_at: beginning
        operators:
          - type: regex_parser
            regex: '^(?P<timestamp>\S+) (?P<hostname>\S+) (?P<unit>\S+?)(?:\[(?P<pid>\d+)\])?: (?P<message>.*)$'
            parse_from: body
            parse_to: attributes
          
          - type: time_parser
            parse_from: attributes.timestamp
            layout: '%Y-%m-%dT%H:%M:%S%z'
          
          - type: add
            field: attributes.source
            value: "ec2-demo"

    processors:
      # Simulate EC2 metadata for demo (no real EC2 instance required)
      resource:
        attributes:
          - key: service.name
            value: "ec2-demo"
            action: insert
          - key: cloud.provider
            value: "aws"
            action: insert
          - key: cloud.platform
            value: "aws_ec2"
            action: insert
          - key: cloud.region
            value: "us-east-1"
            action: insert
          - key: cloud.availability_zone
            value: "us-east-1a"
            action: insert
          - key: host.id
            value: "i-0abc123def456789"
            action: insert
          - key: host.type
            value: "t3.medium"
            action: insert
          - key: host.name
            value: "prod-web-01"
            action: insert

    service:
      pipelines:
        logs/ec2-demo:
          receivers: [filelog/journal]
          processors:
            - resource
            - memory_limiter
            - transform
            - batch
          exporters:
            - clickhouse
    EOF
    ```

    <Note>
      For demo purposes, we manually add EC2 metadata using the `resource` processor. In production with real EC2 instances, use the `resourcedetection` processor which automatically queries the EC2 metadata API.
    </Note>
  </Step>

  <Step>
    <h4 id="run-demo">
      Run ClickStack with demo configuration
    </h4>

    Run ClickStack with the demo logs and configuration:

    ```bash theme={null}
    docker run --name clickstack-demo \
      -p 8080:8080 -p 4317:4317 -p 4318:4318 \
      -e CUSTOM_OTELCOL_CONFIG_FILE=/etc/otelcol-contrib/custom.config.yaml \
      -v "$(pwd)/ec2-host-logs-demo.yaml:/etc/otelcol-contrib/custom.config.yaml:ro" \
      -v "$(pwd)/journal.log:/tmp/host-demo/journal.log:ro" \
      docker.hyperdx.io/hyperdx/hyperdx-all-in-one:latest
    ```
  </Step>

  <Step>
    <h4 id="verify-demo-logs">
      Verify logs in HyperDX
    </h4>

    Once the collector is running:

    1. Open [HyperDX](http://localhost:8080/) and log in to your account (you may need to create an account first)
    2. Navigate to the search view and set the source to `Logs`
    3. Set the time range to **2025-11-10 00:00:00 - 2025-11-13 00:00:00**
    4. Filter by `source:ec2-demo`
    5. Expand a log entry to view EC2 metadata in the resource attributes

    <Image img="https://mintcdn.com/private-7c7dfe99-mintlify-8a08bda2/OwB6o9ddvLojEP8N/images/clickstack/host-logs/ec2/search-view-demo.png?fit=max&auto=format&n=OwB6o9ddvLojEP8N&q=85&s=0d78834ddad2bd50eea830db0a4238f6" alt="EC2 logs search view" width="1920" height="968" data-path="images/clickstack/host-logs/ec2/search-view-demo.png" />

    <Image img="https://mintcdn.com/private-7c7dfe99-mintlify-8a08bda2/OwB6o9ddvLojEP8N/images/clickstack/host-logs/ec2/log-view-demo.png?fit=max&auto=format&n=OwB6o9ddvLojEP8N&q=85&s=40a32e681088e6899b61b980acf09720" alt="EC2 log detail with metadata" width="1920" height="968" data-path="images/clickstack/host-logs/ec2/log-view-demo.png" />

    <Info>
      **Timezone display**

      HyperDX displays timestamps in your browser's local timezone. The demo data spans **2025-11-11 00:00:00 - 2025-11-12 00:00:00 (UTC)**. The wide time range ensures you'll see the demo logs regardless of your location. Once you see the logs, you can narrow the range to a 24-hour period for clearer visualizations.
    </Info>

    You should see logs with simulated EC2 context including:

    * Instance ID: `i-0abc123def456789`
    * Region: `us-east-1`
    * Availability Zone: `us-east-1a`
    * Instance Type: `t3.medium`
  </Step>
</Steps>

<h2 id="dashboards">
  Dashboards and visualization
</h2>

To help you get started monitoring EC2 host logs with ClickStack, we provide essential visualizations with cloud context.

<Steps>
  <Step>
    <h4 id="download">
      <TrackedLink href={'/examples/host-logs-dashboard.json'} download="host-logs-dashboard.json" eventName="docs.ec2_host_logs_monitoring.dashboard_download">Download</TrackedLink> the dashboard configuration
    </h4>
  </Step>

  <Step>
    <h4 id="import-dashboard">
      Import the pre-built dashboard
    </h4>

    1. Open HyperDX and navigate to the Dashboards section
    2. Click **Import Dashboard** in the upper right corner under the ellipses

    <Image img="https://mintcdn.com/private-7c7dfe99-mintlify-8a08bda2/OwB6o9ddvLojEP8N/images/clickstack/import-dashboard.png?fit=max&auto=format&n=OwB6o9ddvLojEP8N&q=85&s=cdfe26f160c1c080b995c8451311241d" alt="Import dashboard button" width="3024" height="556" data-path="images/clickstack/import-dashboard.png" />

    3. Upload the `host-logs-dashboard.json` file and click **Finish Import**

    <Image img="https://mintcdn.com/private-7c7dfe99-mintlify-8a08bda2/OwB6o9ddvLojEP8N/images/clickstack/host-logs/import-dashboard.png?fit=max&auto=format&n=OwB6o9ddvLojEP8N&q=85&s=eb2f0ac225bfd0c96254a6bcb5a7b2ed" alt="Finish import" width="3808" height="1908" data-path="images/clickstack/host-logs/import-dashboard.png" />
  </Step>

  <Step>
    <h4 id="created-dashboard">
      View the dashboard
    </h4>

    The dashboard will be created with all visualizations pre-configured:

    <Image img="https://mintcdn.com/private-7c7dfe99-mintlify-8a08bda2/OwB6o9ddvLojEP8N/images/clickstack/host-logs/host-logs-dashboard.png?fit=max&auto=format&n=OwB6o9ddvLojEP8N&q=85&s=60ac12d228eb29d361c8dfdf7c368c28" alt="EC2 logs dashboard" width="3808" height="1908" data-path="images/clickstack/host-logs/host-logs-dashboard.png" />

    You can filter dashboard visualizations by EC2 context:

    * `cloud.region:us-east-1` - Show logs from specific region
    * `host.type:t3.medium` - Filter by instance type
    * `host.id:i-0abc123def456` - Logs from specific instance

    <Note>
      For the demo dataset, set the time range to **2025-11-11 00:00:00 - 2025-11-12 00:00:00 (UTC)** (adjust based on your local timezone). The imported dashboard won't have a time range specified by default.
    </Note>
  </Step>
</Steps>

<h2 id="troubleshooting">
  Troubleshooting
</h2>

<h3 id="no-metadata">
  EC2 metadata not appearing in logs
</h3>

**Verify the EC2 metadata service is accessible:**

```bash theme={null}
# Get metadata token
TOKEN=$(curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")

# Test metadata endpoint
curl -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/instance-id
```

If this fails, verify:

* The instance metadata service is enabled
* IMDSv2 isn't blocked by security groups
* You're running the collector on the EC2 instance itself

**Check collector logs for metadata errors:**

```bash theme={null}
# If running as systemd service
sudo journalctl -u otelcol-contrib -f | grep -i "ec2\|metadata\|resourcedetection"

# If running in foreground, check stdout
```

<h3 id="no-logs">
  No logs appearing in HyperDX
</h3>

**Verify syslog files exist and are being written:**

```bash theme={null}
ls -la /var/log/syslog /var/log/messages
tail -f /var/log/syslog
```

**Check collector can read the log files:**

```bash theme={null}
cat /var/log/syslog | head -20
```

**Verify network connectivity to ClickStack:**

```bash theme={null}
# Test OTLP endpoint
curl -v http://YOUR_CLICKSTACK_HOST:4318/v1/logs

# Should get a response (even if error, means endpoint is reachable)
```

**Check collector logs for errors:**

```bash theme={null}
# If running in foreground
# Look for error messages in stdout

# If running as systemd service
sudo journalctl -u otelcol-contrib -f | grep -i "error\|failed"
```

<h3 id="logs-not-parsing">
  Logs parsing incorrectly
</h3>

**Verify your syslog format:**

For Ubuntu 24.04+:

```bash theme={null}
# Should show ISO8601 format: 2025-11-17T20:55:44.826796+00:00
tail -5 /var/log/syslog
```

For Amazon Linux 2 / Ubuntu 20.04:

```bash theme={null}
# Should show traditional format: Nov 17 14:16:16
tail -5 /var/log/messages
```

If your format doesn't match, use the appropriate configuration tab in the [Create collector configuration](#create-config) section based on your distribution.

<h3 id="systemd-issues">
  Collector not starting as systemd service
</h3>

**Check service status:**

```bash theme={null}
sudo systemctl status otelcol-contrib
```

**View detailed logs:**

```bash theme={null}
sudo journalctl -u otelcol-contrib -n 50
```

**Common issues:**

* API key not set correctly in environment
* Config file syntax errors
* Permission issues reading log files

<h2 id="next-steps">
  Next steps
</h2>

* Set up [alerts](/clickstack/features/alerts) for critical system events (service failures, authentication failures, disk warnings)
* Filter by EC2 metadata attributes (region, instance type, instance ID) to monitor specific resources
* Correlate EC2 host logs with application logs for comprehensive troubleshooting
* Create custom dashboards for security monitoring (SSH attempts, sudo usage, firewall blocks)

<h2 id="going-to-production">
  Going to production
</h2>

This guide installs the OpenTelemetry Collector directly on EC2 instances, which is the recommended production pattern for host-level monitoring. For managing collectors across many instances, consider using configuration management tools (Ansible, Chef, Puppet) or the OpenTelemetry Operator in Kubernetes environments. See [Sending OpenTelemetry data](/clickstack/ingesting-data/opentelemetry) for production configuration.