grafana-dpm-finder

🚧 Work in Progress 🚧

dpm-finder

This repository contains a Python script designed to identify metrics in Prometheus that exceed a specified rate. It's particularly useful for detecting metrics with high data points per minute (DPM) rates, which can be indicative of issues or important trends.

The script can run in two modes:

• One-time execution: Calculate DPM and output results to files
• Prometheus exporter: Run as a server that exposes DPM metrics for Prometheus scraping

Overview

The dpm-finder script retrieves a list of all metrics from a Prometheus instance, calculates their data points per minute (DPM) rate using PromQL, and identifies metrics whose DPM exceeds a threshold. Results can be output to files or served via a Prometheus-compatible HTTP endpoint.

Functionality

The script does the following:

1. Retrieves all metrics from a Prometheus instance using the /api/v1/label/__name__/values endpoint.
2. Filters metrics automatically to exclude:
   • Metrics ending with _count, _bucket, or _sum (histogram/summary components)
   • Metrics beginning with grafana_ (Grafana internal metrics)
   • Metrics with aggregation rules defined in the cluster
3. Calculates DPM rate for each metric using a PromQL query: count_over_time({metric_name}[Nm])/N (where N is the configurable lookback window in minutes, default 10).
4. Filters results based on a DPM threshold (metrics with DPM > 1 by default).
5. Outputs results in various formats:
   • One-time mode: CSV, JSON, text, or Prometheus exposition format files
   • Exporter mode: Live Prometheus metrics endpoint at /metrics
6. Provides detailed logging with configurable verbosity levels for monitoring progress and debugging.

How To

1. Create .env with the following variables

Please note the prometheus endpoint should not have anything after .net.

See .env_example

PROMETHEUS_ENDPOINT=""
PROMETHEUS_USERNAME=""
PROMETHEUS_API_KEY=""

2. Install all libraries from requirements.txt

python3 -m venv venv
source ./venv/bin/activate
python3 -m pip install -r requirements.txt 

3. Run the script

One-time execution (traditional mode):

./dpm-finder.py -t 4 -f csv 

Prometheus exporter mode:

./dpm-finder.py -e -p 9966

Docker Usage

The dpm-finder can be run as a Docker container for easy deployment and isolation.

Building the Docker Image

# Build the image
docker build -t dpm-finder:latest .

# Or build with a specific tag
docker build -t dpm-finder:v1.0.0 .

Running with Docker

Environment Variables

Set your Prometheus credentials as environment variables:

export PROMETHEUS_ENDPOINT="https://prometheus-prod-13-prod-us-east-0.grafana.net"
export PROMETHEUS_USERNAME="1234567"
export PROMETHEUS_API_KEY="glc_key-example-..."

Exporter Mode

# Run as Prometheus exporter (default)
docker run -d \
  --name dpm-finder \
  -p 9966:9966 \
  -e PROMETHEUS_ENDPOINT="${PROMETHEUS_ENDPOINT}" \
  -e PROMETHEUS_USERNAME="${PROMETHEUS_USERNAME}" \
  -e PROMETHEUS_API_KEY="${PROMETHEUS_API_KEY}" \
  dpm-finder:latest

# With custom options
docker run -d \
  --name dpm-finder \
  -p 8080:8080 \
  -e PROMETHEUS_ENDPOINT="${PROMETHEUS_ENDPOINT}" \
  -e PROMETHEUS_USERNAME="${PROMETHEUS_USERNAME}" \
  -e PROMETHEUS_API_KEY="${PROMETHEUS_API_KEY}" \
  dpm-finder:latest --exporter --port 8080 --update-interval 43200 --min-dpm 5.0

One-time Execution

# Create output directory
mkdir -p ./output

# Run one-time analysis
docker run --rm \
  -v $(pwd)/output:/app/output \
  -e PROMETHEUS_ENDPOINT="${PROMETHEUS_ENDPOINT}" \
  -e PROMETHEUS_USERNAME="${PROMETHEUS_USERNAME}" \
  -e PROMETHEUS_API_KEY="${PROMETHEUS_API_KEY}" \
  dpm-finder:latest --format csv --min-dpm 2.0 --threads 8

# Results will be in ./output/metric_rates.csv

Using Docker Compose

The included docker-compose.yml provides easy orchestration for both exporter and one-time execution modes.

Prerequisites

1. Install Docker Compose (comes with Docker Desktop)
2. Create a .env file with your Prometheus credentials

Environment File Setup

Create a .env file in the project directory:

# .env file
PROMETHEUS_ENDPOINT=https://prometheus-prod-13-prod-us-east-0.grafana.net
PROMETHEUS_USERNAME=1234567
PROMETHEUS_API_KEY=glc_key-example-...

Basic Exporter Setup

# Start the exporter service
docker-compose up -d

# View logs
docker-compose logs -f dpm-finder

# View live logs with timestamps
docker-compose logs -f --timestamps dpm-finder

# Stop the service
docker-compose down

Advanced Exporter Configuration

Override default settings using command line: