lapdog

Datadog APM test agent

The APM test agent is an application which emulates the APM endpoints of the Datadog agent which can be used for testing Datadog APM client libraries.

See the Features section for the complete list of functionalities provided.

See the HTTP API section for the endpoints available.

See the Development section for how to get the test agent running locally to add additional checks or fix bugs.

Installation

The test agent can be installed from PyPI:

pip install ddapm-test-agent

# HTTP on port 8126, OTLP HTTP on port 4318, OTLP GRPC on port 4317, with the web-ui enabled
ddapm-test-agent --port=8126 --otlp-http-port=4318 --otlp-grpc-port=4317 --web-ui-port=8080

or from Docker:

# Run the test agent and mount the snapshot directory
docker run --rm\
        -p 8126:8126\
        -p 4318:4318\
        -p 4317:4317\
        -e SNAPSHOT_CI=0\
        -v $PWD/tests/snapshots:/snapshots\
        ghcr.io/datadog/dd-apm-test-agent/ddapm-test-agent:latest

or from source:

pip install git+https://github.com/Datadog/dd-apm-test-agent

or a specific branch:

pip install git+https://github.com/Datadog/dd-apm-test-agent@{branch}

Features

Trace invariant checks

Many checks are provided by the test agent which will verify trace data. All checks are enabled by default and can be manually disabled.

See the configuration section for the options.

Check description	Check name
Trace count header matches number of traces	trace_count_header
Client library version header included in request	meta_tracer_version_header
Trace content length header matches payload size	trace_content_length

Returning data

All data that is submitted to the test agent can be retrieved.

• Traces can be returned via the /test/traces endpoint documented below.

Helpful logging

The INFO log level of the test agent outputs useful information about the requests the test agent receives. For traces this includes a visual representation of the traces.

INFO:ddapm_test_agent.agent:received trace payload with 1 trace chunk
INFO:ddapm_test_agent.agent:Chunk 0
[parent]
├─ [child1]
├─ [child2]
└─ [child3]
INFO:ddapm_test_agent.agent:end of payload ----------------------------------------

Proxy

The test agent provides proxying to the Datadog agent. This is enabled by passing the agent url to the test agent either via the --agent-url command-line argument or by the DD_TRACE_AGENT_URL or DD_AGENT_URL environment variables.

When proxying is enabled, the response from the Datadog agent will be returned instead of one from the test agent.

At the trace-level, proxying can also be disabled by including the X-Datadog-Agent-Proxy-Disabled header with a value of true. This will disable proxying after a trace is handled, regardless of whether an agent URL is set.

LLM Observability Proxy

The test agent provides a mechanism to dual ship LLM Observability events to Datadog, regardless of whether the Datadog agent is running. If using the Datadog agent, set the DD_AGENT_URL environment variable or --agent-url command-line argument to the URL of the Datadog agent (see Proxy for more details).

If not running a Datadog agent, set the DD_SITE environment variable or --dd-site command-line argument to the site of the Datadog instance to forward events to. Additionally, set the DD_API_KEY environment variable or --dd-api-key command-line argument to the API key to use for the Datadog instance.

To disable LLM Observability event forwarding, set the DISABLE_LLMOBS_DATA_FORWARDING environment variable or --disable-llmobs-data-forwarding command-line argument to true.

Claude Code Hooks

The test agent can receive Claude Code hook events and assemble them into LLM Observability traces.

See .claude/setup-hooks.md for full setup instructions (also usable as a Claude Code prompt to automate the setup).

Snapshot testing