evaluate-langchain4j

Evaluate LangChain4j

Set up Dokimos evaluation for a LangChain4j application. The user will describe their application and evaluation goals via $ARGUMENTS.

Where things live

• LangChain4j support: dokimos-langchain4j/src/main/java/dev/dokimos/langchain4j/LangChain4jSupport.java
• Example: dokimos-examples/src/main/java/dev/dokimos/examples/langchain4j/LangChain4jRAGExample.java
• Maven dependency: dev.dokimos:dokimos-langchain4j

Before writing code, read LangChain4jSupport.java to understand the available utilities.

Key utilities

LangChain4jSupport provides:

• asJudge(ChatModel) — wraps a LangChain4j ChatModel into a JudgeLM
• simpleTask(ChatModel) — creates a Task for simple Q&A evaluation
• ragTask(Function<String, Result<String>>) — creates a Task for RAG evaluation that captures both output and retrieval context
• ragTask(..., inputKey, outputKey, contextKey) — RAG task with custom key names
• customTask(Task) — pass-through for full control
• extractTexts(List<Content>) — extracts text from LangChain4j Content objects

Evaluation patterns

Simple Q&A evaluation

ChatModel model = OpenAiChatModel.builder()
        .apiKey(System.getenv("OPENAI_API_KEY"))
        .modelName("gpt-4o-mini")
        .build();

Task task = LangChain4jSupport.simpleTask(model);

ExperimentResult result = Experiment.builder()
        .name("QA Evaluation")
        .dataset(Dataset.fromJson(Path.of("datasets/qa.json")))
        .task(task)
        .evaluator(ExactMatchEvaluator.builder().build())
        .build()
        .run();

RAG evaluation

The RAG task captures both the model output and the retrieved context, enabling evaluators like FaithfulnessEvaluator and ContextualRelevanceEvaluator:

// 1. Build your LangChain4j AiService that returns Result<String>
interface Assistant {
    Result<String> chat(String userMessage);
}

Assistant assistant = AiServices.builder(Assistant.class)
        .chatModel(chatModel)
        .retrievalAugmentor(retrievalAugmentor)
        .build();

// 2. Create the RAG task
Task task = LangChain4jSupport.ragTask(assistant::chat);

// 3. Create a judge for LLM-based evaluators
JudgeLM judge = LangChain4jSupport.asJudge(judgeChatModel);

// 4. Run with RAG-specific evaluators
// ragTask() stores context under "context" key by default.
// FaithfulnessEvaluator and HallucinationEvaluator default to contextKey="context" (matches).
// ContextualRelevanceEvaluator defaults to retrievalContextKey="retrievalContext",
// so set it explicitly to match the ragTask output.
ExperimentResult result = Experiment.builder()
        .name("RAG Evaluation")
        .dataset(dataset)
        .task(task)
        .evaluators(List.of(
                FaithfulnessEvaluator.builder()
                        .judge(judge)
                        .threshold(0.7)
                        .build(),
                ContextualRelevanceEvaluator.builder()
                        .judge(judge)
                        .retrievalContextKey("context")
                        .threshold(0.5)
                        .build(),
                HallucinationEvaluator.builder()
                        .judge(judge)
                        .threshold(0.5)
                        .build()
        ))
        .build()
        .run();

Dependencies

<dependency>
    <groupId>dev.dokimos</groupId>
    <artifactId>dokimos-langchain4j</artifactId>
    <version>${dokimos.version}</version>
</dependency>

LangChain4j itself is a provided-scope dependency — the user must bring their own version.

Steps

1. Understand from $ARGUMENTS what the LangChain4j application does (Q&A, RAG, chat, etc.)
2. Determine if it's simple Q&A or RAG evaluation (RAG needs Result<String> return type)
3. Choose appropriate evaluators:
   • Q&A: ExactMatchEvaluator, RegexEvaluator, LLMJudgeEvaluator
   • RAG: FaithfulnessEvaluator, ContextualRelevanceEvaluator, HallucinationEvaluator, PrecisionEvaluator, RecallEvaluator
4. Create a dataset matching the application's domain
5. Wire everything together using LangChain4jSupport utilities