evaluate-spring-ai

Evaluate Spring AI

Set up Dokimos evaluation for a Spring AI application. The user will describe their application and evaluation goals via $ARGUMENTS.

Where things live

• Spring AI support: dokimos-spring-ai/src/main/java/dev/dokimos/springai/SpringAiSupport.java
• Examples: dokimos-examples/src/main/java/dev/dokimos/examples/springai/
• Full Spring Boot example: dokimos-examples/src/main/java/dev/dokimos/examples/springai/tutorial/
• Maven dependency: dev.dokimos:dokimos-spring-ai

Before writing code, read SpringAiSupport.java to understand the available utilities.

Key utilities

SpringAiSupport provides:

• asJudge(ChatClient.Builder) — wraps a Spring AI ChatClient.Builder into a JudgeLM
• asJudge(ChatModel) — wraps a ChatModel directly into a JudgeLM
• toTestCase(EvaluationRequest) — converts Spring AI's EvaluationRequest to Dokimos EvalTestCase
• toEvaluationResponse(EvalResult) — converts Dokimos EvalResult back to Spring AI EvaluationResponse

Evaluation patterns

Simple ChatClient evaluation

@SpringBootTest
class MyChatEvaluationTest {

    @Autowired
    private ChatClient.Builder chatClientBuilder;

    @Test
    void evaluateChatbot() {
        ChatClient chatClient = chatClientBuilder.build();

        Task task = example -> {
            String response = chatClient.prompt()
                    .user(example.input())
                    .call()
                    .content();
            return Map.of("output", response);
        };

        JudgeLM judge = SpringAiSupport.asJudge(chatClientBuilder);

        ExperimentResult result = Experiment.builder()
                .name("Chatbot Evaluation")
                .dataset(Dataset.fromJson(Path.of("src/test/resources/datasets/qa.json")))
                .task(task)
                .evaluator(LLMJudgeEvaluator.builder()
                        .name("answer-quality")
                        .judge(judge)
                        .criteria("Is the response helpful and accurate?")
                        .evaluationParams(List.of(
                                EvalTestCaseParam.INPUT,
                                EvalTestCaseParam.ACTUAL_OUTPUT,
                                EvalTestCaseParam.EXPECTED_OUTPUT))
                        .threshold(0.7)
                        .build())
                .build()
                .run();
    }
}

RAG evaluation with advisors

Task task = example -> {
    String input = example.input();
    ChatClient.ChatClientRequestSpec request = chatClient.prompt().user(input);
    request.advisors(new QuestionAnswerAdvisor(vectorStore));

    String response = request.call().content();
    List<Document> docs = vectorStore.similaritySearch(input);
    List<String> context = docs.stream().map(Document::getText).toList();

    return Map.of("output", response, "context", context);
};

Converting between Spring AI and Dokimos types

EvaluationRequest request = new EvaluationRequest(userText, documents, responseContent);
EvalTestCase testCase = SpringAiSupport.toTestCase(request);
EvalResult result = evaluator.evaluate(testCase);
EvaluationResponse response = SpringAiSupport.toEvaluationResponse(result);

Dependencies

<dependency>
    <groupId>dev.dokimos</groupId>
    <artifactId>dokimos-spring-ai</artifactId>
    <version>${dokimos.version}</version>
</dependency>

Spring AI itself is a provided-scope dependency — the user must bring their own version.

Steps

1. Understand from $ARGUMENTS what the Spring AI application does
2. Determine if it's a simple ChatClient app or uses RAG advisors
3. Choose appropriate evaluators for the use case
4. Create a dataset matching the application's domain
5. Wire evaluation using SpringAiSupport utilities
6. For Spring Boot apps, set up tests with @SpringBootTest