Stats
Actions
Available In
Tags
Help us improve
Share bugs, ideas, or general feedback.
Plugin
nemo-evaluator-skills
Manage the full lifecycle of LLM evaluations on Slurm clusters: launch, monitor, debug, and analyze results via NeMo Evaluator SDK. Create custom benchmarks with the BYOB framework and interactively configure evaluation launchers.
$
npx claudepluginhub nvidia-nemo/evaluatorPopularity
Stars
Med: 0·Avg: 273
Installs
Med: 0·Avg: 1
What's Inside
Skills4
accessing-mlflow
/accessing-mlflow
Query and browse evaluation results stored in MLflow. Use when the user wants to look up runs by invocation ID, compare metrics across models, fetch artifacts (configs, logs, results), or set up the MLflow MCP server. ALWAYS triggers on mentions of MLflow, experiment results, run comparison, invocation IDs in the context of results, or MLflow MCP setup.
launching-evals
/launching-evals
Run, monitor, analyze, and debug LLM evaluations via nemo-evaluator-launcher. Covers running evaluations, checking status and live progress, debugging failed runs, exporting artifacts and logs, and analyzing results. ALWAYS triggers on mentions of running evaluations, checking progress, debugging failed evals, analyzing or analysing runs or results, run directories or artifact paths on clusters, Slurm job issues, invocation IDs, or inspecting logs (client logs, server logs, SSH to cluster, tail logs, grep logs). Do NOT use for creating or modifying evaluation configs. WARNING — this skill performs privileged operations; it SSHes to remote cluster hosts, executes shell commands on cluster infrastructure, transfers files via rsync, and may modify cluster configuration fields (e.g., the SLURM account). Only use with trusted cluster credentials and configs from trusted sources, and require explicit user confirmation before SSH/rsync operations or any cluster-config change.
nel-assistant
/nel-assistant
Interactive config wizard for NeMo Evaluator Launcher (NEL). Use when the user wants to create a new evaluation config from scratch, set up an evaluation from existing configs, or modify a NEL config (deployment, tasks, multi-node, interceptors). ALWAYS triggers on mentions of creating configs, setting up evaluations, configuring models for evaluation, or modifying NEL YAML files. Do NOT use for monitoring, debugging, or analyzing already-running evaluations.
byob
/byob
Create custom LLM evaluation benchmarks using the BYOB decorator framework. Use when the user wants to (1) create a new benchmark from a dataset, (2) pick or write a scorer, (3) compile and run a BYOB benchmark, (4) containerize a benchmark, or (5) use LLM-as-Judge evaluation. Triggers on mentions of BYOB, custom benchmark, bring your own benchmark, scorer, or benchmark compilation.
Stats
Version1.0.0
Released
LanguagePython
Stars287
Forks48
MaintenanceExcellent
LicenseApache-2.0
Last Commit
Added
Tags
Technologies
Topics
Categories
Help us improve
Share bugs, ideas, or general feedback.
