confident-client

Confident AI Admin SDK

Use this skill to manage a Confident AI account programmatically with the Admin SDK — the confidentai package, available for both Python and TypeScript. The work is account and project administration: organizations, projects, members and invitations, RBAC (permissions, policies, roles), governance policies, and API keys.

This skill stops at administration. Sending traces and running evaluations against a project is a different job with a different key — see the deepeval, deepeval-tracing, and deepeval-otel skills.

Scope: Account Administration Only

Use the Admin SDK for administrative workflows that would otherwise be done by hand in the platform UI — project provisioning, member onboarding, role synchronization, and API key rotation. Every operation is scoped to either the organization (account-wide resources) or a single project (project-scoped resources).

Do not reach for this skill to instrument an app or score outputs. Tracing and evals authenticate with a project key (CONFIDENT_API_KEY); administration authenticates with an Organization API Key (CONFIDENT_ORG_API_KEY).

When to Use vs the Eval / Tracing Skills

• This skill (confident-client) — administer the account: create and rename the organization, CRUD projects, invite and manage members, define RBAC, and provision or rotate API keys. Requires an Organization API Key.
• deepeval skill — build pytest eval suites: datasets, metrics, traced evals, deepeval test run, iteration. Requires a project key.
• deepeval-tracing skill — instrument an app with the DeepEval SDK (@observe, integrations). Requires a project key.
• deepeval-otel skill — export raw OpenTelemetry / OTLP traces to Confident AI. Requires a project key.

Prerequisites

• The Admin SDK installed: pip install confidentai (Python) or npm install confidentai (TypeScript).
• An Organization API Key (e.g. confident_us_org_...), read by default from the CONFIDENT_ORG_API_KEY environment variable. This is separate from the project-scoped CONFIDENT_API_KEY used for tracing and evals, so both can be configured at once. Retrieve the org key from the Confident AI settings under organization-level authentication.

Language Detection

Each reference file contains both Python and TypeScript examples. Before writing code, determine which language the user is working in, then use the matching code block (```python or ```typescript) within the reference.

1. Honor an explicit request first — if the user names a language or shows a snippet, use that regardless of the project files.
2. Otherwise infer from the project — but only for a clean single-language project (markers from one ecosystem and none from the other):
   • Python markers: *.py, pyproject.toml, requirements.txt, setup.py, Pipfile, a .venv/venv
   • TypeScript/Node markers: *.ts, *.tsx, package.json, tsconfig.json, node_modules (JavaScript-only counts as TypeScript — same confidentai package)
3. If markers from BOTH ecosystems are present — even if one side has more code — or the project is empty/unclear, it is AMBIGUOUS: STOP and ask which language before writing any code. Do not rationalize a "dominant" or "obvious" language from which files happen to have more content (e.g. a populated ts/ next to a bare .venv is still ambiguous). Fall back to Python (never TypeScript) only after asking and getting no answer, and say you're defaulting.

Within each reference, the code block for the target language is authoritative for exact method names, argument shapes (keyword arguments vs an options object), and casing (snake_case vs camelCase). Do not guess a binding from the other language's examples.

Workflow

1. Confirm the task is administration (managing orgs, projects, members, roles, or keys). If the user instead wants to send traces or run evals, stop and use the deepeval / deepeval-tracing / deepeval-otel skills.
2. Determine the target language — Python or TypeScript — using the Language Detection section above, then read references/<topic>.md and use the code blocks for that language.
3. Configure the client with an Organization API Key. Read references/quickstart.md.
4. Pick the right scope: organization-wide (client.organization()) vs a single project (client.project(project_id)). Read references/introduction.md for scope and capabilities.
5. Read the exact reference for the resource being managed (organization, projects, members and invitations, RBAC, governance policies, or API keys) before writing code, and follow its examples.
6. For RBAC, compose in order: list permissions, bundle them into policies, bundle policies into roles, then assign roles to members. Read references/roles-policies-permissions.md.
7. When creating an API key (or a project, which mints its first key), capture the full secret value immediately — it is shown only once. Read references/api-keys.md.

Core Principles

1. Administration uses an Organization API Key (CONFIDENT_ORG_API_KEY), never a project key. Keep it out of source control and logs.
2. Choose scope deliberately: organization-scoped calls affect the whole account; project-scoped calls (client.project(id)) affect one project.
3. An API key's full secret value is returned only at creation (creating a project also mints its first project key). Store it securely then; later reads are masked.
4. RBAC composes in one direction: permissions → policies → roles → members. Permissions are predefined and read-only; you can only list them.
5. A user must be an organization member before being added to a project. Invitations create membership when accepted.
6. Destructive operations are irreversible — deleting a project permanently removes its datasets, prompts, traces, and evaluations; deleting a key immediately revokes it. Prefer disabling a key (set valid to false) over deleting when you only need to revoke temporarily.
7. Never hardcode secrets or raw sensitive data; prefer environment variables for keys.
8. Governance policies are organization-scoped only and read-only through the SDK — list them and assign/unassign them to projects, but do not expect create/update/delete. (Python only: every method also has an async a_* variant.)
9. Clarify before mutating. For create/update calls, confirm the required inputs and ask about consequential optional ones the user hasn't specified rather than silently omitting them or guessing. In particular: whether a new project should have an owner (email), and which role (role_id) to grant when inviting members or adding them to a project. Ask first, then act.

References

Each topic has one reference file containing both Python and TypeScript examples; use the code block matching the target language.

Topic	File
Install, configure the Organization API Key, first call	references/quickstart.md
Read and rename the organization	references/organization.md
Create, read, update, delete projects	references/projects.md
Members, invitations, and role assignment	references/members-and-invitations.md
RBAC: permissions, policies, and roles	references/roles-policies-permissions.md
Provision, rotate, enable/disable, and delete API keys	references/api-keys.md
Governance policies: list and assign/unassign to projects (org-only)	references/governance.md
Admin SDK overview, org vs project scope, and key capabilities	references/introduction.md
Background: why observability matters for AI products	references/ai-pm-observability.md