Lumen AI Skill

Overview

Lumen AI is an open-source, agent-based framework for conversational data exploration. Users ask questions in plain English and receive visualizations, SQL queries, and insights automatically generated by large language models.

Key Features

Natural Language Interface: Ask questions in plain English
Multi-LLM Support: OpenAI, Anthropic, Google, Mistral, local models
Agent Architecture: Specialized agents for SQL, charts, analyses
Extensible: Custom agents, tools, and analyses
Privacy-Focused: Full local deployment option

When to Use Lumen AI

Feature	Lumen AI	Lumen Dashboards
Interface	Conversational	Declarative YAML
Use Case	Ad-hoc exploration	Fixed dashboards
Users	Non-technical	Developers

Use Lumen AI when: Users need ad-hoc exploration, questions vary unpredictably, enabling self-service analytics.

Use Lumen Dashboards when: Dashboard structure is fixed, no LLM costs desired.

Quick Start

Installation

pip install lumen[ai]
pip install openai  # or anthropic for Claude

Launch Built-in Interface

export OPENAI_API_KEY="sk-..."
lumen-ai serve data/sales.csv
# Or with database
lumen-ai serve "postgresql://user:pass@localhost/mydb"

Python API

import lumen.ai as lmai
import panel as pn
from lumen.sources.duckdb import DuckDBSource

pn.extension()

# Configure LLM
lmai.llm.llm_type = "anthropic"
lmai.llm.model = "claude-3-5-sonnet-20241022"

# Load data
source = DuckDBSource(tables=["./data/sales.csv"])

# Create UI
ui = lmai.ExplorerUI(source=source, title="Sales Analytics AI")
ui.servable()

Example Queries

"What tables are available?"
"Show me total sales by region"
"Create a scatter plot of price vs quantity"
"What were the top 10 products last month?"

Core Concepts

1. Agents

Specialized components that handle specific tasks:

TableListAgent: Shows available tables and schemas
ChatAgent: General conversation and summaries
SQLAgent: Generates and executes SQL queries
hvPlotAgent: Creates interactive visualizations
VegaLiteAgent: Publication-quality charts
AnalysisAgent: Custom domain-specific analyses

See: Built-in Agents Reference

2. LLM Providers

Use Case	Provider	Model
Production	OpenAI	gpt-4o
Complex SQL	Anthropic	claude-3-5-sonnet
High volume	OpenAI	gpt-4o-mini
Sensitive data	Ollama	llama3.1

See: LLM Provider Configuration

3. Memory and Tools

Agents share a memory system for context persistence. Extend capabilities with tools:

DocumentLookup: RAG for document context
TableLookup: Schema and metadata access

Common Patterns

Pattern 1: Basic Analytics Interface

import lumen.ai as lmai
from lumen.sources.duckdb import DuckDBSource

lmai.llm.llm_type = "openai"
lmai.llm.model = "gpt-4o"

source = DuckDBSource(tables=["sales.csv"])
ui = lmai.ExplorerUI(source=source, title="Business Analytics")
ui.servable()

Pattern 2: With Document Context (RAG)

source = DuckDBSource(
    tables=["sales.csv", "products.parquet"],
    documents=["./docs/data_dictionary.pdf", "./docs/business_rules.md"]
)

ui = lmai.ExplorerUI(source=source, tools=[lmai.tools.DocumentLookup])

Pattern 3: Custom Agent

from lumen.ai.agents import Agent
import param

class SentimentAgent(Agent):
    """Analyze sentiment in text data."""
    requires = param.List(default=["current_source"])
    provides = param.List(default=["sentiment_analysis"])

    purpose = """
    Analyzes sentiment in text columns.
    Keywords: sentiment, emotion, positive, negative, tone
    """

    async def respond(self, query: str):
        source = self.memory["current_source"]
        yield "Sentiment analysis results..."

ui = lmai.ExplorerUI(source=source, agents=[SentimentAgent, lmai.agents.ChatAgent])

See: Custom Agents Guide

Pattern 4: Custom Analysis

from lumen.ai.analyses import Analysis
from lumen.pipeline import Pipeline
import param

class CohortAnalysis(Analysis):
    """Customer cohort retention analysis."""
    columns = param.List(default=['customer_id', 'signup_date', 'purchase_date'])

    def __call__(self, pipeline: Pipeline):
        df = pipeline.data
        # ... calculate cohorts ...
        return results

ui = lmai.ExplorerUI(
    source=source,
    agents=[lmai.agents.AnalysisAgent(analyses=[CohortAnalysis])]
)

Pattern 5: Multi-Source Data

source = DuckDBSource(
    tables={
        "sales": "./data/sales.parquet",
        "customers": "./data/customers.csv",
        "products": "https://data.company.com/products.csv"
    }
)
ui = lmai.ExplorerUI(source=source)

Configuration

Agent Selection

agents = [
    lmai.agents.TableListAgent,
    lmai.agents.SQLAgent,
    lmai.agents.hvPlotAgent,
]
ui = lmai.ExplorerUI(source=source, agents=agents)

Coordinator Types

# DependencyResolver (default): Recursively resolves agent dependencies
ui = lmai.ExplorerUI(source=source, coordinator="dependency")

# Planner: Creates execution plan upfront
ui = lmai.ExplorerUI(source=source, coordinator="planner")

UI Customization

ui = lmai.ExplorerUI(
    source=source,
    title="Custom Analytics AI",
    accent_color="#00aa41",
    suggestions=["Show me revenue trends", "What are the top products?"]
)

Best Practices

1. Security

import os

# Environment variables for API keys
lmai.llm.api_key = os.getenv("OPENAI_API_KEY")

# Never hardcode secrets

2. Performance

# Limit table sizes for exploration
source = DuckDBSource(
    tables=["large_table.parquet"],
    table_kwargs={"large_table": {"nrows": 100000}}
)

3. User Experience

# Provide example queries
ui = lmai.ExplorerUI(
    source=source,
    suggestions=["Show me revenue trends", "Top 10 products by sales"]
)

Deployment

Development

lumen-ai serve app.py --autoreload --show

Production

panel serve app.py \
  --port 80 \
  --num-procs 4 \
  --allow-websocket-origin=analytics.company.com

Docker

FROM python:3.11-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY app.py data/ ./
CMD ["panel", "serve", "app.py", "--port", "5006", "--address", "0.0.0.0"]

See: Deployment Guide

Troubleshooting

LLM Not Responding

# Test API connection
curl https://api.openai.com/v1/models -H "Authorization: Bearer $OPENAI_API_KEY"

Agent Not Selected

# Debug which agent was selected
print(ui.agent_manager.last_selected_agent)

# View agent purposes
for agent in ui.agents:
    print(f"{agent.__class__.__name__}: {agent.purpose}")

SQL Generation Errors

Add data dictionary as document for context
Provide example queries in agent prompts
Check table schemas match query expectations

Resources

Reference Documentation

LLM Provider Configuration - Setup for all providers
Built-in Agents Reference - Complete agent documentation
Custom Agents Guide - Build specialized agents
Deployment Guide - Production deployment

External Links

Summary

Lumen AI transforms data exploration through natural language interfaces powered by LLMs.

Strengths: No SQL required for users, flexible LLM support, extensible architecture, privacy-focused options.

Ideal for: Ad-hoc exploration, non-technical users, rapid insights, self-service analytics.

Consider alternatives when:

Fixed dashboards needed → Lumen Dashboards
No LLM budget → Traditional BI tools
Highly custom logic → Panel Dashboards

Related Skills

Lumen Dashboards - Declarative YAML dashboards
Panel Dashboards - Interactive dashboard development
Plotting Fundamentals - Quick plotting with hvPlot

lumen-ai