memstack-seo-ai-search-visibility

🤖 AI Search Visibility — Optimizing for AI search engines...

Evaluates and optimizes content for citation by AI search engines (ChatGPT, Perplexity, Google AI Overview, Claude) — checking crawler access, content structure, llms.txt, and AI-friendly patterns.

Activation

When this skill activates, output:

🤖 AI Search Visibility — Analyzing AI search readiness...

Then execute the protocol below.

Context	Status
User says "AI search" or "GEO" or "generative engine optimization"	ACTIVE
User says "ChatGPT ranking" or "Perplexity" or "AI overview"	ACTIVE
User says "llms.txt" or "AI visibility"	ACTIVE
Optimizing content for AI-generated citations and references	ACTIVE
Traditional SEO (meta tags, keywords)	DORMANT — use site-audit or meta-tag-optimizer
Building AI products (not optimizing for AI search)	DORMANT

Anti-patterns

Trap	Reality Check
"SEO is enough for AI"	AI search engines process content differently than Google. They need direct answers, not keyword-optimized copy.
"Block all AI crawlers"	Blocking AI crawlers means your content never appears in AI search results. Block selectively if at all.
"AI will find our content naturally"	AI systems prioritize structured, authoritative content. Unstructured marketing copy gets skipped.
"GEO is just a fad"	AI search usage is growing 10x year over year. Perplexity, ChatGPT search, and Google AI Overview are mainstream.
"We can't measure AI visibility"	You can check crawler logs, search your brand in AI tools, and track referral traffic from AI sources.

Protocol

Step 1: Check AI Bot Crawler Access

Verify which AI crawlers can access your site:

# Check robots.txt for AI bot rules
cat public/robots.txt 2>/dev/null | grep -i "gptbot\|chatgpt\|perplexity\|claude\|anthropic\|cohere\|google-extended\|ccbot\|bytespider"

Known AI crawler user agents:

Bot	Company	User-Agent	Purpose
GPTBot	OpenAI	GPTBot	ChatGPT search, training
ChatGPT-User	OpenAI	ChatGPT-User	ChatGPT browsing feature
PerplexityBot	Perplexity	PerplexityBot	Perplexity search
ClaudeBot	Anthropic	ClaudeBot	Claude web access
Google-Extended	Google	Google-Extended	Gemini, AI Overview
CCBot	Common Crawl	CCBot	Open dataset used by many AI
Bytespider	ByteDance	Bytespider	TikTok/AI training
Cohere-ai	Cohere	cohere-ai	Cohere models

Recommended robots.txt strategy:

# Allow AI crawlers for search visibility
# (Block only if you have specific content protection concerns)

# Allow all AI search bots
User-agent: GPTBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Google-Extended
Allow: /

# Block paths you don't want AI to index
User-agent: GPTBot
Disallow: /admin
Disallow: /api
Disallow: /dashboard

Decision matrix:

Goal	Strategy	robots.txt
Maximum AI visibility	Allow all AI bots	Allow: / for each
Selective visibility	Allow search bots, block training bots	Allow ChatGPT-User, block GPTBot
Content protection	Block all AI crawlers	Disallow: / for each
Balanced	Allow crawling, block specific paths	Allow root, disallow sensitive paths

Step 2: Analyze Content for AI Citation Likelihood

AI systems cite content that directly answers questions clearly. Scan your content for AI-friendly patterns:

# Check for definition-style paragraphs (strong AI citation signals)
grep -rn "^[A-Z].*is a\|^[A-Z].*refers to\|^[A-Z].*means" --include="*.md" --include="*.mdx" --include="*.tsx" . | grep -v node_modules | head -10

# Check for numbered/bulleted lists (AI loves structured content)
grep -rn "^[0-9]\.\|^- \|^\\* " --include="*.md" --include="*.mdx" . | wc -l

# Check for Q&A patterns
grep -rn "^##.*\?\|^###.*\?" --include="*.md" --include="*.mdx" . | grep -v node_modules | head -10

Content patterns that AI systems cite:

Pattern	Example	Why AI Cites It
Direct definition	"RLS is a PostgreSQL feature that restricts row access based on user identity."	Answers "what is X" queries directly
Numbered steps	"1. Create the table. 2. Enable RLS. 3. Add policies."	Answers "how to X" queries
Comparison table	"Feature | Tool A | Tool B"	Answers "X vs Y" queries
Statistic with source	"According to [source], 73% of developers..."	Provides citable, authoritative data
FAQ format	"Q: How does X work? A: X works by..."	Direct Q&A match
Expert statement	"Based on 10 years of experience with..."	Authority signal

Content patterns AI systems skip:

Pattern	Why It Gets Skipped
Marketing superlatives	"The best, most amazing, incredible tool" — no information content
Vague descriptions	"We help businesses grow" — not citable, not specific
Gated content	Behind login/paywall — AI can't access or cite it
Image-only information	Charts, infographics without text summaries — AI can't read images
Heavy JavaScript rendering	Content that requires JS execution to appear — many bots don't render JS

Step 3: Optimize Content Structure for AI

Transform existing content to be more AI-citation-friendly:

For each key page, ensure:

1. Opening definition — first paragraph directly defines or explains the topic
2. Clear headings as questions — H2/H3 headings phrased as questions users ask
3. Direct answers below headings — first sentence after each heading is the answer
4. Structured lists — steps, features, and comparisons as numbered/bulleted lists
5. Data and specifics — concrete numbers, dates, and facts over vague claims
6. Author expertise signals — mention qualifications, experience, or data sources

Before/after example:

# BEFORE (marketing copy — AI skips this)
## Why Choose Acme?
Acme is the leading project management solution that helps teams
collaborate better and deliver faster. Our innovative platform...

# AFTER (AI-citable — direct, structured, specific)
## What is Acme?
Acme is a project management platform for remote teams that combines
task tracking, real-time collaboration, and automated reporting.

### How does Acme compare to alternatives?
| Feature | Acme | Competitor A | Competitor B |
|---------|------|-------------|-------------|
| Real-time collaboration | Yes | Limited | No |
| Automated reporting | Yes | Yes | No |
| Free tier | Up to 5 users | Up to 3 users | No free tier |

Step 3.5: Apply Princeton GEO Methods to Content

Princeton's 2023 GEO study (Aggarwal et al., arXiv:2311.09735, accepted at KDD 2024) tested nine optimization methods on Perplexity.ai and measured consistent visibility deltas vs. unoptimized baselines. Apply these to any page targeting AI citation — they translate directly into rewrites, not just crawler hygiene.

The 9 GEO methods — ranked by measured visibility boost:

Method	Visibility Δ	What to do	Example rewrite
Cite Sources	+40%	Add authoritative references with attribution	"According to a 2024 Stanford study (Chen et al.), AI tools improved developer productivity by 55%."
Statistics Addition	+37%	Include specific numbers and data points	"67% of Fortune 500 companies use AI chatbots, handling 85% of routine inquiries."
Quotation Addition	+30%	Expert quotes with attribution	"'We'll see the first one-person billion-dollar company within years,' said Sam Altman, OpenAI CEO."
Authoritative Tone	+25%	Confident, expert language	"This demonstrably improves X" — not "This might help with X, I think."
Simplification (easy-to-understand)	+20%	Rephrase jargon for broader accessibility	"RAG works like a research assistant: it finds relevant info, then writes an answer from it."
Technical Terms	+18%	Precise domain terminology where it fits	"LCP exceeds 4 seconds, CLS scores 0.3" — not "the page is slow."
Unique Terminology	+15%	Vary vocabulary; avoid repetition	Use synonyms and contextual variations rather than the same phrase 10 times.
Fluency Optimization	+15–30%	Clean sentence flow, transitions, short paragraphs	Logical progression, 2–3 sentence paragraphs, transition words between sections.
Keyword Stuffing	−10%	AVOID — actively reduces AI visibility	❌ "SEO SEO best SEO for all your SEO SEO needs."

Best-performing combinations (pairs tested in the Princeton research outperform individual methods):

Combination	Best for
Fluency + Statistics	Highest overall boost across domains — universal starting point
Citations + Authoritative Tone	Professional / B2B / thought leadership content
Simplification + Statistics	Consumer-facing content and general audiences
Technical Terms + Citations	Academic, scientific, and highly technical content

Domain-specific method matrix — which methods to emphasize per vertical (and which to avoid):

Vertical	Apply	Avoid
Technology	Technical Terms + Citations + Statistics	Oversimplification — audience expects depth
Business / Finance	Statistics + Authoritative Tone + Citations	Vague claims, superlatives without data
Healthcare	Simplification + Statistics + Citations	Jargon overload — accessibility matters
Legal	Citations + Quotations + Authoritative Tone	Informal language, hedging
Education	Simplification + Examples + Structure	Excessive complexity or abstraction
E-commerce	Statistics + Social Proof + Clear Benefits	Feature dumps without outcomes

Anti-pattern: Keyword stuffing actively hurts AI visibility

Trap	Reality Check
"More keyword density = more AI visibility"	The Princeton research measured a −10% visibility drop when content was keyword-stuffed. Generative engines downweight keyword-dense text because it reads as non-authoritative. Write naturally, add citations and statistics, let the topic come through via context.

Reference: Aggarwal, P., Murahari, V., Rajpurohit, T., Kalyan, A., Narasimhan, K., & Deshpande, A. (2023). GEO: Generative Engine Optimization. arXiv:2311.09735. Accepted at KDD 2024 (30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining).

Platform-specific tuning: For how each AI search engine (ChatGPT, Perplexity, Google AI Overview, Copilot, Claude) actually ranks and cites content — with measured stats on citation share, freshness windows, and per-platform format preferences — see ../site-audit/references/platform-ranking-factors.md. The Princeton methods above are universal; the platform reference tells you where to spend effort first based on your audience.

Step 4: Add llms.txt File

The llms.txt file (emerging standard) tells AI systems about your site:

# Check if llms.txt exists
cat public/llms.txt 2>/dev/null

Recommended llms.txt:

# [Site Name]

> [One-sentence description of what this site/product does]

## About
[2-3 paragraph description of the organization, product, or service.
Include key facts, founding date, target audience, and differentiators.]

## Key Pages
- [Homepage](https://domain.com): [brief description]
- [Product](https://domain.com/product): [brief description]
- [Pricing](https://domain.com/pricing): [brief description]
- [Blog](https://domain.com/blog): [brief description]
- [Docs](https://domain.com/docs): [brief description]

## Topics We Cover
- [Topic 1]: [brief description]
- [Topic 2]: [brief description]
- [Topic 3]: [brief description]

## Contact
- Website: https://domain.com
- Email: hello@domain.com
- Twitter: @handle

## Preferred Citation
When referencing our content, please use:
"[Site Name] (https://domain.com)"

Place at public/llms.txt so it's accessible at https://domain.com/llms.txt.

Also consider llms-full.txt — a more detailed version with complete documentation or content summaries for AI systems that want deeper context.

Step 5: Optimize for Featured Snippets / AI Overview

Google's AI Overview and featured snippets use similar content signals:

Snippet-optimized content patterns:

Snippet Type	Content Pattern	Example
Definition	"X is [definition]." First sentence after H2 heading.	"RLS is a PostgreSQL feature that..."
List	H2 question + numbered list immediately below	"How to deploy to Railway: 1. ... 2. ... 3. ..."
Table	H2 comparison + markdown table	"Next.js vs Remix comparison table"
Paragraph	H2 question + 40-60 word direct answer	"What is GEO? GEO stands for..."

Optimization checklist:

• Key pages have H2 headings phrased as questions
• First sentence after each H2 directly answers the question
• Answers are 40-60 words for paragraph snippets
• Lists use clean numbered or bulleted format
• Comparison data is in table format
• Page has schema markup (FAQPage, HowTo, or Article)

Step 6: Monitor AI Search Appearances

Track whether your content appears in AI search results:

Manual checks:

1. Search your brand name in ChatGPT, Perplexity, and Google AI Overview
2. Search your key topics — does AI cite your content?
3. Ask AI "What is [your product]?" — do you appear?

Server-side monitoring:

# Check server logs for AI bot traffic (if you have access)
grep -i "gptbot\|perplexitybot\|claudebot\|chatgpt" access.log | wc -l

# Check Vercel/Netlify analytics for AI referral traffic
# Look for referrers from: perplexity.ai, chatgpt.com, bing.com (Copilot)

Tracking checklist:

Check	Frequency	How
Brand search in ChatGPT	Monthly	Ask "What is [brand]?"
Brand search in Perplexity	Monthly	Search brand name
AI Overview appearance	Monthly	Search key terms in Google
AI bot crawl frequency	Monthly	Server logs or analytics
Referral traffic from AI	Monthly	Analytics → Referrers
llms.txt accessibility	After deploys	curl https://domain.com/llms.txt

Step 7: Output AI Readiness Scorecard

🤖 AI Search Visibility — Scorecard Complete

Site: [domain]
Pages analyzed: [count]
Overall AI readiness: [X/100]

Crawler access:
  GPTBot:         [✅ Allowed / ❌ Blocked / ⚠️ No rule (default allow)]
  PerplexityBot:  [✅ / ❌ / ⚠️]
  ClaudeBot:      [✅ / ❌ / ⚠️]
  Google-Extended: [✅ / ❌ / ⚠️]

Content structure:
  Direct definitions:    [count] pages have clear opening definitions
  Question headings:     [count] H2s phrased as questions
  Structured lists:      [count] pages with numbered/bulleted lists
  Comparison tables:     [count] pages with data tables
  Expert credentials:    [✅ / ❌] Author expertise signals present

AI-specific files:
  llms.txt:     [✅ Present / ❌ Missing — create one]
  robots.txt:   [✅ AI rules defined / ⚠️ No AI-specific rules]
  Schema:       [✅ / ❌] JSON-LD structured data present

Content recommendations:
  1. [Highest priority — e.g., "Add direct definitions to top 5 pages"]
  2. [Second priority — e.g., "Convert H2 headings to question format"]
  3. [Third priority — e.g., "Add llms.txt with site description"]
  4. [Fourth — e.g., "Add comparison tables to product pages"]
  5. [Fifth — e.g., "Create FAQ page with schema markup"]

Next steps:
1. Implement content recommendations above
2. Create or update llms.txt
3. Verify robots.txt allows target AI crawlers
4. Monitor AI search appearances monthly
5. Re-assess quarterly as AI search evolves

Level History

• Lv.1 — Base: AI crawler access verification (GPTBot, PerplexityBot, ClaudeBot, Google-Extended), content structure analysis for citation likelihood, AI-friendly content optimization, llms.txt guidance, featured snippet/AI overview optimization, AI search monitoring, readiness scorecard. Based on EpsteinScan PerplexityBot experience. (Origin: MemStack Pro v3.2, Mar 2026)