analyzing-database-indexes

Database Index Advisor

Overview

Analyze database index usage, identify missing indexes causing sequential scans, detect redundant or unused indexes wasting write performance, and recommend optimal index configurations for PostgreSQL and MySQL.

Prerequisites

• Database credentials with access to pg_stat_user_indexes, pg_stat_user_tables, and pg_stat_statements (PostgreSQL) or performance_schema and sys schema (MySQL)
• pg_stat_statements extension enabled for PostgreSQL query statistics
• psql or mysql CLI for executing analysis queries
• Representative workload running (analysis during off-peak hours may miss important query patterns)
• At least 24 hours of statistics accumulation since the last pg_stat_reset()

Instructions

1. Identify tables with high sequential scan activity (candidates for missing indexes):

   • PostgreSQL: SELECT relname, seq_scan, seq_tup_read, idx_scan, n_live_tup FROM pg_stat_user_tables WHERE seq_scan > 100 AND n_live_tup > 10000 ORDER BY seq_tup_read DESC LIMIT 20
   • A table with high seq_scan count and high seq_tup_read relative to n_live_tup is scanning most of the table repeatedly

2. Find the queries causing sequential scans by correlating with pg_stat_statements:

   • SELECT query, calls, mean_exec_time, rows FROM pg_stat_statements WHERE query ILIKE '%table_name%' ORDER BY mean_exec_time DESC LIMIT 10
   • Run EXPLAIN (ANALYZE, BUFFERS) on the top queries to confirm sequential scan usage

3. Analyze query WHERE clauses and JOIN conditions to determine which columns need indexes. Extract the filtering columns and their selectivity:

   • SELECT column_name, n_distinct, correlation FROM pg_stats WHERE tablename = 'target_table'
   • High n_distinct (close to row count) indicates good index selectivity
   • correlation close to 1.0 or -1.0 suggests the column benefits from a B-tree index

4. Recommend composite indexes for multi-column queries. Follow the equality-first, range-second ordering:

   • Place columns used with = operators first in the index
   • Place columns used with >, <, BETWEEN, or LIKE 'prefix%' last
   • Example: WHERE status = 'active' AND created_at > '2024-01-01' -> CREATE INDEX ON orders (status, created_at)

5. Identify unused indexes wasting write performance:

   • PostgreSQL: SELECT indexrelname, idx_scan, pg_size_pretty(pg_relation_size(indexrelid)) AS index_size FROM pg_stat_user_indexes WHERE idx_scan = 0 AND indexrelname NOT LIKE '%pkey' ORDER BY pg_relation_size(indexrelid) DESC
   • Indexes with zero scans over a representative period are candidates for removal (verify they are not used by foreign key constraints or unique enforcement)

6. Detect redundant indexes where one index is a prefix of another:

   • A single-column index on (customer_id) is redundant if a composite index on (customer_id, created_at) exists, because the composite index serves both single-column and multi-column queries
   • Generate DROP INDEX recommendations for the redundant subset indexes

7. Evaluate partial indexes for filtered queries. If a query always filters WHERE status = 'active':

   • CREATE INDEX idx_orders_active ON orders (created_at) WHERE status = 'active'
   • Partial indexes are smaller and faster than full indexes when the filter eliminates most rows

8. Consider covering indexes (INCLUDE clause in PostgreSQL 11+) for index-only scans:

   • CREATE INDEX idx_orders_covering ON orders (customer_id, created_at) INCLUDE (total_amount, status)
   • The INCLUDE columns are stored in the index leaf pages, enabling index-only scans without heap access

9. Estimate the impact of each recommendation:

   • Index size: SELECT pg_size_pretty(pg_relation_size('index_name')) for existing similar indexes
   • Write overhead: each additional index adds approximately 5-15% write latency per INSERT/UPDATE
   • Read improvement: compare EXPLAIN plans with and without the proposed index

10. Generate a prioritized recommendations report with CREATE INDEX and DROP INDEX statements, estimated storage impact, expected query improvement, and write overhead trade-off analysis.

Output

• Missing index recommendations as ready-to-execute CREATE INDEX statements with CONCURRENTLY option
• Unused index report with DROP INDEX candidates and their storage savings
• Redundant index report identifying prefix-overlapping indexes
• Index usage statistics showing scan counts, tuple reads, and sizes for all indexes
• Impact analysis estimating read improvement vs. write overhead for each recommendation

Error Handling

Error	Cause	Solution
pg_stat_statements not available	Extension not installed	CREATE EXTENSION pg_stat_statements and add to shared_preload_libraries
Index creation blocks writes	CREATE INDEX acquires exclusive lock on the table	Use CREATE INDEX CONCURRENTLY which does not block writes (takes longer but safe for production)
Index not used after creation	Statistics not updated or query planner choosing sequential scan	Run ANALYZE table_name; check random_page_cost setting (reduce to 1.1 for SSD); verify query uses indexed columns without functions
Statistics reset unexpectedly	pg_stat_reset() called or database restart cleared stats	Wait 24-48 hours for statistics to accumulate; set up periodic stats collection to a metrics table
Too many indexes on write-heavy table	Each INSERT/UPDATE must update all indexes	Target 5-7 indexes per table maximum; use composite indexes to replace multiple single-column indexes; remove unused indexes

Examples

Identifying a missing composite index for an API endpoint: The /orders?customer_id=123&status=active endpoint takes 2 seconds. Analysis shows the orders table (5M rows) has indexes on (id) and (customer_id) but not (customer_id, status). The query filters on both columns. Adding CREATE INDEX CONCURRENTLY idx_orders_customer_status ON orders (customer_id, status) reduces the query to 5ms.

Cleaning up 8 unused indexes saving 12GB: Index usage analysis reveals 8 indexes with zero scans over 30 days, totaling 12GB of storage. After confirming none are used for FK enforcement or unique constraints, dropping them reduces write latency by 18% and frees disk space. Command: DROP INDEX CONCURRENTLY idx_name.

Replacing 3 single-column indexes with 1 composite covering index: Table has separate indexes on (user_id), (created_at), and (status). Most queries filter on all three. A single composite index (user_id, status, created_at) INCLUDE (amount) replaces all three, reduces total index storage by 40%, and enables index-only scans for the dashboard query.

Resources

• PostgreSQL index types: https://www.postgresql.org/docs/current/indexes.html
• Use The Index, Luke: https://use-the-index-luke.com/
• pg_stat_user_indexes: https://www.postgresql.org/docs/current/monitoring-stats.html#MONITORING-PG-STAT-ALL-INDEXES-VIEW
• MySQL index optimization: https://dev.mysql.com/doc/refman/8.0/en/optimization-indexes.html
• Dexter (automatic PostgreSQL index advisor): https://github.com/ankane/dexter