From aws-data-analytics
Stores and queries vector embeddings using Amazon S3 Vectors for cost-effective long-term storage in RAG, semantic search, similarity search, and low-QPS workloads.
npx claudepluginhub aws/agent-toolkit-for-aws --plugin aws-data-analyticsThis skill uses the workspace's default tool permissions.
Amazon S3 Vectors is a cost-effective AWS service for storing and querying vector embeddings at scale. Optimized for long-term storage with subsecond latency for cold queries, as low as 100ms for warm queries.
Implements Cloudflare Vectorize vector database for semantic search, RAG, and AI apps on Workers. Manages indexes, vector CRUD, metadata filtering, embeddings from Workers AI or OpenAI.
Guides vector database selection for embeddings and semantic search, compares managed options like Pinecone and self-hosted like pgvector/Milvus, explains ANN algorithms like HNSW.
Creates, manages, and queries Databricks Vector Search endpoints and indexes with filters and embeddings for RAG, semantic search, and similarity matching.
Share bugs, ideas, or general feedback.
Amazon S3 Vectors is a cost-effective AWS service for storing and querying vector embeddings at scale. Optimized for long-term storage with subsecond latency for cold queries, as low as 100ms for warm queries.
"Using S3 Vectors with OpenSearch Service".references/limits-and-patterns.md.For latest guidance, search AWS docs for "S3 Vectors best practices".
Classify the request before starting:
references/limits-and-patterns.md first, then Steps 2-6You MUST execute commands using AWS MCP server tools when connected. Fall back to AWS CLI only if AWS MCP is unavailable. You MUST explain each step to the user before executing.
Constraints:
You MUST confirm bucket name with user. Names: 3-63 chars, lowercase letters, numbers, hyphens only. Encryption (SSE-S3 default or SSE-KMS for compliance) is immutable after creation.
aws s3vectors create-vector-bucket \
--vector-bucket-name <BUCKET_NAME>
Constraints:
kms:GenerateDataKey and kms:Decrypt to the S3 Vectors service principal indexing.s3vectors.amazonaws.com. You MUST use full KMS key ARN (not alias). See references/limits-and-patterns.md for command example.Every parameter is immutable after creation.
Pre-flight checklist (confirm ALL with user):
cosine or euclidean. Use embedding model's recommended metric;"S3 Vectors Bedrock Knowledge Bases prerequisites" to get the required key names.aws s3vectors create-index \
--vector-bucket-name <BUCKET_NAME> \
--index-name <INDEX_NAME> \
--dimension <DIM> \
--distance-metric <cosine|euclidean> \
--data-type float32 \
--metadata-configuration '{"nonFilterableMetadataKeys":["<KEY1>","<KEY2>"]}'
Omit --metadata-configuration if no non-filterable keys are needed.
Index names: 3-63 chars, lowercase, numbers, hyphens, dots. Unique within bucket. Filterable metadata: 2 KB limit. Total metadata (filterable + non-filterable combined): 40 KB. See references/metadata-filtering.md.
Skip to Step 5 (store) or Step 6 (query) if user already has embeddings.
Constraints:
Generate embeddings with Bedrock invoke-model:
aws bedrock-runtime invoke-model \
--model-id <MODEL_ID> \
--content-type application/json \
--cli-binary-format raw-in-base64-out \
--body '{"inputText": "your text"}' \
invoke-model-output.json
You MUST use --cli-binary-format raw-in-base64-out for CLI v2. Output file is required for CLI. The response key is model-dependent (e.g., embedding for Titan, embeddings for Cohere). For Titan, parse with json.load(open('invoke-model-output.json'))['embedding']. Use embedding array as float32 in put-vectors or query-vectors. For batch embedding generation, use AWS SDK or CLI.
aws s3vectors put-vectors \
--vector-bucket-name <BUCKET_NAME> \
--index-name <INDEX_NAME> \
--vectors '[{"key":"<ID>","data":{"float32":[<EMBEDDING>]},"metadata":{"topic":"science"}}]'
Constraints:
429 TooManyRequestsExceptionreferences/limits-and-patterns.md for batch patternsGenerate embedding if needed (Step 4), then query:
aws s3vectors query-vectors \
--vector-bucket-name <BUCKET_NAME> \
--index-name <INDEX_NAME> \
--query-vector '{"float32":[<EMBEDDING>]}' \
--top-k 10 \
--return-distance
Optional: add --return-metadata and/or --filter '{"topic":{"$eq":"science"}}' (both require GetVectors permission). See references/metadata-filtering.md.
Example response body: {"vectors": [{"key": "id1", "distance": 0.45, "metadata": {"topic": "science"}}, ...], "distanceMetric": "cosine"}
Constraints:
--filter or --return-metadata requires both s3vectors:QueryVectors AND s3vectors:GetVectors IAM permissions. Without GetVectors, these options return 403.| Error | Cause | Fix |
|---|---|---|
DimensionMismatch | Dims don't match index | Use matching model, or delete/recreate index (confirm with user -- destroys all vectors). |
403 Forbidden with --filter or --return-metadata | Missing s3vectors:GetVectors | Add s3vectors:GetVectors to IAM policy. |
Fewer results than --top-k | Few vectors match filter | Expected -- filtering is inline. Broaden filter. |
429 TooManyRequestsException | Exceeded per-index rate limits | Retry with backoff. Shard across indexes for sustained throughput. Search AWS docs for "S3 Vectors limitations and restrictions" for current limits. |
AccessDeniedException | Missing s3vectors:* IAM actions | S3 Vectors uses s3vectors:* namespace, not s3:*. Update IAM policy. |
RequestTimeoutException or service unavailable | Request timeout or region not supported | Retry request. For regional availability, search AWS docs for "S3 Vectors limitations and restrictions". |