Amazon S3 Vectors — S3 Embedding Series — Part 1: Serverless Semantic Search with Amazon S3 Vectors
Introduction
In the age of AI, vector embeddings are at the heart of semantic search, recommendations, and Retrieval-Augmented Generation (RAG). Until now, if you wanted to store and search embeddings, you needed to run a vector database like Qdrant, Pinecone, or Weaviate.
But now, AWS has changed the game.
Meet “Amazon S3 Vectors” — a serverless vector database embedded directly into Amazon S3. It lets you store, index, and query high-dimensional embeddings without managing any infrastructure.
Part of the S3 Embedding Series
Part 1: Introduction to S3 Vectors & Serverless Semantic Search with Amazon S3 Vectors
Part 2: Beyond Text: Storing and Searching Image Embeddings in S3 Vectors *(coming soon)*
Part 3: Serverless Enterprise Search with S3 Vectors, Lambda, and Athena *(coming soon)*
What Is S3 Vectors?
S3 Vectors is a new feature of Amazon S3 that turns your S3 bucket into a vector-capable store. It enables:
- Native storage of vectors (`float32`)
- Automatic indexing
- Semantic similarity search with the `QueryVectors` API
- Metadata filtering
- Seamless integration with Bedrock, SageMaker, and OpenSearch
Why It Matters for AI & RAG
Traditional RAG pipelines need:
- A retriever (typically FAISS or Qdrant)
- An embedding store (local or cloud)
- A generator (LLM)
With S3 Vectors, the retriever and store are merged into S3, meaning:
- No infra to deploy
- Scalable with your storage
- Integrated with your existing S3 data lake
Hands-on Example: PDF Ingestion & Search
- Craete s3 vector bucket in AWS
2. Install dependencies:
pip install boto3 python-dotenv PyPDF2 openai3. Set `.env` variables:
AWS_REGION=us-west-2
VECTOR_BUCKET=your_s3_vector_bucket_name
VECTOR_INDEX=your_s3_vector_index4. Ingest PDFs into S3 Vectors
Use the script from `ingest_vector.py` to:
1. Extract text from each PDF page
2. Chunk into ~500 tokens
3. Generate embeddings (OpenAI or any model)
4. Store in S3 Vector Index with metadata
python ingest_vector.py
import os
import json
from utils import extract_pdf_text, chunk_text, get_embedding
from dotenv import load_dotenv
import boto3
from botocore.exceptions import ClientError
load_dotenv()
VECTOR_BUCKET=your_s3_vector_bucket_name
VECTOR_INDEX=your_s3_vector_index
REGION = "us-west-2"
if not VECTOR_BUCKET or not VECTOR_INDEX:
raise ValueError("S3_VECTOR_BUCKET and S3_VECTOR_INDEX environment variables must be set.")
s3vectors = boto3.client("s3vectors", region_name=REGION)
BATCH_SIZE = 10 # Number of vectors to insert per batch
# Only create the index if it does not exist
print(VECTOR_BUCKET)
def ensure_index_exists():
try:
s3vectors.get_index(
vectorBucketName=VECTOR_BUCKET,
indexName=VECTOR_INDEX
)
print(f"Index '{VECTOR_INDEX}' already exists.")
except ClientError as e:
error_code = e.response['Error']['Code']
if error_code == 'NotFoundException':
s3vectors.create_index(
vectorBucketName=VECTOR_BUCKET,
indexName=VECTOR_INDEX,
dimension=1536, # or your embedding dimension
dataType="float32",
distanceMetric="cosine" # or "Euclidean"
)
print(f"Index '{VECTOR_INDEX}' created!")
else:
raise
def ingest_pdf_to_vector_index(pdf_file):
pdf_path = os.path.join("pdfs", pdf_file)
pages = extract_pdf_text(pdf_path)
vectors = []
for page in pages:
page_num = page["page_number"]
chunks = chunk_text(page["text"])
for idx, chunk in enumerate(chunks):
embedding = get_embedding(chunk)
vector_id = f"{pdf_file}_p{page_num}_c{idx}"
metadata = {
"file_name": pdf_file,
"page_number": str(page_num),
"chunk_index": str(idx),
"chunk_text": chunk[:500] # Truncate for metadata
}
vectors.append({
"key": vector_id,
"data": {"float32": embedding},
"metadata": metadata
})
if len(vectors) == BATCH_SIZE:
s3vectors.put_vectors(
vectorBucketName=VECTOR_BUCKET,
indexName=VECTOR_INDEX,
vectors=vectors
)
vectors = []
# Insert any remaining vectors
if vectors:
s3vectors.put_vectors(
vectorBucketName=VECTOR_BUCKET,
indexName=VECTOR_INDEX,
vectors=vectors
)
print(f"Ingestion to S3 Vector index complete for {pdf_file}.")
if __name__ == "__main__":
ensure_index_exists()
pdf_dir = "pdfs"
for file_name in os.listdir(pdf_dir):
if file_name.lower().endswith(".pdf"):
ingest_pdf_to_vector_index(file_name) You’ll see:
Index ‘your_vector_index’ created!
Ingestion to S3 Vector index complete for report.pdf.
5. Semantic Search via Query
The `search_vector.py` script sends a user query, gets its embedding, and retrieves the top K matches:
python search_vector.py “What is the impact of GenAI on banking?”
import os
import sys
import json
from utils import get_embedding
from dotenv import load_dotenv
import boto3
load_dotenv()
VECTOR_BUCKET=your_s3_vector_bucket_name
VECTOR_INDEX=your_s3_vector_index
REGION = "us-west-2"
if not VECTOR_BUCKET or not VECTOR_INDEX:
raise ValueError("S3_VECTOR_BUCKET and S3_VECTOR_INDEX environment variables must be set.")
s3vectors = boto3.client("s3vectors", region_name=REGION)
def search_vector_index(query: str, top_k: int = 3):
query_emb = get_embedding(query)
response = s3vectors.query_vectors(
vectorBucketName=VECTOR_BUCKET,
indexName=VECTOR_INDEX,
queryVector={"float32": query_emb},
topK=top_k,
returnDistance=True,
returnMetadata=True
)
matches = response.get("vectors", [])
if not matches:
print("No matches found.")
return
for match in matches:
score = match.get("distance")
meta = match.get("metadata", {})
print(f"Score: {score:.3f}\nFile: {meta.get('file_name')} | Page: {meta.get('page_number')} | Chunk: {meta.get('chunk_index')}\nText: {meta.get('chunk_text', '')[:200]}...\n---")
if __name__ == "__main__":
if len(sys.argv) < 2:
print("Usage: python search_vector.py 'your query text'")
sys.exit(1)
query = sys.argv[1]
search_vector_index(query) Example output:
Score: 0.12
File: report1.pdf | Page: 3 | Chunk: 1
Text: Generative AI is redefining banking by…
Qdrant vs. S3 Vectors: Feature
When to Use S3 Vectors
Use S3 Vectors if you want:
- Document/semantic search at scale
- Serverless architecture (no DBs to maintain)
- Integration with AWS-native workflows
- Simplicity over complexity
When Qdrant Makes Sense
Use Qdrant if you need:
- Real-time response for interactive apps
- Score filtering, complex query logic
- Open-source extensibility and custom logic
- AI experiments needing full vector query control
Hybrid Strategy (Best of Both Worlds)
You can even combine the two:
- S3 Vectors as cold, scalable embedding store
- Qdrant as real-time hot index for frequently queried data
Final Thoughts
Amazon S3 Vectors is a big leap forward for AI developers building RAG, search, and assistant systems. By embedding vector search into storage, AWS makes semantic retrieval as simple as uploading to S3.
For many teams, it eliminates the need to manage another database — and for others, it becomes a perfect long-term backend for embedding-driven systems.
Resources
- [Official AWS S3 Vectors Documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-vectors.html)
Contributors
This article was written by Mousumi Bakshi (Senior Solutions Architect)
Let’s Build the Future Together!
For your Generative AI solutions and project development needs, get in touch with MSS — we’re here to help bring your ideas to life.
👉 Follow us on LinkedIn and Medium for weekly tech insights and practical tips to supercharge your development journey!
