Python is the default language for AI tooling today, not because it’s the fastest, but because its ecosystem (sentence-transformers, FAISS, FastAPI, LangChain) lets you go from idea to working prototype in an afternoon. This guide covers three things you’ll actually build: a Retrieval-Augmented Generation (RAG) pipeline, a FastAPI endpoint to serve it, and a simple autonomous agent loop that calls tools. Every snippet below runs as-is with pip-installable packages.
Building a RAG Pipeline from Scratch
RAG solves a specific problem: a language model only knows what it was trained on, so it can’t answer questions about your own documents, your product docs, or anything written after its training cutoff. RAG fixes this by retrieving relevant text chunks first, then feeding them to the model as context.
The three moving parts are: splitting documents into chunks, embedding those chunks into vectors, and searching those vectors at query time. Here’s a minimal but complete version using sentence-transformers for embeddings and faiss-cpu for the vector index — both run locally, no API key required:
pip install sentence-transformers faiss-cpu numpy
import faiss
import numpy as np
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("all-MiniLM-L6-v2") # 384-dim, fast, good default
documents = [
"Flutter uses a reactive widget tree; every state change rebuilds affected widgets.",
"Kotlin coroutines run on a thread pool managed by a CoroutineDispatcher.",
"FastAPI validates request bodies automatically using Pydantic models.",
"FAISS IndexFlatL2 does exact nearest-neighbor search using Euclidean distance.",
]
embeddings = model.encode(documents, convert_to_numpy=True)
dimension = embeddings.shape[1] # 384 for this model
index = faiss.IndexFlatL2(dimension)
index.add(embeddings.astype(np.float32))
def retrieve(query: str, k: int = 2) -> list[str]:
query_vector = model.encode([query], convert_to_numpy=True).astype(np.float32)
distances, indices = index.search(query_vector, k)
return [documents[i] for i in indices[0]]
results = retrieve("How does FastAPI validate input?")
print(results)
# ["FastAPI validates request bodies automatically using Pydantic models.", ...]
For real documents, replace the hardcoded documents list with a chunker: split text into 300–500 token windows with roughly 50 tokens of overlap, so a fact that straddles a chunk boundary still gets retrieved. IndexFlatL2 is exact search and fine up to a few hundred thousand vectors; past that, switch to an approximate index like IndexIVFFlat to keep query latency low.
Serving It with FastAPI
Once retrieval works, wrap it in an API so a mobile app or frontend can call it. FastAPI’s automatic request validation (via Pydantic) and auto-generated OpenAPI docs make this fast to stand up:
pip install fastapi uvicorn
from fastapi import FastAPI
from pydantic import BaseModel
app = FastAPI(title="RAG Search API")
class QueryRequest(BaseModel):
query: str
top_k: int = 2
class QueryResponse(BaseModel):
query: str
matches: list[str]
@app.post("/search", response_model=QueryResponse)
def search(request: QueryRequest) -> QueryResponse:
matches = retrieve(request.query, k=request.top_k)
return QueryResponse(query=request.query, matches=matches)
# Run with: uvicorn main:app --reload
# Visit http://127.0.0.1:8000/docs for the auto-generated Swagger UI
Two things worth knowing before you ship this: FastAPI’s response_model enforces the output shape even if your function returns extra fields, and the dependency-injection system (Depends()) is the right place to load the FAISS index once at startup rather than per-request — rebuilding an embedding model on every call is the single most common FastAPI+ML performance mistake.
A Simple Autonomous Agent Loop
An “agent” is just a loop: the model decides which tool to call, your code runs that tool, the result gets fed back in, and the loop repeats until the model says it’s done. Here’s that loop with two real tools (no external LLM API required to see the pattern — swap the decide_action stub for a real model call when you plug in an actual LLM):
def tool_calculator(expression: str) -> str:
try:
return str(eval(expression, {"__builtins__": {}}, {}))
except Exception as e:
return f"error: {e}"
def tool_search_docs(query: str) -> str:
return "n".join(retrieve(query, k=1))
TOOLS = {
"calculator": tool_calculator,
"search_docs": tool_search_docs,
}
def decide_action(user_goal: str, history: list[str]) -> tuple[str, str] | None:
"""Stub: in production this calls an LLM with function-calling enabled
and the model returns which tool to use plus its input."""
if "calculate" in user_goal.lower() and not history:
return ("calculator", "12 * 7")
if history:
return None # agent decides it has enough information, stops
return ("search_docs", user_goal)
def run_agent(user_goal: str, max_steps: int = 5) -> list[str]:
history = []
for step in range(max_steps):
action = decide_action(user_goal, history)
if action is None:
break
tool_name, tool_input = action
result = TOOLS[tool_name](tool_input)
history.append(f"{tool_name}({tool_input}) -> {result}")
return history
for line in run_agent("calculate something for me"):
print(line)
# calculator(12 * 7) -> 84
The decide_action stub is where a real agent framework (LangChain’s AgentExecutor, or a raw OpenAI/Anthropic function-calling loop) plugs in — the model reads the tool definitions and conversation history, and returns a structured tool call instead of free text. The loop structure around it stays exactly the same regardless of which model you use.
Practical Notes Before You Ship
- Chunk size matters more than embedding model choice. 300–500 tokens with 10–15% overlap is a solid default for technical docs; too small loses context, too large dilutes the match.
- eval() in the calculator tool above is for demonstration only — in production, use a safe expression parser (e.g.
asteval) since agent tool inputs can come from untrusted model output. - Cache embeddings, not just responses. Re-embedding the same document set on every server restart is wasted compute — persist the FAISS index to disk with
faiss.write_index(). - FastAPI + async: if your embedding or LLM call is I/O-bound (an HTTP call to a hosted model), use
async defendpoints so FastAPI doesn’t block the event loop on every request.
Frequently Asked Questions
Do I need a GPU to run RAG locally?
No. all-MiniLM-L6-v2 and similar small embedding models run comfortably on CPU for development and low-traffic production use. A GPU only becomes worthwhile once you’re embedding millions of documents or running the generation step (the LLM itself) locally rather than via a hosted API.
FAISS vs. a managed vector database like Pinecone — which should I use?
FAISS is a library, not a server — you run it in-process, which is simplest for a single server or prototype. Reach for a managed vector database once you need the index shared across multiple servers, persisted independently of app restarts, or updated by a separate ingestion pipeline.
How is this different from just using ChatGPT’s file upload feature?
ChatGPT’s file upload is RAG under the hood, but it’s a black box you don’t control — you can’t choose the chunking strategy, the embedding model, or how many results get retrieved, and you can’t wire it into your own app’s API. Building it yourself gives you that control and lets you serve it to your own users.