From a local script to a GPU-served AI app on Kubernetes.

00 Why I built this

The honest starting point: I wasn't sure whether working in AI meant going back to full-time software development.

As a software engineer I owned the whole chain — system design, data modeling, business objects, middleware APIs, and the single-page apps that consumed them. That end-to-end ownership was the part I missed. Skilling up on AI in scattered pieces never gave me the same confidence.

So instead of another course, I built one complete system from scratch: a retrieval-augmented chat app with an agent, running on real cloud infrastructure I provisioned myself. The goal was muscle memory — the kind you only get by shipping something with real stakes and debugging it when it breaks. It broke plenty. That was the point.

Two things I specifically wanted to prove out

1. What actually makes an agent an agent. "Agent" is an overloaded word — most things called agents are a chatbot with a system prompt. I wanted to build the real thing and understand the mechanics: a model that reasons about a request, decides on its own which tool to call, executes it, reads the result, and loops until it can answer — while carrying memory across turns so context accumulates. That means genuine function-calling wired to real tools (a live weather API, a calculator, time, a knowledge-base search), a guard against redundant tool calls, and per-session conversation state. The judgment I care about surfacing here is the design reasoning: knowing the difference between prompting a model to sound agentic and architecting a loop where the model is actually the orchestrator deciding what to do next.

2. How models are served in a modern, cloud-native world. The other reason was to serve an LLM the way it's done at scale — not a model baked into a container as an afterthought, but GPU inference running as a first-class Kubernetes workload: a dedicated GPU nodepool, the NVIDIA device plugin exposing nvidia.com/gpu as a schedulable resource, taints and tolerations so only inference lands on the expensive hardware, and vLLM serving an OpenAI-compatible endpoint that the rest of the cluster consumes over a normal service. This is the same pattern the large AI clusters and HPC shops use to pack accelerators into orchestrated, schedulable pools. Getting an A10 to serve a 7B model as a clean Kubernetes citizen — schedulable, tear-down-able, reproducible from code — was the point as much as the app on top of it.

01 Architecture

A RAG + agent application on Oracle Container Engine for Kubernetes, serving an open LLM on an A10 GPU.

A question flows from the browser through an OCI load balancer to the FastAPI backend, which embeds it, searches Qdrant for relevant document chunks, injects them as context, and asks the LLM to answer — grounded in the source material with citations. Agent mode adds autonomous tool-calling. Worker nodes sit on a private subnet with no public IPs; only the load balancer is exposed.

w1 FastAPI + local LLM

Get a local model answering through code I wrote — not a chat box.

The first week was about making the environment feel like home. FastAPI stands in for .NET Web API — same controller-and-route mental model, different syntax. Ollama runs an open model entirely on the laptop, exposing a REST API the backend calls. The deliverable was a streaming /chat endpoint.

w2 Embeddings + Qdrant

The week the whole thing clicked: meaning as math.

An embedding is just a vector that represents meaning. Two similar sentences land close together; unrelated ones don't. Cosine similarity measures the distance. Once that lands, semantic search stops being magic — it's a query against a vector database, the same shape as a SQL WHERE clause, but by meaning instead of exact match.

w3 Full RAG + React UI

Wire retrieval and generation into one grounded answer, then put a face on it.

RAG is the assembly of the previous two weeks: embed the question, retrieve the closest chunks, inject them into the prompt as context, and ask the model to answer using only that context. The result is grounded — it cites sources and admits when it doesn't know, instead of hallucinating. A React chat UI streams the answer back token by token.

w4 Agents + tool calling

The part I most wanted to get right: a real agent, not a chatbot dressed up as one.

This is where the app stops just answering and starts doing — and where the design thinking matters most. A real agent isn't a prompt that sounds autonomous; it's a loop where the model is the orchestrator. It reads the request, decides on its own which tool fits the intent, calls it, reads the result, and decides whether it now has enough to answer or needs another step. I wired genuine function-calling to four real tools — current time, a calculator, a live weather API, and a knowledge-base search — so the model's decisions execute against actual systems, not stubs.

Two pieces make it behave like an agent rather than a one-shot call. A reasoning loop with a step budget lets the model chain tools (look something up, then calculate on it) instead of firing once. And per-session memory carries context across turns, so a follow-up like "and what about tomorrow?" resolves against what came before. I also added a dedup guard so it won't re-call an identical tool in a loop — a small but real correctness fix that only shows up once you're running a genuine multi-step agent.

w5 Cloud deployment

The week my infrastructure background became the advantage.

Everything to here was application logic. Week five was pure infrastructure: containerize the app, provision an OKE cluster with Terraform, serve the model on an A10 GPU with vLLM, and automate the whole thing through GitHub Actions. Corporate policy blocked Docker on my laptop — so images build in the pipeline instead, which is arguably the more correct pattern anyway.

The portable design paid off here: the same backend image runs locally against Ollama and in-cluster against vLLM, switched by a single environment variable.

The piece I most wanted to do properly was serving the model the cloud-native way — GPU inference as a first-class Kubernetes workload, the same pattern the large AI clusters use. That meant a dedicated GPU nodepool separate from the CPU workers, the NVIDIA device plugin exposing nvidia.com/gpu as a schedulable resource, a taint on the GPU node with a matching toleration on vLLM so only inference lands on the expensive hardware, and vLLM serving an OpenAI-compatible endpoint that the rest of the cluster consumes over an ordinary service. The GPU becomes just another schedulable, reproducible, tear-down-able resource in the pool — which is exactly how accelerators are managed at scale.

→ Deploy it yourself

The entire stack is one-click deployable into your own OCI tenancy.

Because the infrastructure is fully described in Terraform and packaged for OCI Resource Manager, anyone can deploy it. Fill in a form, pick a compartment, toggle the GPU, and go. Or run it locally:

→ Stack & endpoints

What runs where, and the API surface.

Backend endpoints

Infrastructure (Terraform-managed)

• VCN with public/private subnets, NAT + service gateways
• IAM dynamic group + policies for OKE
• OCIR private repositories for images
• OKE cluster + CPU nodepool (E5.Flex)
• Toggleable A10 GPU nodepool for vLLM
• In-cluster image pull secret

! Gotchas & fixes

The real education. None of these were in any tutorial — each cost real debugging time.

→ Teardown

The whole point of infrastructure as code: it all comes back from one command.

Because every resource is Terraform-managed, teardown is complete and clean — no orphaned load balancers quietly billing, no manual console cleanup. And it all rebuilds from terraform apply whenever I want it back.

» What's next — Day 2

Getting it running was Day 1. What separates a demo from a system is Day 2: keeping it updated, making it better, and being able to see and control what it costs. Here's what I'm building next.

1 · Lifecycle & updates

Right now a model or vLLM upgrade is a manual redeploy. The next step is treating the inference layer like any other production service with a real update path:

• vLLM version upgrades — pin and roll vLLM image versions deliberately instead of :latest, with a canary step before promoting.
• Model swaps & A/B — run two model versions behind the same service and shift traffic gradually, so a new model earns its rollout instead of a hard cutover.
• Persistent model cache — a PVC-backed Hugging Face cache so a pod restart doesn't re-download ~15GB of weights, cutting cold-start from minutes to seconds.
• GPU autoscaling — scale the GPU pool to zero when idle and back up on demand (Karpenter-style provisioning), so the expensive hardware only exists when there's work.

2 · Model improvement

The base model is a starting point. Making it genuinely good for a specific domain is its own discipline:

• Fine-tuning — LoRA / QLoRA adapters on domain data, served by vLLM alongside the base weights, to lift answer quality without retraining from scratch.
• An eval harness — a real evaluation set and scoring (retrieval hit-rate, answer faithfulness, tool-call accuracy) so "is the new model better?" is a measured answer, not a vibe. This gates the A/B rollouts above.
• Retrieval quality — chunking strategy, re-ranking, and embedding-model comparisons, measured against the eval set rather than guessed.

3 · Observability & cost control

The piece I care about most as an operator: you cannot run what you cannot see. Full-stack visibility, top to bottom — app through GPU — with cost as a first-class signal:

• App & API tracing — OpenTelemetry spans across the request path (frontend → backend → retrieval → LLM call) so a slow answer can be traced to the exact hop that caused it.
• Middleware & dependency metrics — latency and error rates for Qdrant queries, vLLM calls, and external tool APIs, surfaced as SLIs.
• Cluster & infra metrics — Prometheus + Grafana for node, pod, and control-plane health across the whole OKE cluster.
• GPU observability — the NVIDIA DCGM exporter feeding Prometheus: utilization, memory, temperature, and throughput per GPU. This is where performance tuning actually happens — is the A10 saturated, or paying for idle silicon?
• Cost control — tie GPU-hours and utilization together so cost per request is visible, idle spend is obvious, and autoscaling decisions are driven by data. On accelerators, cost is a performance metric.