Skip to content
IC Israel Chidera
All projects
Personal project In progress

AI Gateway API

A FastAPI service that puts LLM chat and retrieval-augmented Q&A behind one stable API. It splits documents into chunks, embeds them, and answers questions only from the most relevant passages, citing its sources. When nothing relevant is stored, it says so instead of guessing. Built with retries and exponential backoff, timeouts, typed error handling and a 44-test suite.

GitHub

POST /documents

  1. Chunk text

    overlapping windows

  2. Embed chunks

    Gemini embeddings

  3. Store

    vector store

POST /ask

  1. Embed question

    query embedding

  2. Rank chunks

    cosine, top-k

  3. Score ≥ threshold?

    grounding check

  4. Answer + sources

    cites [1], [2]

nothing relevant → "not found", model never called

GeminiProvider

chat + embeddings
timeout per request
retry 429 / 500 / 503, backoff + jitter

MockProvider

offline, deterministic
no API key needed
powers the 44 tests

Typed errors → HTTP

  • 504

    Timeout

  • 503

    Unavailable

  • 429

    Rate limited + Retry-After

  • 502

    Anything else

/ask response

{ "answer": "… [1]", "grounded": true,
  "sources": [{ "doc_id": "…", "chunk_id": "…", "score": …, "preview": "…" }],
  "model": "…", "usage": { "input_tokens": …, "output_tokens": … },
  "latency": … }
Documents sent to POST /documents are chunked, embedded and stored. POST /ask embeds the question, ranks stored chunks by cosine similarity and keeps those above a threshold. If none remain, it answers "not found" without calling the model; otherwise the model answers from the numbered chunks and cites them. Providers retry 429, 500 and 503 responses with exponential backoff and jitter, and errors map to 504, 503, 429 or 502.

The problem

When an application calls an LLM API directly, it becomes tied to one vendor's SDK, error format and failure modes. Rate limits, slow responses and temporary server errors surface straight in the product, and a model asked about your documents will happily answer from general knowledge instead.

AI Gateway puts one stable API in front of the model provider. It handles retries, timeouts and error mapping, and grounds answers in stored documents, refusing when it has nothing relevant to go on.

What I built

Retrieval-augmented Q&A
POST /documents splits text into overlapping chunks, embeds them and stores them. POST /ask embeds the question, ranks stored chunks by cosine similarity, and sends the top matches, numbered [1], [2] and so on, to the model with instructions to answer only from them and cite their numbers. The response lists each source with its similarity score.
A grounding guardrail
Chunks below a similarity threshold are dropped. If none remain, the API answers "not found" without calling the model at all, so it never answers from general knowledge and never pays for a call it can't ground.
Provider-agnostic interface
Chat and embeddings sit behind one AIProvider interface. Gemini is implemented; a mock provider implements the full interface, including embeddings based on word overlap, so the API runs with no API key.
Resilient provider calls
Every chat and embedding call shares one wrapper: a per-request timeout, and retries with exponential backoff and jitter on 429, 500 and 503. Client errors such as 400 fail immediately.
Typed errors, predictable responses
Provider failures map to HTTP statuses: timeout 504, unavailable 503, rate limited 429 with a Retry-After header, anything else 502. Every response reports the model, token usage and latency.
Tested and containerised
44 pytest tests cover the API, retrieval, retries, error mapping and configuration, with no network calls. A Dockerfile runs the service as a non-root user.

Technical decisions

  1. 01

    Refuse rather than guess

    Below the similarity threshold the model is never called. That avoids confident answers the stored documents don't support.

  2. 02

    Retry only what can succeed on retry

    Rate limits and 500/503 errors are often transient, so they're retried. A 400 fails the same way every time, so retrying it only adds latency. Jitter stops clients that failed together from retrying together.

  3. 03

    A mock provider, not just mocked tests

    Because the mock implements real (if simple) embeddings, retrieval tests check actual ranking behaviour rather than hard-coded results.

  4. 04

    A vector store behind a small interface

    Today's store is in memory and searched by brute force, which is fine for a demo. It exposes three methods, so a database such as pgvector can replace it without touching the service.