A Local LLM Chatbot with Tool-Calling and RAG (No Vector Database)

MobiShare’s chatbot reserves vehicles and files maintenance tickets by talking to the app’s own API, running entirely on a local model — no cloud LLM, no per-token bill, no vector database either. A year after shipping it, the architecture holds up better than I expected. One part of it doesn’t: a one-line “simplification” that quietly broke per-request isolation for every concurrent chat user.

The constraint that shaped everything

The brief was a smart-mobility platform judged live by a panel: a chatbot that actually does things — reserve a scooter, open a repair ticket — not just answer FAQs, and it had to run on hardware we controlled. No data leaving the box, no API key, no bill that scales with usage. I’d already made that tradeoff explicit once, for a personal tool, in Building a Privacy-First Local RAG System. MobiShare is where I first had to ship the same constraint inside a real multi-user web app, under a deadline, with a panel of judges who’d poke at it live.

The stack: OllamaSharp talking to a local Ollama instance (qwen3 for chat, nomic-embed-text for embeddings), orchestrated from a SignalR hub, with the tool-calling surface generated at compile time rather than hand-written.

Tool-calling without hand-writing the schema

Most “give an LLM tools” tutorials have you hand-write a JSON schema per function and hope it stays in sync with the actual method signature. OllamaSharp’s source generator does that work for you: mark a method [OllamaTool] and it emits the wrapper class the model needs at compile time — no runtime reflection, no schema drift, and no trace of the generated class in your source tree (which briefly had me convinced the tool-calling code was dead until I did a clean rebuild and watched it actually execute).

public class VehicleTools
{
    private static readonly IVehicleTool _vehicleTool = new VehicleTool();

    [OllamaTool]
    public static async Task<string> ReportIssue(string description, int vehicleId)
        => await _vehicleTool.ReportVehicleIssueAsync(description, vehicleId);

    [OllamaTool]
    public static async Task<string> ReserveVehicleAsync(string userRequest)
        => await _vehicleTool.ReserveVehicleAsync(userRequest);
}

Those generated tool classes get handed straight to the chat call, and OllamaSharp handles the round trip — deciding whether to call a tool, executing it, and feeding the result back to the model — without me writing a dispatch loop:

var tools = new object[] { new ReportIssueTool(), new ReserveVehicleAsyncTool(), new RoutingPageTool() };

The tradeoff is exactly what you’d expect from any source generator: you get less boilerplate and less drift, in exchange for a class that formally exists (it compiles, it runs) but that you’ll never find with a text search — which matters the first time you’re debugging and go looking for ReportIssueTool in vain.

Retrieval, the un-clever way

For RAG, I didn’t reach for a vector database. Every message pair marked eligible for retrieval lives in the same SQLite database as everything else, and similarity is a plain loop, not an index:

public async Task<List<ChatMessage>> GetRelevantPairsAsync(float[] input, int topN = 3)
{
    var messagePairs = await _dbContext.MessagePairs
        .Include(mp => mp.UserMessage).Include(mp => mp.AiMessage)
        .Where(mp => mp.IsForRag && mp.UserMessage.Embedding != null)
        .ToListAsync();

    return messagePairs
        .Select(mp => (mp.UserMessage, mp.AiMessage,
            sim: CosineSimilarity(input, JsonSerializer.Deserialize<float[]>(mp.UserMessage.Embedding)!)))
        .OrderByDescending(x => x.sim)
        .Take(topN)
        .Select(x => x.AiMessage)
        .ToList();
}

That’s a full table scan plus an O(n) cosine pass, entirely in application memory, every single query. No approximate nearest-neighbor index, no external service to deploy, no extra container in the Docker Compose file. For the scale this actually runs at — a handful of vehicles, a demo’s worth of chat history — it’s fast enough to be invisible, and it’s one fewer moving part for a team under deadline to reason about. This is the row in the numbers table below worth sitting with: the “unscalable” approach was the correct engineering call for the actual size of the problem, and swapping in a vector database on day one would have been solving a scale problem MobiShare didn’t have, at the cost of an extra service to run and monitor.

Where the un-clever part goes wrong

The generated tool classes (ReportIssueTool, ReserveVehicleAsyncTool) have parameterless constructors — the source generator instantiates them, so there’s no constructor-injection path to hand them per-request context like the current user’s ID or GPS position. The fix in this codebase was a static ambient-context class, UserContext, read by the tool methods at call time. That’s a reasonable pattern for exactly this constraint — if the ambient value is scoped correctly.

The first version used AsyncLocal<string>, which is precisely the right primitive here: it flows a value down the async call chain for this request without leaking into any other concurrent one — the same guarantee that makes it safe to store, say, a distributed trace ID. A later commit in the project’s history “simplified” it:

- private static readonly AsyncLocal<string> _UserId = new();
+ private static string _UserId = "";

   public static string UserId
   {
-      get => _UserId.Value;
-      set => _UserId.Value = value;
+      get => _UserId;
+      set => _UserId = value;
   }

The commit message is honest about the tradeoff it’s making: “removes per-async-context isolation, making the values global across all threads.” In a single-user demo, that’s invisible — there’s only ever one active chat, so “global” and “per-request” happen to coincide. Under two concurrent chats, it’s a real bug: whichever user’s message sets UserContext.UserId last wins for every in-flight tool call across every connected user, until it’s overwritten again. A reservation or a location-based lookup can silently execute against the wrong account, and nothing about the request would look malformed — it would just be quietly attributed to the wrong person.

What makes this worth dwelling on is that the fix was already sitting one file over. HttpClientContext — the sibling class doing the same “hand ambient context to a constructor-less tool class” job for the Ollama client and chat handle — still uses AsyncLocal<T> correctly. I used it as the template for a related fix while revisiting this code: those API controllers this chatbot calls into had no endpoint-level authorization at all, and extending HttpClientContext with an AsyncLocal<string?> AuthCookie closed that gap the same way — full writeup in the auth post.

What this cost, and what it didn’t

Verdict

Skip the vector database until you have a concrete reason not to — a linear cosine scan over a few thousand rows is genuinely fast enough at this scale, and it’s one less service a judging panel, and later you, has to reason about. What you can’t skip, at any scale, is per-request isolation the moment more than one user can be in flight at once: a static field standing in for “per-call context” is a landmine in any concurrent host, full stop. The part worth remembering isn’t that AsyncLocal<T> exists — it’s that the correct pattern was already implemented one file over, in the same codebase, and got overwritten anyway. This is rarely a knowledge gap. It’s an unreviewed “simplification” that nobody diffed against the sibling class doing the same job correctly.