I Wanted to Run AI Locally in C# ecosystem— But I Didn’t Want to Use Ollama
I’ve been experimenting with local AI models for a while, and like many developers, Ollama was one of the first tools I reached for.
And honestly, it’s easy to see why.
Install it.
Download a model.
Run a command.
Start chatting with a local LLM.
It doesn’t get much simpler than that.
But recently, I started asking myself a different question:
Can I run a modern AI model locally without using Ollama?
Not because Ollama isn’t good. It’s actually one of the easiest ways to get started with local LLMs.
I was simply curious about what else is available.
More specifically, as someone who spends most of my time working with C#, .NET, Windows, and NVIDIA hardware, I wanted to explore Microsoft’s approach to local AI.
That led me to Microsoft Foundry Local.
And that’s where things started getting interesting.
🎥 Watch the Video
Prefer watching instead of reading?
In this video, I explain why I started exploring Microsoft Foundry Local as an alternative approach to running AI models locally—and what I want to test with C#, .NET, and my RTX 4050.
Video: Run AI Locally with C# & Microsoft Foundry Local | RTX 4050 Demo
What Is Microsoft Foundry Local?
Microsoft Foundry Local is designed to let developers run AI models directly on their own machines.
Instead of sending a prompt to a cloud-based AI service:
the model can run locally:
The important difference is simple:
The inference happens on your machine.
Your application doesn't necessarily need to send prompts and application data to a remote AI provider.
For certain workloads, that can change the architecture of an application quite significantly.
Why Run AI Locally?
Local AI isn't just about avoiding the cloud.
There are several practical reasons why developers might want inference to happen on the same machine as the application.
For example:
Run AI without relying on a cloud API
Work offline
Reduce network-related latency
Keep sensitive data on the device
Avoid per-token cloud inference costs
Simplify certain application architectures
Experiment with AI without continuously calling a remote service
Of course, local AI comes with its own trade-offs, particularly around hardware requirements and model size.
But for the right workload, the architecture is compelling.
Your Data Can Stay on the Device
One of the biggest reasons local inference interests me is data privacy.
Consider an application that processes proprietary or sensitive information.
With a cloud-based model, the application typically has to communicate with a remote AI service.
With local inference, the architecture can instead look like:
Application Data
↓
Local AI Model
↓
Local Response
The data doesn't need to leave the machine simply to perform inference.
That doesn't automatically make an application secure or private—there are still plenty of security considerations—but keeping inference local can remove an entire category of network data transfer.
For privacy-sensitive applications, that's worth paying attention to.
No Network Round Trip
Cloud inference requires communication between your application and a remote service.
Even if the AI model itself responds quickly, the request still has to travel over the network and the response has to come back.
With a local model:
Application
↓
Local Model
↓
Response
the model is already sitting on the same machine.
That can be particularly useful for interactive applications where responsiveness matters.
Offline AI
Another interesting advantage is offline operation.
Once the required model is available locally, inference doesn't necessarily require an internet connection.
That opens up possibilities for applications such as:
Desktop developer tools
Field applications
Edge applications
Offline assistants
Privacy-sensitive software
Applications running in isolated environments
Instead of:
Internet
↓
Cloud AI
↓
Response
you can potentially have:
Application
↓
Local AI
↓
Response
That's a very different deployment model.
What About the Cost?
Cloud AI usually introduces a variable cost.
The more your application uses the model, the more API calls—and potentially tokens—you consume.
With local inference, there isn't a per-token cloud inference charge.
You still have the cost of the hardware, electricity, model storage, and development effort. But once the model is running locally, you can experiment and repeatedly test your application without worrying about every prompt creating another cloud inference charge.
For developers building and testing AI applications, that can be attractive.
Do You Really Need an AI Backend?
A traditional AI-powered application might look something like this:
Frontend
↓
Backend API
↓
Authentication
↓
AI Service
↓
Cloud Model
For some local applications, the architecture can become much simpler:
Application
↓
Foundry Local
↓
Local Model
There may be no separate AI inference server to deploy, scale, or monitor.
That doesn't mean every application should be built this way.
For large-scale production systems serving thousands or millions of users, cloud inference may still make much more sense.
But for desktop, edge, development, and privacy-sensitive scenarios, local inference is an interesting architectural option.
Why Look Beyond Ollama?
Ollama has become almost synonymous with local LLM experimentation.
You've probably seen commands like:
ollama run llama3
or:
ollama run qwen3
And that's exactly what makes Ollama so popular.
It's simple.
But simplicity isn't the only thing I'm interested in.
As a .NET developer, I wanted to understand what the Microsoft ecosystem offers for local AI.
I wanted to answer a few practical questions:
Can I run an LLM locally without Ollama?
Can I use my NVIDIA GPU?
Can I interact with the model from C#?
How good is the .NET developer experience?
Can I build an actual application around the local model?
Can local AI eventually become part of an AI agent or larger application?
Those questions are much more interesting to me than simply asking which tool is better.
Ollama vs Foundry Local
At a high level, both approaches address a similar problem:
Run AI locally.
But they come from different ecosystems.
Ollama focuses heavily on making local LLM usage straightforward.
Foundry Local is interesting to me because it sits much closer to the Microsoft developer ecosystem, particularly for developers already working with Windows, .NET, and Microsoft's AI tooling.
So I'm not looking at this as:
"Which product wins?"
Instead, I think the more useful question is:
Which approach makes more sense for the application I'm building?
That's a much more practical way to look at it.
What About My NVIDIA GPU?
This was one of the parts I was most interested in testing.
My development machine has an NVIDIA RTX 4050, and I've already spent time experimenting with CUDA, PyTorch, and other local AI workloads.
So naturally, I wanted to see how Foundry Local would use the hardware available on my machine.
The important question isn't simply:
"Can it run?"
I want to know:
How well does it run?
GPU utilization, VRAM consumption, inference speed, model size, and responsiveness all matter when you're running AI locally on a laptop.
And this is where an actual experiment becomes much more useful than documentation alone.
The Real Reason I'm Interested: C#
This is probably the biggest reason Foundry Local caught my attention.
My primary development ecosystem is C# and .NET.
I don't want my local AI experimentation to stop at:
"I downloaded a model and had a conversation with it."
That's interesting—but it's not the end goal.
I want to build software.
Imagine a .NET application where the AI model is simply another component:
┌─────────────────────────────┐
│ .NET Application │
│ │
│ Business Logic │
│ ↓ │
│ AI Integration │
│ ↓ │
│ Local AI Model │
└─────────────────────────────┘
Now we're talking about something much more interesting.
The model isn't the application.
The model becomes part of the application.
My First Experiment
Rather than immediately building a complicated AI agent, I decided to start small.
The first goal is straightforward:
Run a local model through Foundry Local and call it from C#.
Conceptually, I'm looking for an experience along the lines of:
var response = await client
.GetResponseAsync("Explain dependency injection in C#");
Console.WriteLine(response);
The exact API isn't the interesting part here.
What's interesting is what happens underneath.
There isn't necessarily a remote OpenAI request sitting behind the application.
The model is running locally.
That opens the door to something much bigger than a chatbot.
Is Foundry Local Better Than Ollama?
I'm not ready to make that claim.
And honestly, I don't think that's the right question.
Ollama has an excellent developer experience and a large community around local LLMs.
Foundry Local brings something different to the table.
For me, the attraction is its relationship with the Microsoft/.NET ecosystem.
So instead of asking:
"Which one is better?"
I'd rather ask:
"Which one is the better fit for my application?"
That's a much more useful question.
What I Want to Test
I don't want this experiment to end with a simple "Hello World."
I want to compare the practical developer experience.
1. Installation
How difficult is it to get Foundry Local running compared with Ollama?
2. Model Availability
What models are available, and which ones make sense for a laptop?
3. GPU Utilization
How effectively can Foundry Local use my RTX 4050?
4. Inference Performance
How quickly can the model generate responses?
5. Memory Usage
How much RAM and VRAM does the model actually consume?
6. C# Integration
How clean and practical is the .NET integration?
7. AI Agents
Can a local model become the foundation for an actual AI agent?
8. Ollama vs Foundry Local
Finally, I want to put the two approaches side by side.
Not a benchmark designed to make one product look better.
A practical developer comparison.
Why Local AI Matters
I don't think every AI workload needs a massive cloud model.
For some applications, a smaller model running locally may be more than sufficient.
Consider an application where the workflow is simply:
User Data
↓
Local AI Model
↓
Local Response
There may be:
No API key
No per-token cloud inference charge
No network dependency for inference
No separate AI inference backend
Less data leaving the device
Compare that with:
Application
↓
Cloud API
↓
LLM
↓
Response
These are fundamentally different architectures.
Neither is universally better.
But having both options is valuable.
I'm Not Abandoning Ollama
Let me be clear:
This isn't an "Ollama is dead" article.
Far from it.
Ollama remains one of the easiest ways to get started with local LLMs.
I'm simply curious about what exists beyond it.
And Microsoft Foundry Local looks particularly interesting for developers already working in the C#/.NET ecosystem.
The Experiment Starts Here
I've been experimenting with local AI across several areas:
Ollama.
Qwen.
CUDA.
PyTorch.
.NET.
AI agents.
And now:
Microsoft Foundry Local.
The next step is to stop talking about local AI and actually put it through its paces.
The question I'm interested in answering is:
How well can an RTX 4050 laptop run a modern AI model through Microsoft's local AI stack?
That's what I'm going to find out next.
And if the results are interesting, I don't want to build another chatbot.
I want to build something real.
A proper C# application powered by a local AI model.
That's where this experiment gets interesting.
If you're interested in building AI applications with .NET, check out my guide to Microsoft Agent Framework.
Comments
Post a Comment