Reality Original text

My experience using locally installed LLMs on a Mac

1 July 2025

I'm writing to share my experience using local LLMs (Large Language Models) on a Mac, in case someone is wondering about their practicality, usability and if it's worth the hassle. I'm using a MacBook Pro with an M4 Pro chip and 24 GB of RAM. And although I consider myself fairly good with technical stuff, I’m no expert in the matter, so these experiences I’m sharing come more from the perspective of a common power user. Bear that in mind if something I say turns out to be inaccurate.

Apps/Platforms

I’ve tried three different ways of installing and using models: LM Studio, AnythingLLM, and Ollama.

LM Studio

If you want the most Mac-like experience, the best option is definitely LM Studio. The interface feels more like a Mac app and it’s probably the easiest to install and use. It is pretty straightforward: you install the app, open it, and there you have the option to install models. The app is just a way to interact with the models (LLMs), so to use AI, you have to pick a model and download it. They have a directory to pick models from, but you can also import models from Hugging Face, which is a platform where people share AI models. You’ll be up and running pretty fast. Something I like about LM Studio is that it has three UI modes: User, Power User, and Developer. So you can have the app be as simple or as complex as you want. I also like that you can give context to the models and save them as presets. I’ll talk more about that later. Something I also like is that you can load and unload models inside a chat. So you can have a chat with one model, unload it, and load another model to continue the same conversation using a different LLM, in case for example, you’re not liking the answers you're getting. You can also branch out conversations, which is helpful in some cases. For example, suppose you’re having a conversation with a model and want to go to a specific answer it gave you and pick up a new conversation from that message. You can also regenerate the last message received if you want a different answer. Lastly, you can make the model continue the last message if you want it to go deeper into it.

AnythingLLM

AnythingLLM is a very cool app with lots of functionality, but I found it to be a bit more confusing, especially when importing models. It has a great deal of functionality because it adds the ability to make AI agents, but it has a lot more things to set and configure, so it can feel daunting for some users. The UI isn’t great; if you like the consistency that comes from most Mac apps, you won’t like this. The general design, from the colors to the typography, didn’t work for me. I tried the agent functionality to scrape websites, for which I had to use a Google API, and the results weren’t that good, even though I researched which models could work best for that and tried those. If you want to nerd out with this, definitely try it, but it wasn’t for me.

Ollama

Ollama is basically a command-line tool that lets you install and interact with LLMs. You download an app, and every time you launch it, what you’re doing is activating a server on your Mac that runs the models. So, to use the models, you have to keep the server running. All the installation and interaction with LLMs is done through the command line in the Terminal app. I think there are third-party chat UIs you can install to interact with the models, but I didn’t try those. What’s cool about this app is that you can integrate other apps with the models it runs. As an example, I use a snippet app called Snippety that supposedly can integrate with Ollama to use AI, but as I don’t have a use case, I haven’t used it. After a couple of weeks playing with the three options, I uninstalled Ollama and AnythingLLM and stuck with LM Studio. If you try these options, make sure to research all the hidden directories they install so you can delete them completely.

Models

Now let’s talk about models. You’ll find a lot of models from different sources, based on various open-source LLMs and with different levels of functionality and sophistication. You don’t want to download the dumbest models, but you probably won’t want the smartest either, because your machine won’t be able to run them. Also, you’ll find there are a lot of versions of each model, and that will be confusing at first. So here’s the simple, non-technical explanation I wish I had when I first dabbled in this. Don’t come at me if the way I phrase things is inaccurate, I’m trying to explain this as if I were talking to someone without any AI knowledge at a party. You’ll see some models have different versions like 4B, 8B, 12B, that refers to the number of parameters they’ve been trained on. All else being equal, a model with more parameters (like 12B vs. 4B) can handle more complexity. But quality also depends on architecture, training data, and fine-tuning. You may also see that versions include things like 4bit and 8bit, that refers to the level of quantization of the model. Think of it as a way of compression. Reducing the precision of the model’s weights from 16-bit or 32-bit floats down to 4-bit integers drastically reduces memory usage and makes the model faster, but can reduce the quality of the answers. A model with lower compression (like 8-bit) typically performs better than one with higher compression (like 4-bit), although some 4-bit models can retain surprisingly good quality thanks to advanced quantization techniques. Oversimplification: 12B is smarter than 4B, and 8bit will perform better than 4bit. Really easy explained like that, isn’t it? So what models am I using? Currently, I’m using:

Gemma 3 12B 4bit
Meta Llama 3 8B Instruct Q8_0
Deepseek R1 0528 Qwen 8B 8bit

Now let’s break down the names and what they mean, because as you can see, they contain a lot of information:

Gemma 3 12B 4bit

Gemma: This is the name of the model family. "Gemma" is an open-source large language model developed by Google DeepMind.
3: This indicates the version of the model, in this case, Gemma version 3.
12B: This tells us the number of parameters in the model, which are 12 billion.
4bit: This refers to quantization.

Meta Llama 3 8B Instruct Q8_0

Llama: The name of the model (Large Language Model Meta AI) developed as an open-source project by Meta.
3: Indicates this is Llama version 3, released in 2024, the third major version with improvements in reasoning and instruction following.
8B: 8 billion parameters.
Instruct: This means the model has been fine-tuned for instruction-following tasks. These models are optimized to be helpful, and follow user instructions.
Q8_0: 8-bit quantization with 0 groupwise compression or activation scaling. This keeps most of the model's quality intact while still reducing size compared to full float precision.

Deepseek R1 0528 Qwen 8B 8bit

This model is a hybrid. The reasoning capabilities of R1‑0528 were transferred to the smaller Qwen3 8B base model via a “chain‑of‑thought distillation” process.

Deepseek: This is the name of the organization or model family. DeepSeek is a Chinese AI company that develops open-source LLMs.
R1 0528: Indicates the version of the Deepseek model.
Qwen: Qwen is a separate model family developed by Alibaba Cloud.
8B: 8 billion parameters.
8bit: This refers to quantization.

How much resources do these models use? Generally, I’ve noticed that when loaded, they use around 8 to 9 GB of RAM and about 30% of the CPU. LM Studio gives you this information in a bottom bar, which I really like. Each model takes around 8 GB of space.

Conclusion

Finally, is this worth it? What are some use cases? Based on my experience, I’d say these local models are good for text tasks. Things like: summarize this, make this text more concise or professional, make it into bullet points; things like that. I use a lot of context presets in LM Studio for different use cases. In the context preset, you give the LLM a prompt so it knows how to behave or act when you give it instructions or ask it things. I have a preset for SEO work, one for making text clear, concise, and professional, and even one to make my angry email responses more diplomatic and professional while being assertive and direct. These models are also good at solving some math problems and date calculations, which are things I usually need. Also, translations or explaining some language questions. For these kinds of tasks, I find them very useful on a daily basis. It also feels great to not hit any tier/quota constraints (the only constraint is your machine’s resources) and being able to work offline if needed. Now, based on what I’ve seen so far, these models are way behind online models in a lot of ways. For random questions, ChatGPT is still my go-to, for coding Claude, and for research Perplexity. But if you work a lot with text and/or want to play a little with these, go for it.