Meta released a model called Muse Glimmer this week under an Apache 2.0 license — meaning the weights are published, and anyone can run them on their own infrastructure without asking permission or paying per request.
We put it through the same internal scoring harness we use to evaluate every model before it goes anywhere near client work. It is the first model to score perfectly across every category we test. It did that running on a single desktop-class machine in our office, using roughly 22 gigabytes of memory, alongside several other workloads on the same box.
That combination — top-tier capability, modest hardware, no per-request bill, and no documents handed to a third-party service — is worth explaining, because it changes the arithmetic for a lot of business automation.
It also came with a real cost, which we want to put up front rather than bury: it is meaningfully slower than the models we already run. Speed is the honest weakness of local AI, and no benchmark result makes that go away.
What we measured
We score every candidate on three independent results, and we treat them as three separate questions rather than blending them into one number:
- Quality (0–100) — a weighted average across our scenarios, weighted by consequence rather than difficulty. A task whose failure reaches a customer counts for more than one whose failure costs a click.
- Latency (band A–D) — the median time to complete one realistic task. A is interactive (under 20 seconds), B is responsive, C is background work you queue, D is batch-only.
- Safety (per capability) — certain scenarios are marked critical, meaning a failure has consequences we will not accept: inventing a figure, leaking an internal note, citing a document that does not exist. Scoring below 90% on one of those blocks the model for that capability no matter how good its average is.
The scenarios themselves cover the capabilities that determine whether a model is safe to put behind a business process: following a documented procedure exactly, producing valid structured output every time, answering only from the reference material provided, declining when a request is too vague to act on, and recovering when something fails mid-task. Every check is graded by plain code against a known-correct answer, so there is no room for a persuasive model to talk its way to a better grade.
Muse Glimmer scored full marks on all of it. For context, the models we currently run in production scored 97 and 99 out of 100 on the same scenarios — both very good. Nothing had previously come through without a single defect anywhere.
We want to be precise about the size of that win, because it is smaller than a perfect score makes it sound. The margin on quality is a few points. What makes it notable is the absence of failure modes — including on the category we consider hardest, where a model has to catch and correct a subtly wrong instruction rather than pass it through. Every other model we run misses some of those.
The part nobody advertises: it is slow
Here is the same comparison with latency included, which is how we actually make the decision:
- Muse Glimmer — quality 100, latency band C, median 93 seconds per task
- Our general-purpose model — quality 99, latency band A/B, median 17–33 seconds
- Our fastest production model — quality 97, latency band A, median 11 seconds
Three points of quality for roughly eight times the wait. Stated that way, the decision is obvious for most work, and it is not the one the headline suggests.
This is the real trade-off of running AI locally, and it deserves to be stated plainly. A hosted API runs on datacenter accelerators that cost more than most small businesses' entire IT budget, and they are simply faster than anything sitting in an office. When you run a model on your own infrastructure you are trading response time for cost predictability, privacy, and control. That trade is excellent for some work and bad for other work, and anyone who tells you local AI is straightforwardly better is selling something.
Where it is bad: anything a person is sitting and waiting for. Nobody wants to watch a spinner for ninety seconds. Where it is good: work that was already going to take a while, or that runs unattended — overnight batches, queued jobs, anything that feeds a process rather than a person. In those cases the wait costs nothing, and the reliability is worth having.
So we did not replace anything wholesale. We assigned this model to the specific jobs where being right matters more than being quick, and left the faster models on the paths where someone is waiting on an answer. That is the entire decision, and it is the same one we would make on a client's behalf.
The part that makes this practical: compression
Published at full precision, this model is a 57-gigabyte download. At that size it demands serious, expensive hardware.
We did not run the full-precision version. We ran a 4-bit quantized build — a compression technique that stores the model's parameters at reduced precision. The result is an 18.7 GB file, roughly a third of the original, and Meta's published figures put the accuracy cost at about 0.2 percent.
That is the trade that matters. Two-tenths of one percent of accuracy, in exchange for a model that runs on genuinely modest infrastructure. In our testing, that compressed version still scored perfectly.
The full memory picture on our machine, for anyone who wants real numbers rather than a marketing claim:
- 18.7 GB — the compressed model itself (4-bit, versus 57 GB at full precision)
- 1.3 GB — the vision component, which lets it read images and screenshots
- 1.5 GB — a small companion model used to accelerate generation
- ≈22.1 GB — total memory actually held while running, including working memory for a 32,000-token context window
The model natively supports a context window of 131,072 tokens — enough to hold a very long document set in a single request — though we run it at a smaller window because reserving space for the maximum costs memory you usually do not need.
Speed, and the trick that made it usable
Raw generation started at about 10 tokens per second, which is too slow for comfortable use on longer tasks. Two of our test scenarios failed outright on time.
Meta ships a small companion model alongside the main one — a technique called speculative decoding, where the small model guesses several tokens ahead and the large model verifies them in a single pass. When its guesses are right, you get several tokens for the cost of one. When they are wrong, they are discarded and nothing is lost but a little compute.
Enabling it took generation from 10.25 to 24.7 tokens per second sustained, with bursts near 35. About two-thirds of the small model's guesses were accepted. The two scenarios that had been failing on time both passed comfortably.
That is a 2.4x speedup for 1.5 GB of extra memory, and it is the difference between this model being a curiosity and being usable.
Why open weights matter for business
There is a meaningful difference between renting AI capability and running a model you control.
With a hosted API, you are billed per request, your documents travel to a third party's servers, and the model underneath you can be changed or retired on a schedule you do not control. Businesses have already been through this cycle — a model version deprecated, and a workflow quietly behaving differently the next morning.
An open-weight model is a file. Once it is deployed, it behaves the same way tomorrow as it does today. The cost is fixed infrastructure rather than metered usage, so it does not climb as your volume grows. Proprietary records stay within a private environment instead of being sent to a public service. And nothing external can deprecate it out from under a process your staff depends on.
For a long time the trade-off was that open models were noticeably worse. That gap has been closing for a while. This week, on the specific business-shaped tasks we care about, it closed entirely.
You do not need to buy any of this
The obvious objection to everything above is capital cost. Specialized hardware is expensive to purchase, and it is a genuinely bad idea for most small and mid-size businesses to buy a machine, staff someone to maintain it, and discover a year later that the model landscape moved.
That is exactly the problem we exist to solve. We buy, run, and maintain the infrastructure, and deploy tailored workflows on top of it for our clients. You get fixed, predictable operating costs and a private processing environment without the upfront investment, the maintenance burden, or the risk of owning depreciating hardware. When a better model arrives — as one did this week — we test it and migrate the workloads that benefit. That is our job, not yours.
That said, the door is open in both directions. For organizations where data residency is not negotiable — regulated records, contractual obligations to a customer, or an internal policy that simply does not permit certain information to leave your own equipment — we will deploy and maintain the same workflows on hardware you own, sitting in your building. The software is identical; only the location changes. Most businesses do not need that, and we will say so. The ones that do usually already know it.
What this does not mean
It does not mean AI is now the right answer to more problems than it was last week.
The bar for using a model at all has not moved: the input has to be genuinely unstructured, the alternative has to be a person doing repetitive work by hand, and the measured reliability has to be good enough to trust behind a review step. Most business problems still fail that test and are better solved with a form, a database, and a well-built workflow. The strongest model in the world does not change that.
What it does change is the economics of the cases that do clear the bar — specifically the ones that can tolerate a slower answer. Workflows that were only viable as a metered cloud service can now run on dedicated infrastructure at a fixed cost, with your data kept out of public services, and in most cases without a hardware purchase on your side. Workflows where someone is drumming their fingers waiting for a response are a different conversation, and often still belong elsewhere.
That is the shift worth paying attention to — not the benchmark number.
If you want to understand where this fits in your operation, read about how we approach AI implementation, or see how we test these models before recommending anything. If you would rather just talk it through, get in touch.