Laya on CPU, No GPU: What I Measured on Three Machines Before Putting It in Production

In the previous article on Laya I quoted a number: 38ms per decision. That is the figure the project publishes, and it is true, with a GPU. I wanted to know what happens on a machine without one, which is where almost everything I run lives. So I measured it on three machines, deployed it on one of them, and put it behind my LLM gateway. This article is the numbers, including the ones I did not like.

Short version: on CPU, one decision takes between 94ms and 848ms depending on machine and checkpoint. Zero-shot accuracy in Spanish is 84%. And there is a configuration trap that multiplies latency by 30 if you do not know about it.

The Test Bench

25 hand-written cases in Spanish, split across 5 suites that look like what a customer-support backend decides every day:

Suite	Type	Cases	Example
department	choose 1 of 4	6	”You charged my monthly fee twice” → billing
intent about an auction	choose 1 of 4	5	”How long do I have to make the transfer?” → payment
is it spam?	yes/no	6	”Crypto investment opportunity, 300% returns” → yes
is the customer angry?	yes/no	4	”If this isn’t fixed tomorrow I’m reporting you” → yes
does it contain personal data?	yes/no	4	”Set up the direct debit on ES91 2100…” → yes

Each case runs 3 times, with no prior training (zero-shot). I measure the median (p50) and 95th percentile latency per decision, plus hits out of 25.

Laya ships three checkpoints. I tested all three:

• english: ModernBERT-large, 421M parameters. Despite the name, it handles Spanish.
• typed-decisions: 421M, fine-tuned for typed decisions.
• multilingual: mmBERT-base, 322M. Smaller, faster.

Three Machines, One Table

Same test bench, same Laya version (0.3.23), best configuration found per machine.

Machine	Cost	Runtime	typed-decisions p50 / p95	multilingual p50 / p95	Hits td / ml
Laptop, i5-7300HQ (4 cores, 2017)	already owned	PyTorch	489 / 752ms	173 / 291ms	21/25 · 18/25
ARM VPS, 2 OCPU, 11GB	€0/month (free tier)	ONNX fp32	848 / 1,593ms	297 / 375ms	21/25 · 18/25
x86 VPS, 16 vCPU EPYC, 62GB	general-purpose VPS	ONNX fp32, 4 threads	326 / 680ms	94 / 243ms	21/25 · 18/25

Three things stand out.

Accuracy does not change. 21/25 with typed-decisions and 18/25 with multilingual, on all three machines, PyTorch or ONNX. The model is deterministic. The only thing hardware changes is time.

multilingual is 3 to 4 times faster, and 12 points worse. 72% vs 84%. And it misses things a human never would: it routes “I want to cancel” to the technical department and fails to spot a national ID or IBAN written in plain sight. In the previous article I wrote “multilingual without degradation”. Measured, that is not the case.

The free 2-core ARM VPS works. Almost a second per decision with typed-decisions, but it works. To classify an email on arrival, flag spam on a form, or tag records in a cron job, one second is irrelevant. For a synchronous web response, it is not.

The Thread Trap

This is the most useful part of the article, and it is in no documentation.

The 16 vCPU VPS gave, on the first run, a p50 of 12,134ms. Twelve seconds. Twelve times worse than the free 2-core VPS. The machine was 60% idle with no CPU steal from the hypervisor.

The cause: by default, PyTorch and ONNX Runtime use as many threads as cores they see. With 16 threads splitting small operations, the cost of synchronizing them exceeds the compute. Measured with the multilingual checkpoint and PyTorch:

Threads per process	p50	p95
1	266ms	403ms
2	190ms	514ms
4	170ms	779ms
8	1,051ms	3,228ms
16 (default)	5,097ms	10,291ms

The sweet spot is 2 to 4 threads per process. The 16 cores get used by running several low-thread processes, not one process with many threads. In the final deployment: 4 workers with 4 threads each.

If you run Laya (or any BERT-class model) on a many-core machine and it is slow, before touching anything else pin OMP_NUM_THREADS or intra_op_num_threads to 2 or 4.

ONNX Yes, int8 No

I exported the checkpoints to ONNX (opset 18) and tested fp32 and int8 (dynamic quantization of the MatMuls).

ONNX fp32 vs PyTorch, same accuracy:

• On the x86 EPYC: 2.4 to 2.6 times faster. That is the difference between 795ms and 326ms per decision.
• On ARM: 10-12% faster. Modest, but without the p95 spikes.
• On the i5: nothing. Same as PyTorch, with p95 spikes near 3 seconds whenever ONNX Runtime re-plans memory for a new input shape.

int8: slower and loses 3 hits (15/25). It flags normal messages as spam and sees personal data where there is none. On ARM, quantization also ran out of memory inside the container’s 4GB limit. Discarded.

And one counterintuitive detail: asking 5 questions about the same text in a single call is slower than 5 calls with one question each (2,579ms vs 2,182ms with 4 threads). The input sequence grows with every question and attention costs more than re-encoding a short state. If you want parallelism, use more workers, not a fatter request.

Accuracy: What It Gets Right and What It Does Not

84% with typed-decisions over 25 cases. By suite:

Suite	typed-decisions	multilingual
department (6)	5	4
auction intent (5)	4	4
spam (6)	5	6
angry (4)	4	2
personal data (4)	3	2

The typed-decisions misses are ambiguous criteria, not dumb errors: “How much would the annual plan cost?” goes to billing instead of sales, and “do you ship to Portugal?” goes to payment instead of pickup. These get fixed by refining the wording of each option’s criteria, long before thinking about training anything.

For national IDs, IBANs, emails or phone numbers: a regular expression is always right and the model is not. Do not use a model for fixed-format data.

On calibration. In the previous article I spoke of “rigorous mathematical reliability”. Let me qualify that. The 4 misses in the test bench have confidence between 0.34 and 0.67: with a 0.7 threshold, all four would have gone to human review instead of being automated. That is good. But across a wider battery of new cases, the model also gets things wrong at 0.9 or above. One example: “Can you recommend a movie?” classified as product_info at 0.95. Confidence reduces error; it does not remove it. The threshold has to be measured on your own data, not copied.

How the Deployment Ended Up

                 ┌──────────────────────────────────────┐
                 │  Container  python:3.12-slim         │
                 │  laya[onnx,serve] 0.3.23             │
                 │  checkpoint: typed-decisions (ONNX)  │
                 │  4 workers × 4 threads  ·  ~7GB RAM  │
                 │  self-signed TLS  ·  API key         │
                 └──────────────────┬───────────────────┘
                                    │ :8000
                 ┌──────────────────┴───────────────────┐
                 │  UFW: known IPs + VPN only           │
                 └──────────────────┬───────────────────┘
                                    │
              ┌─────────────────────┴─────────────────────┐
              ▼                                           ▼
   [Own systems, direct]                   [LLM gateway in Rust]
                                            model `laya-decide`
                                            same API as every other
                                            model; +~240ms

Deployment decisions:

• Own image. python:3.12-slim, torch only to export the ONNX once, unprivileged user, healthcheck against /health. The model and the .onnx live on a volume; they survive restarts and are never re-downloaded.
• 4 × 4. Four uvicorn workers, four ONNX Runtime threads each. Covers the 16 vCPUs without the thread trap. 16GB memory limit, real usage around 7GB.
• Self-signed TLS with certificate pinning. Clients carry the certificate as the only trusted CA. Disabling verification (-k) still encrypts, but leaves the server spoofable.
• Firewall by IP. From any unauthorized IP, the port does not answer. Not 401, not 403: timeout.
• Behind the gateway. My LLM gateway in Rust exposes Laya as just another model, laya-decide, with the same API and API key as the rest. Client systems do not know whether an LLM or a classifier sits behind it. The gateway adds about 240ms. Measured end to end: 21/25 hits (the same), p50 864ms, and with 8 parallel calls, 75 of 75 correct at 2.8 decisions per second.

That 2.8 is below what I estimated (5 to 10). With 8 simultaneous requests, 4 workers × 4 threads saturate the 16 vCPUs. If more were needed, the lever is 8 workers × 2 threads. For now it is not needed.

What It Is Good For, and What It Is Not

With my own data from a separate test battery (holdout: new cases written after tuning the questions, with nothing re-adjusted):

Reliable, can be automated: does the customer want a human? (92%), is there personal data? (92%), is the action destructive? (90%), is it urgent? (90%).

Useful as a first layer, with a human behind it: spam (90%), threats (90%), churn risk (70%), rude tone (80%), which support team (80%).

Signal only: intent across 9 options (50%), whether tools are needed (58%), prompt-injection guardrails (67%). Here it helps prioritize, not decide alone.

And what it is not: it does not generate text, does not summarize, does not answer knowledge questions. And it is not fit for a synchronous response under 100ms with typed-decisions on CPU. For that, multilingual at 94ms accepting the accuracy loss, or a GPU.

One rule that came out of measuring question variants and holds for any model of this kind: ask about the problem, not the virtue. “Is the tone appropriate?” scores 58%. “Is it rude, sarcastic or dismissive?” scores 88%. And prefer yes/no over scales: “priority from 1 to 4” scores 33%; “does it need attention today because something is broken or costing money?” scores 83%.

Where It Is Today

Today it is consumed, through the gateway, by the customer-support flows of a repair partner for an international warranty insurer: whether the customer wants to talk to a person, what the message is asking before it enters the flow, and yes/no validations in background workers. None of those decisions is irreversible; the ones that matter are confirmed by a person.

The cheap part of Laya is not the latency. It is that one decision costs €0.000016 through the gateway, €0 direct, and the customer’s text never leaves the network. For what it does, on the machine where it does it, 326ms is fine.