The measurements were taken on my own benchmark suite via Ollama 0.32.1 on
July 19, 2026. 26 unique generative models (tasks: code generation,
translation, summarization, custom tasks) and 5 embedding models (retrieval,
classification, STS semantic similarity) were run. The avg_score values are
in the 0–1 range (higher is better); avg_latency is the average latency in
seconds.
Generative models#
A summary table over four tasks. The columns are avg_score per benchmark;
Avg.score — the average score across all tasks; Avg.lat — the average
latency, seconds. Sorted by descending average score.
| # | Model | Context | Code | Translation | Summaries | Custom | Avg.score | Avg.lat |
|---|---|---|---|---|---|---|---|---|
| 1 | gemma4:12b |
256k | 1.000 | 0.772 | 0.347 | 1.000 | 0.780 | 2.41 |
| 2 | gemma4:26b |
240k | 1.000 | 0.722 | 0.396 | 1.000 | 0.780 | 2.09 |
| 3 | kimi-k2.7-code:cloud |
256k | 1.000 | 0.820 | 0.281 | 1.000 | 0.775 | 4.20 |
| 4 | glm-5.2:cloud |
256k | 1.000 | 0.762 | 0.335 | 1.000 | 0.774 | 7.97 |
| 5 | minimax-m3:cloud |
256k | 1.000 | 0.794 | 0.289 | 1.000 | 0.771 | 3.35 |
| 6 | deepseek-v4-flash:cloud |
256k | 1.000 | 0.749 | 0.308 | 1.000 | 0.764 | 2.41 |
| 7 | deepseek-v4-pro:cloud |
256k | 1.000 | 0.777 | 0.278 | 1.000 | 0.764 | 3.67 |
| 8 | gemma4:12b-it-qat |
256k | 1.000 | 0.714 | 0.336 | 1.000 | 0.762 | 2.39 |
| 9 | gemma4:26b-a4b-it-qat |
256k | 1.000 | 0.655 | 0.355 | 1.000 | 0.753 | 1.75 |
| 10 | nemotron-3-ultra:cloud |
256k | 1.000 | 0.635 | 0.281 | 1.000 | 0.729 | 7.04 |
| 11 | qwen3.6:27b |
96k | 1.000 | 0.588 | 0.313 | 1.000 | 0.725 | 3.57 |
| 12 | gpt-oss:20b |
128k | 1.000 | 0.633 | 0.246 | 1.000 | 0.720 | 3.48 |
| 13 | devstral-small-2:24b |
52k | 1.000 | 0.601 | 0.264 | 0.982 | 0.712 | 3.81 |
| 14 | gemma4:e2b-it-qat |
128k | 0.938 | 0.581 | 0.284 | 1.000 | 0.701 | 4.35 |
| 15 | gemma4:e4b-it-qat |
128k | 1.000 | 0.481 | 0.302 | 0.991 | 0.694 | 4.50 |
| 16 | glm-4.7-flash:latest |
88k | 1.000 | 0.487 | 0.295 | 0.991 | 0.693 | 1.45 |
| 17 | gemma4:e2b |
128k | 0.950 | 0.522 | 0.285 | 1.000 | 0.689 | 4.12 |
| 18 | gemma4:e4b |
128k | 1.000 | 0.434 | 0.266 | 0.991 | 0.673 | 4.08 |
| 19 | ornith:9b |
256k | 0.950 | 0.485 | 0.267 | 0.975 | 0.669 | 3.86 |
| 20 | laguna-xs-2.1:latest |
80k | 0.950 | 0.483 | 0.242 | 1.000 | 0.669 | 1.80 |
| 21 | translategemma:4b |
128k | 0.950 | 0.375 | 0.228 | 0.984 | 0.634 | 2.03 |
| 22 | t-lite-it-1.0:7b |
32k | 0.988 | 0.390 | 0.265 | 0.877 | 0.630 | 3.74 |
| 23 | north-mini-code-1.0:latest |
80k | 1.000 | 0.477 | 0.258 | 0.659 | 0.599 | 3.28 |
| 24 | lfm2.5:8b |
125k | 0.967 | 0.293 | 0.215 | 0.681 | 0.539 | 2.51 |
| 25 | gigachat3.1:10b |
256k | 1.000 | 0.264 | 0.178 | 0.481 | 0.481 | 4.11 |
| 26 | t-lite-it-2.1:8b |
40k | 0.550 | 0.499 | 0.171 | 0.651 | 0.468 | 5.42 |
Per-task leaders (generative)#
- Code generation:
deepseek-v4-flash:cloud(1.000) ·glm-5.2:cloud(1.000) ·kimi-k2.7-code:cloud(1.000) - Translation:
kimi-k2.7-code:cloud(0.820) ·minimax-m3:cloud(0.794) ·deepseek-v4-pro:cloud(0.777) - Summarization:
gemma4:26b(0.396) ·gemma4:26b-a4b-it-qat(0.355) ·gemma4:12b(0.347) - Custom tasks:
deepseek-v4-flash:cloud(1.000) ·glm-5.2:cloud(1.000) ·kimi-k2.7-code:cloud(1.000) - The fastest (by average latency):
glm-4.7-flash:latest(1.45 s) ·gemma4:26b-a4b-it-qat(1.75 s) ·laguna-xs-2.1:latest(1.80 s)
Embedding models#
A summary table over three tasks: retrieval (NDCG), classification (micro-F1), STS (semantic similarity, Spearman). Sorted by descending average score.
| # | Model | Retrieval | Classification | STS | Avg.score |
|---|---|---|---|---|---|
| 1 | nomic-embed-text-v2-moe:latest |
0.942 | 1.000 | 0.802 | 0.915 |
| 2 | qwen3-embedding:8b |
0.942 | 1.000 | 0.791 | 0.911 |
| 3 | qwen3-embedding:4b |
0.972 | 1.000 | 0.750 | 0.907 |
| 4 | embeddinggemma:300m |
0.972 | 1.000 | 0.726 | 0.899 |
| 5 | qwen3-embedding:0.6b |
0.972 | 1.000 | 0.666 | 0.880 |
Per-task leaders (embedding)#
- Retrieval (NDCG):
embeddinggemma:300m(0.972) ·qwen3-embedding:0.6b(0.972) ·qwen3-embedding:4b(0.972) - Classification (F1):
nomic-embed-text-v2-moe:latest(1.000) ·embeddinggemma:300m(1.000) ·qwen3-embedding:0.6b(1.000) - STS (Spearman):
nomic-embed-text-v2-moe:latest(0.802) ·qwen3-embedding:8b(0.791) ·qwen3-embedding:4b(0.750)
Methodology and notes#
- Environment: Ollama 0.32.1, measured on 2026-07-19. Models with the
:cloudsuffix were queried via a cloud endpoint, the rest — via local Ollama instances. - Metrics: for text tasks
avg_scoreaggregates F1/BLEU/ROUGE/Levenshtein/code-sim;avg_latency(sec) was additionally measured. For embeddings,avg_scoreis NDCG (retrieval), micro-F1 (classification), Spearman (STS). - Deduplication: if a model appeared in several runs, the last one was
kept (the greater
run_id). - Context is given in thousands of tokens (
k); for embedding models context is not relevant. - The raw CSV also contains intermediate metrics (BLEU, ROUGE1/2/L, Levenshtein, code-sim, MSE, Pearson, cosine, etc.) — only the summarized ones are kept here.
The current model setups (glm-5.3, Qwen3.8-27B on llama.cpp, etc.) are on the LLM tab.