Current LLM setups

A live LLM fleet as of October 2026 — three working setups plus the coding agent harness. The cloud glm-5.3 and glm-5.3-flash serve the bots and agents, while the local Qwen3.8-27B on llama.cpp handles long-context and vision tasks. From the outside it all looks like a single OpenAI-compatible API: the provider/model routing is done by any-router (llm.e7.by), and the traces go to Langfuse.

glm-5.3-flash:cloud — the workhorse#

The main model for everyday tasks. It lives on the Ollama relay: models with the :cloud suffix are executed by Ollama in its cloud — locally there is only the relay, no VRAM and no weights. The context goes up to 1M tokens, so in long agent sessions it fits entirely. The Flash variant is about latency/price: on benchmarks the glm family sits at the top for code and custom tasks.

glm-5.3:cloud — the full version#

The same family, the same relay — the full (non-flash) model for heavy tasks where the quality ceiling matters, not the latency. It goes into any-router as glm-5.3:cloud; other cloud models live on the relay alongside (kimi-k3, deepseek-v4, minimax-m3, nemotron-3-ultra) — switching by provider/model in the request, without reconfiguring the clients.

qwen3.8:27b — local llama.cpp#

A local stand on llama.cpp (RTX 3090, 24 GiB VRAM): Qwen3.8-27B in the UD-IQ4_XS quant from unsloth (Dynamic 3.0, 13.3 GiB)

• the mmproj-F16 vision projector. The hybrid architecture (48 linear DeltaNet layers + only 16 full attention layers) gives a cheap KV cache — 200k of context fits into 24 GiB along with the weights when the KV is in q8_0. Thinking is enabled by default and controlled per-request; the API is an OpenAI-compatible /v1.

The key server flags:

llama-server \
  -m /opt/llm/models/Qwen3.8-27B-UD-IQ4_XS.gguf \
  --mmproj /opt/llm/models/mmproj-F16.gguf \
  -ngl 999 -c 200000 -fa on \
  -ctk q8_0 -ctv q8_0 \
  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0 \
  --host 0.0.0.0 --port 8080

Deployment: an Ansible playbook#

The stand is deployed and updated in a single idempotent run (the ansible/ of the monorepo): stop on-demand competitors for the card (comfyui, unsloth) → driver check → build dependencies → CUDA toolkit 12+ (if absent) → clone/pull llama.cpp → cmake + build → download missing models → systemd service → /health health-check. The core of the playbook:

---
# llama.cpp + Qwen3.8-27B on the GPU host (RTX 3090, 24 GiB VRAM).
# "Deploy" and "update" are the same run; the playbook is idempotent.
#   ansible-playbook llamacpp.yaml --key-file meta/msi.key
- name: "llama.cpp + Qwen3.8-27B"
  hosts: [gpu_host]
  become: yes
  vars:
    llamacpp_ref: master             # tracked version: a branch or a tag
    llamacpp_dir: /opt/llama.cpp
    cuda_arch: "86"                  # sm_86 = RTX 3090
    cuda_min_version: "12"          # llama.cpp won't build with gcc-11 on older ones
    models_dir: /opt/llm/models
    hf_url: https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/resolve/main
    llamacpp_models:                # only the missing ones are downloaded
      - Qwen3.8-27B-UD-IQ4_XS.gguf  # 13.3 GiB — along with 200k of context
      - mmproj-F16.gguf             # 0.9 GiB — the vision projector
    llamacpp_model: "{{ models_dir }}/Qwen3.8-27B-UD-IQ4_XS.gguf"
    llamacpp_mmproj: "{{ models_dir }}/mmproj-F16.gguf"
    llamacpp_ctx: 200000            # the model's native ceiling — 262144
    llamacpp_kv_k: q8_0             # KV-cache quant: f16 / q8_0 / q4_0
    llamacpp_kv_v: q8_0
    llamacpp_port: 8080
    llamacpp_api_key: ""            # empty = no auth (the host is VPN-only)
  tasks:
    - name: "Stop on-demand competitors for the card"
      ansible.builtin.systemd:
        name: "{{ item }}"
        state: stopped
      loop: [comfyui, unsloth]
      ignore_errors: true

    # ... nvidia-smi check → build dependencies (git, cmake, build-essential)
    # → CUDA toolkit 12+ from the NVIDIA repo (if nvcc is older) → clone/pull llama.cpp
    # → wipe build/ when nvcc/sm changes → cmake -DGGML_CUDA=ON → build → models:

    - name: "llama-server service"
      ansible.builtin.copy:
        dest: /etc/systemd/system/llama-server.service
        mode: "0644"
        content: |
          [Unit]
          Description=llama-server (Qwen3.8-27B, llama.cpp)
          After=network-online.target
          Wants=network-online.target

          [Service]
          ExecStart={{ llamacpp_dir }}/build/bin/llama-server \
            -m {{ llamacpp_model }} \
            --mmproj {{ llamacpp_mmproj }} \
            -ngl 999 \
            -c {{ llamacpp_ctx }} \
            -fa on \
            -ctk {{ llamacpp_kv_k }} -ctv {{ llamacpp_kv_v }} \
            --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0 \
            --host 0.0.0.0 --port {{ llamacpp_port }}{% if llamacpp_api_key %} \
            --api-key {{ llamacpp_api_key }}{% endif %}
          Restart=on-failure
          RestartSec=5

          [Install]
          WantedBy=multi-user.target
      notify: "Restart llama-server"

    - name: "Start llama-server"
      ansible.builtin.systemd:
        name: llama-server
        state: started
        enabled: yes
        daemon_reload: yes

    - name: "Wait for the model (/health)"
      ansible.builtin.uri:
        url: "http://127.0.0.1:{{ llamacpp_port }}/health"
      register: llamacpp_health
      until: "llamacpp_health.status == 200"
      retries: 60
      delay: 5

  handlers:
    - name: "Restart llama-server"
      ansible.builtin.systemd:
        name: llama-server
        state: restarted
        daemon_reload: yes

The harness — pi#

A separate item of the setup is the coding agent harness: pi — simple and configurable. On the first request it consumes 11 times less than claude desktop.

Summary#

The full benchmark ranking of models is in the article "LLM ranking — my own benchmarks".