HyperQwen on one RTX 3090: a first-time developer’s guide to the 381-token claim

The internet-friendly version says Qwen3.8-27B can run at 381 tokens a second on a single RTX 3090. The useful version has one extra clause: when the answer largely reproduces text already in the prompt. That is a real and interesting inference-engineering result, but it is not the speed to expect from every chat, coding or reasoning request.

The project behind the result is HyperQwen, maintained by syv-ai (formerly qwen38-27b-rtx3090). It combines a prepared quantized checkpoint, a pinned and patched vLLM serving stack, DFlash2 speculative decoding, context lookup and prefix caching. Its README currently reports about 127 tokens/s for ordinary single-user generation, 381 tokens/s when reproducing prompt text, and about 1,035 tokens/s aggregate across 64 concurrent requests in a different batch profile. Those are different measurements, not three promises for one workload. They are maintainer-reported figures on an RTX 3090 at a 250 W power limit; we have not reproduced them independently.

What actually became faster?

An ordinary language model generates one token after another. Speculative decoding asks a smaller, faster drafter to suggest several tokens, then has the main model verify them. HyperQwen uses DFlash2 to draft seven tokens. When the answer starts copying from material already in the input—say a quoted paragraph or a code block—the project’s lookup mechanism can propose additional tokens straight from that context. The target then verifies a block of up to 16 tokens in one step. In the maintainer’s 25,000-token document-reproduction test, roughly 15 of those 16 tokens were accepted per step.

That explains both the headline and its limit. In the project’s same-server comparison, reproducing the first 60 lines of a supplied document went from 260 to 381 tokens/s when the longer verification block was enabled, whereas eight short chat prompts moved from 131 to 133. A free-form summary or question about the document did not get a 381-token result; one row was around 67–72 tokens/s. The widely repeated 82-to-381 “4.6×” progression describes successive configurations and workloads, not a universal speedup against an interchangeable baseline.

Prefix caching is another, separate lever. It saves work on a repeated prompt prefix, such as the same document and system instructions across follow-up requests. The project reports a second question against a roughly 25,000-token document falling from 22.4 seconds to 0.56 seconds of time to first token. That is not the same metric as output tokens per second. A changed prefix may miss the cache, and the first request still has to process the document.

Decide whether this is your first local-AI project

This is a good experiment if you have a supported NVIDIA card with around 24 GB of VRAM, enjoy measuring workloads, and want an OpenAI-compatible local endpoint. It is a less friendly first step if you have only a CPU, a small GPU, no spare disk, or a need for a turnkey app. The repository’s Docker path is easier than manually patching vLLM, but it still pulls a large image, downloads and requantizes a model, and compiles GPU kernels on first boot. Its measured path is an RTX 3090; other cards can work, but do not assume the same numbers or every optimization transfers. The repo documents an unresolved speculative-decoding fault on some sm80 cards.

Check the prerequisites before cloning: a working nvidia-smi, a recent NVIDIA driver that supports CUDA 13 (the project says version 580 or newer), Docker Compose with GPU access through NVIDIA Container Toolkit on Linux, and roughly 40 GB of free disk as a practical minimum. A first run also needs substantial host RAM during preparation; the project warns that low-memory hosts may need a different download path. Windows users need Docker Desktop’s WSL2 backend, not native Windows vLLM. If nvidia-smi does not see the GPU inside your intended environment, fix that before tuning the model. NVIDIA’s Container Toolkit installation guide covers the host runtime setup.

Try the documented, local-only route

The repository’s Docker quickstart is the best starting manual. On a suitable Linux/WSL2 machine, its sequence is:

git clone https://github.com/syv-ai/HyperQwen && cd HyperQwen

cp .env.example .env

make keygen

docker compose --profile single up -d

make logs

make keygen creates a local server key without displaying it. The supplied Compose file binds port 18020 to 127.0.0.1 by default, so the endpoint is local to your machine. Keep .env out of Git. Do not change BIND to 0.0.0.0 just to make a first test work: that exposes the service beyond localhost and calls for an authentication key, network controls and deliberate deployment decisions.

The first boot pulls a roughly 9.5 GB container image, downloads and prepares about 20 GB of model files, then compiles kernels. The quickstart says preparation often takes 2–15 minutes after the image pull and may take longer on slow storage or networks. make logs lets you see progress; Ctrl-C stops following logs, not the service. When the server is healthy, run make doctor for a read-only check of configuration, GPU, port, health and recent error. If you want to stop the task-owned service afterwards, use make down in that checkout; the model files remain for another run.

Start with .env.example unchanged. It already selects the single-user DFlash2 profile and prefix caching. This is a useful baseline for normal chat and coding. HyperQwen serves an OpenAI-compatible API at http://127.0.0.1:18020/v1; the default served model name is qwen3.8-27b. For a client, set its base URL to that /v1 endpoint and use the key generated in .env. The project’s client manual includes a concrete OpenCode configuration. Test a small, original prompt first, verify that an answer arrives, and note its time to first token and output length. The headline number is not the goal of this first check; a healthy, correct response is.

For a direct API smoke test from the checkout, this local-only request uses the generated key without printing it:

API_KEY="$(sed -n 's/^VLLM_API_KEY=//p' .env | tail -1)"

curl -sS http://127.0.0.1:18020/v1/chat/completions \

-H "Authorization: Bearer $API_KEY" -H 'Content-Type: application/json' \

-d '{"model":"qwen3.8-27b","messages":[{"role":"user","content":"Explain speculative decoding in one sentence."}],"max_tokens":128,"chat_template_kwargs":{"enable_thinking":false}}'

unset API_KEY

If you changed the port or served model name, substitute the values shown by your server. This is an example request based on the project’s documented endpoint and API smoke tests, not a claim that we ran it on your hardware.

Run the experiment that makes the headline meaningful

Only after the baseline works, add DFLASH_TOKENS=15 to .env and recreate that profile with docker compose --profile single up -d --force-recreate; follow make logs until it is healthy, then run make doctor again. This activates its longer verification block for quote/reproduction-heavy work. Use the same document and prompt in both configurations—for example, ask the model to reproduce a specified section of text that you supplied, and separately ask it to summarize that text. The former is a favourable case for context lookup; the latter is a useful negative control. Do not treat a copied paragraph as evidence that all coding agents will run at that rate.

Record GPU model, driver, project commit, profile, power limit, prompt and output token counts, sampling settings, context length, number of simultaneous requests, first-token delay and decode rate. Warm-up matters: the project says a first benchmark after startup can read 30–50% low. Its field-test instructions therefore run docker compose exec single bash bench/run_benchmarks.sh single twice, discarding the first result, and provide a quality-battery command if you want to compare accuracy too. Never multiply the 127, 381 and 1,035 figures together: the last is total output across many requests, not one user’s reading speed.

The longer block has a cost. The project documents four request slots and about 56,000 tokens of context in the reproduction profile, versus eight slots and about 64,000 tokens in the default DFlash2 profile. For ordinary conversation, leave the default alone unless your own measurements show a benefit. If you need much longer documents, the repo has CTX=long and CTX=huge modes with different memory and speed trade-offs. They are not free context-window upgrades; read the single-user manual before switching.

The first failures worth checking

If the container cannot see the GPU, verify the host driver and Container Toolkit rather than changing speculative-decoding knobs. On WSL2, the supplied .env enables VLLM_WSL2_ENABLE_PIN_MEMORY=1; removing it can trigger “UVA is not available.” The project documents GPU_UTIL=0.93 as a WSL2 fallback for a startup memory gate, but that is a targeted fix, not a speed setting. Exit code 137 during the CPU-side preparation stage can mean insufficient WSL host RAM or swap; the quickstart gives a 20 GB memory/8 GB swap example. If the model starts but dies on the first prompt, VRAM headroom may be tighter than the boot log suggests. The gotchas manual lists the specific signatures and fixes; match the error before changing settings.

Do not benchmark a machine that is also doing other GPU work without recording it. Do not silently swap to a different Qwen checkpoint or quantization and compare the result to the published 3090 table: this project’s drafter, vocabulary and some kernels are model-specific. A faster but wrong output is not a win. The maintainers publish quality checks, but those are limited evaluations, not a guarantee for your task. Keep a few representative answers and compare correctness as well as latency.

Where to go next

If your goal is a local coding assistant, first connect one OpenAI-compatible client to the local endpoint and try a small repository task you can verify with tests. If your workload mostly quotes files or applies precise edits, compare the reproduction mode. If you are serving many simultaneous users, read the batch-mode manual instead; its roughly 1,035 tokens/s figure is aggregate throughput at 64 concurrent requests. If you want to understand the engineering rather than just run it, the optimization notes explain the quantized embeddings, draft vocabulary, lookup verifier and prefix cache, while PATCHES.md maps the changes to code.

The lesson is more useful than the headline: speed depends on what the model is doing. Copying text already in its context, composing an original answer, processing a new long prompt and answering 64 people at once stress different parts of the system. HyperQwen gives developers a unusually well-documented way to see those differences on their own hardware. Start with a working local server, change one knob at a time, and publish your measurements with the workload attached.