Skip to content

Search wahloon.net

Search models, news, guides and FAQs.

All guides
Guides#quickstart#ollama#lm-studio#llama-cpp#qwen#27b

Quickstart: Run Huihui Qwen3.8-27B-abliterated with Ollama / LM Studio

A step-by-step setup for the popular Qwen3.8-27B abliterated build on Ollama, LM Studio, and llama.cpp, with hardware notes and sane defaults.

4 min readMarkdown
On this page

The Huihui Qwen3.8-27B abliterated build is one of the most downloaded 27B variants for a reason: it is widely mirrored, ships in GGUF, and fits comfortably on a 24 GB GPU at Q4. This guide gets you from zero to a working chat without guessing at flags.

Before you start

  • VRAM: roughly 16–18 GB at Q4 with a moderate context window. Leave 2–4 GB of headroom.
  • Disk: about 17 GB for a Q4_K_M GGUF, plus space for the Ollama blob.
  • Software: Ollama 0.5+ or LM Studio 0.3+. Both support GGUF and Hugging Face downloads.

If you are on a 16 GB card, use a Q3_K_M or IQ4 variant and keep context shorter. On Apple Silicon, unified memory changes the math: a 24 GB Mac can usually run the Q4 build, but the GPU budget is shared with the rest of the system.

Option 1: Ollama (fastest)

Pull the official tag

ollama run huihui_ai/Qwen3.8-abliterated:27b

This pulls the publisher's Ollama tag and starts a chat. If the tag has moved, check the model card for the current name.

Pull a specific GGUF quant

If you want to pin the exact file:

ollama run hf.co/huihui-ai/Huihui-Qwen3.8-27B-abliterated-GGUF:Q4_K_M

Ollama downloads the GGUF, creates a local model, and launches it. The first run is the slow one; later runs load from disk.

Verify what is loaded

ollama ps

You should see the model name, size, and whether it is running on GPU or CPU. If it says 100% CPU, the model did not fit and Ollama offloaded layers.

Tune the context

Ollama's default context may be smaller than you expect. Create a Modelfile if you need a longer window:

cat > Modelfile.qwen38 <<'EOF'
FROM huihui_ai/Qwen3.8-abliterated:27b
PARAMETER num_ctx 16384
PARAMETER temperature 0.7
PARAMETER top_p 0.9
PARAMETER repeat_penalty 1.15
EOF
 
ollama create qwen38-16k -f Modelfile.qwen38
ollama run qwen38-16k

Start with 8k–16k context on a 24 GB card. Longer contexts consume more KV cache and can push the model into slower memory.

Option 2: LM Studio (GUI)

  1. Install LM Studio and open the Search tab.
  2. Search for Huihui-Qwen3.8-27B-abliterated-GGUF.
  3. Pick a Q4_K_M or Q5_K_M file. Check the file size before downloading.
  4. In the Chat tab, load the model and set GPU offload to the maximum your card can hold.
  5. Set context length to 8192 or 16384 to start.
  6. Open Server if you want an OpenAI-compatible endpoint on localhost:1234.

LM Studio shows the estimated memory requirement before loading. If it exceeds your VRAM, lower the quant or reduce context.

Option 3: llama.cpp (most control)

# Install or build llama.cpp first, then:
llama-cli \
  -hf huihui-ai/Huihui-Qwen3.8-27B-abliterated-GGUF:Q4_K_M \
  -c 16384 \
  -ngl 99 \
  --temp 0.7 \
  --top-p 0.9 \
  --repeat-penalty 1.15

-ngl 99 asks llama.cpp to offload all layers to the GPU. Lower it if you run out of memory. For a server:

llama-server -hf huihui-ai/Huihui-Qwen3.8-27B-abliterated-GGUF:Q4_K_M -c 16384 -ngl 99 --port 8080

Chat template and sampling

Use the template recommended on the model card. Qwen builds generally expect the Qwen chat format; loading a Llama or ChatML template will produce broken replies.

Reasonable starting settings:

SettingStart withIf output loops
Temperature0.70.6
Top-p0.90.9
Repeat penalty1.151.25–1.3
Repeat last n5121024

If the model repeats itself, raise repeat_penalty and repeat_last_n before changing the quant. The FAQ covers this in more detail.

Common problems

It runs on CPU only. The model did not fit. Lower the quant, reduce context, or offload fewer layers. Check ollama ps or your LM Studio GPU indicator.

Replies are incoherent. Confirm the chat template, then try a different quant. A corrupted or partial download is rare but possible.

The first token takes a long time. The model is loading from disk. Subsequent requests should be faster until it is unloaded.

Context seems truncated. Raise num_ctx in Ollama or the context slider in LM Studio, and watch memory use.

Is this the right model for you?

The 27B Q4 build is the sweet spot for 24 GB cards and 32 GB+ unified-memory Macs. If you have 8–12 GB, look at a 9B Heretic build or a 12B Gemma variant instead. If you have 24 GB and want more throughput, the 35B-A3B MoE builds are worth testing, though they need more memory.

Browse the catalog to compare VRAM, licenses, and methods, and check the model card before downloading. You are responsible for how you use the model.