tlg da35e94b16 fix: add triton kernels for MXFP4, fix GGUF KV cache quantization ...

- Add 'kernels' package to Dockerfile for native MXFP4 execution
  (fixes gpt-oss-20b OOM: 15.2GB→13.5GB)
- Reduce GGUF n_ctx from 8192 to 4096 and quantize KV cache to Q8_0
  to reduce VRAM usage
- Use GGML_TYPE_Q8_0 constant instead of string for type_k/type_v

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

2026-04-05 22:49:16 +02:00

kischdle/llmux

fix: add triton kernels for MXFP4, fix GGUF KV cache quantization

2026-04-05 22:49:16 +02:00

.gitignore

Initial commit with .gitignore

2026-03-31 17:58:54 +02:00

Description

No description provided

144 KiB

Languages

Python 91.2%

Shell 6%

Dockerfile 2.8%