Hacker News Viewer

Inkling-2-Mini in 5.4 GB Memory

by k__ on 11/21/2026, 6:02:19 PM

https://github.com/ShinMegamiBoson/lookatmyquant

Comments

by: Aurornis

If you&#x27;re new to quantized models: Don&#x27;t get too excited about claims of fitting large models into tiny RAM footprints from random GitHub repos. There is always a tradeoff, and it&#x27;s too easy to benchmaxx KL-divergence numbers and even single benchmark scores. Just because you can get something to run on a device doesn&#x27;t mean it&#x27;s going to be useful. The headline claim of this repo (12X better) doesn&#x27;t even apply to Inkling-2-Mini<p>This repo&#x27;s README is a mess of different histories as the author bounces from one popular model to the next. It refers to vague things like &quot;the 1.2GB file&quot; that probably made sense in an early vibecoding pass, but are now lost after the repo has been pushed around to different popular models.<p>The truth is that random GitHub repos with messy READMEs almost never hold a real secret that the mainstream quantizations have missed. I would recommend sticking to mainstream quantizations like the official GGML release, Unsloth&#x27;s quants, or the more well known community quantizers like bartowski or AesSedai. They typically follow an optimized size-quality tradeoff so you can find the one that fits your needs.<p>When you see a repo like this where everything is so obviously vibecoded that the README is a rambling mix of projects and claims, you should probably go looking for a more authoritative source. The local LLM communities get spammed with 10 of these types of projects per day but I have yet to see a real breakthrough come from any of the vibecoded ones. The real work is coming from people who study this and have compute to test at scale or work on implementing advanced techniques which require inference engine support, like trellis quants.

11/21/2026, 8:47:35 PM


by: an0malous

Our future hinges on the capability gap between frontier lab models and open source local models

11/21/2026, 7:10:57 PM


by: minraws

I had a rough time with llama.cpp so fixed the bugs in it for performance some notes you need a minimum of 12gb of vram for credible performance with decent context length above 100k...<p>The best I could do with established techniques and staying at nvfp4 levels of quant, is around 40-45 toks&#x2F;s on 5070 Ti with 800 toks&#x2F;s prefill around 128k kv cache... with the kernels matching output byte for byte and basic manual verification of the patched code.<p>3060 sits around 20-25 toks&#x2F;s with prefill around 350-400 toks&#x2F;s i think maybe it can be improved but my 3060 system was ddr4 so not the greatest.<p>Either way at below 6gb you cant run beyond q4 quants at above 10-20 tok&#x2F;s with pretty small context size, afaik that gets awfully close to the roofline depending on expert cache hit rate... unless you quant the logits into the ground.

11/22/2026, 4:54:30 AM


by: syntaxing

I really wanted to like this model but it is sooooo slow. Third of the speed of Qwen 27B on my R9700. I don’t know if it’s an artifact of looped transformer or unoptimized kernel.

11/21/2026, 7:54:26 PM


by: manlymuppet

My Quantitive!

11/21/2026, 7:24:38 PM


by: XCSme

So, if we run 4 or those on a 3090, can we get better answers through boosting?

11/21/2026, 11:49:07 PM


by: Dwedit

How does this method compare to the use of an importance matrix?

11/21/2026, 6:32:57 PM


by: kroaton

50 comments, 234 points and literally nobody has tried this model. This either smells like astroturfing or the quality of HN has gone to shit completely.

11/21/2026, 9:20:19 PM


by: woadwarrior01

From the model card:<p>&gt; The model is distributed as a single checkpoint of about 20GB, in mixed NVFP4 and MXFP8 precisions.<p>What&#x27;s the point in quantizing a QAT model? Just for social media karma?

11/21/2026, 7:06:42 PM


by: cheapcheerful

&quot;Look at my quant 2&quot;<p>&#x2F;igotthatreference.jpg

11/21/2026, 7:12:39 PM


by: hugmynutus

*TL;DR* Promising initial results but shoddy research. OP&#x27;s quant did not get 1st or 2nd prize in that math competition.<p>----<p>Gave it a quick browse to verify what my local clanker told me, it seemed to check out<p>I believe it is Dynamically allocating weights based on fisher information of conversation traces &amp; For some K-quant look up table stuff it can store in an importance matrix to get better compression.<p>The results are complete nonsense and really don&#x27;t back up any of the README claims. For Inkling-2-Mini; at 5.4 GB its KL is worse than Unsloth&#x27;s UD-IQ1_S: 0.0895 vs 0.0808 on English&#x2F;chat&#x2F;code, and 0.287 vs 0.124 across 13 languages (e.g.: horrible). It then achieved better results by post-training the qaunt&#x27;d model on sample inputs (AFAIT).<p>According to anthropomorphic-clud their results for GSM8K &amp; HumanEval are within the noise regime and vastly over stated.

11/21/2026, 7:48:27 PM


by: IronWolve

Only mx version?

11/21/2026, 6:49:14 PM


by: collabspherestu

[flagged]

11/21/2026, 8:06:58 PM


by: peter_d_sherman

[dead]

11/21/2026, 8:27:47 PM