Current bitnet-b1.58-2B-4T checkpoints contain all-zero MLP weight tensors (model generates garbage)

[grave0x]

Current bitnet-b1.58-2B-4T checkpoints contain all-zero MLP weight tensors (model generates garbage)

Summary

The current releases of microsoft/bitnet-b1.58-2B-4T, microsoft/bitnet-b1.58-2B-4T-bf16, and microsoft/bitnet-b1.58-2B-4T-gguf contain weight tensors that are entirely zero in several layers. Running the GGUF through bitnet.cpp produces repeated garbage tokens, consistent with the zeroed weights. The original releases (April 2025) are intact — the breakage was introduced by a later re-upload ("Update Model").

Affected repos / files (current main)

Repo	File	Size
microsoft/bitnet-b1.58-2B-4T	model.safetensors (U8 "quantized" format)	1,178,623,988 B
microsoft/bitnet-b1.58-2B-4T-bf16	model.safetensors	4,825,679,400 B
microsoft/bitnet-b1.58-2B-4T-gguf	ggml-model-i2_s.gguf	1,187,801,280 B

Evidence (verified 2026-08-10, direct tensor inspection via safetensors/torch)

microsoft/bitnet-b1.58-2B-4T-bf16 (current):

model.layers.0.mlp.down_proj.weight   [2560, 6912]  min=0 max=0 mean=0 nonzero=0
model.layers.0.mlp.gate_proj.weight   [6912, 2560]  min=0 max=0 mean=0 nonzero=0
model.layers.0.mlp.up_proj.weight     [6912, 2560]  min=0 max=0 mean=0 nonzero=0
model.layers.1.mlp.up_proj.weight     nonzero=0 / 17,694,720
model.layers.10.mlp.up_proj.weight    nonzero=0 / 17,694,720
model.layers.2.mlp.up_proj.weight     nonzero=6,733,450 (has data — only some layers zeroed)

microsoft/bitnet-b1.58-2B-4T (current, U8):

model.layers.0.mlp.down_proj.weight   uint8, all 0x00; weight_scale = 0.0
model.layers.0.mlp.gate_proj.weight   uint8, all 0x00
model.layers.0.self_attn.q_proj.weight  (valid 2-bit packed ternary data)

microsoft/bitnet-b1.58-2B-4T-gguf (current):

blk.0.ffn_up   I2_S, nonzero=0 / 4,423,712
blk.2.ffn_up   I2_S, nonzero=0 / 4,423,712
blk.1.ffn_up   I2_S, nonzero=4,383,655 (has data)

Note the zeroed layers differ between the bf16 repo ({0,1,10}) and the GGUF repo ({0,2}) — the GGUF was evidently built from a different broken snapshot.

Reproduction (5 lines):

from safetensors import safe_open
with safe_open("model.safetensors", framework="pt") as f:
    w = f.get_tensor("model.layers.0.mlp.up_proj.weight")
    print((w != 0).sum().item(), "/", w.numel())   # -> 0 / 17694720

Running the current ggml-model-i2_s.gguf through bitnet.cpp (llama-cli, temp=0) outputs only ? tokens — the model is non-functional.

History / root cause

The original releases are intact and work:

• microsoft/bitnet-b1.58-2B-4T commit 9ff478e2487b → model.safetensors = 1,835,292,112 B (original)
• microsoft/bitnet-b1.58-2B-4T-gguf commit 9f43072f6949 → ggml-model-i2_s.gguf = 1,844,472,032 B (original)

The re-upload commit 24edd43d41aa ("Update Model") replaced the file with the broken 1.18 GB version.

Secondary issue (tooling)

The new checkpoint format is not supported by any current converter:

• bitnet.cpp utils/convert-ms-to-gguf-bitnet.py fails with KeyError: 'U8' in SAFETENSORS_DATA_TYPES (new uint8 dtype not registered)
• upstream llama.cpp convert_hf_to_gguf.py cannot map the new ffn_sub_norm / attn_sub_norm tensor names
• Only the old-format files (original bf16 / original GGUF) can be converted and run today

Suggested fix

1. Restore the correct weights (re-upload from 9ff478e2 / 9f43072f), or fix whatever produced the zeroed tensors
2. Regenerate the GGUF from a verified-good checkpoint
3. Register the U8 dtype (and ffn_sub_norm mapping) in bitnet.cpp's converter so the new format is usable

[grave0x]

Correction/update (2026-08-11): I've now verified the original GGUF as well, and it is also broken.

The original release microsoft/bitnet-b1.58-2B-4T-gguf at commit 9f43072f6949 (ggml-model-i2_s.gguf, 1,844,472,032 B) loads and runs after renaming its arch from bitnet-25 to bitnet/bitnet-b1.58, but every I2_S quantized tensor is all-zero bytes:

output.weight        (F16): real data (mixed bytes)          <- OK
blk.0.ffn_up.weight  (I2_S): 0x00 x1MB ... all zeros         <- BROKEN
blk.0.attn_q.weight  (I2_S): 0x00 x1MB ... all zeros         <- BROKEN
blk.29.ffn_up.weight (I2_S): 0x00 x1MB ... all zeros         <- BROKEN

Inference therefore produces pure repeated garbage tokens (?), exactly like the current 1.19 GB GGUF.

Combined with the earlier findings (current -bf16 repo: layers.{0,1,10}.mlp.*_proj all-zero; current main U8 repo: gate_proj/down_proj all-zero with weight_scale = 0), every publicly-served artifact of BitNet b1.58 2B-4T appears to have corrupted/zeroed quantized weights. The only intact tensors are the (F16/BF16) embedding/output/norm tensors.

Please re-verify the quantization/upload pipeline end-to-end from a known-good training checkpoint — the zeroed tensors suggest a quantizer or upload step is emitting zeros.

[grave0x]

Final update — the entire artifact lineage is broken, including the original release.

I downloaded and inspected the original model.safetensors at commit 9ff478e2487b (1,835,292,112 B — matches the LFS size exactly). It is malformed/truncated:

hdr len: 58464   file size: 1,835,292,112
max declared tensor end offset: 3,670,057,680   <- declares ~3.67 GB of data

lm_head.weight          (BF16) off=0            first bytes: ea be 68 3f ...  <- REAL DATA
model.embed_tokens.weight (BF16) off=656670720  all 0x00                       <- ZEROS
model.layers.0.input_layernorm.weight (BF16)    all 0x00                       <- ZEROS
model.layers.0.mlp.up_proj.weight     (U8)      all 0x00                       <- ZEROS
model.layers.0.self_attn.q_proj.weight (U8)     all 0x00                       <- ZEROS

So: the file declares ~3.67 GB of tensors but serves only 1.84 GB; only lm_head contains real values, everything after is zeros. The original GGUF (9f43072f) was evidently converted from this broken state (all its I2_S tensors are 0x00 as well), and the later "Update Model" re-uploads (U8 / bf16) inherited the same zeroed weights.

Net: every publicly served artifact of bitnet-b1.58-2B-4T (original commit, current main, -bf16, -gguf) has zeroed/corrupted weights. The embeddings, norms, and projections beyond lm_head are all zeros, so inference produces only repeated garbage tokens.

Please regenerate the release files from the actual trained checkpoint — and consider an integrity check (sha256/xxhash + nonzero-weight validation) in the release pipeline.

[dev-sohaib-asif]

i also get this issue