Skip to content

compression: support zstd negative ("fast") levels - #10084

Merged
ThomasWaldmann merged 1 commit into
borgbackup:masterfrom
ThomasWaldmann:zstd-negative-levels-9950
Aug 12, 2026
Merged

compression: support zstd negative ("fast") levels#10084
ThomasWaldmann merged 1 commit into
borgbackup:masterfrom
ThomasWaldmann:zstd-negative-levels-9950

Conversation

@ThomasWaldmann

Copy link
Copy Markdown
Member

Adds support for zstd's negative ("fast") levels: zstd,-128 .. zstd,22. Fixes #9950.

Why

Negative levels trade compression ratio for speed. How much they are worth depends strongly on the data, which is why the full range the level byte can hold is accepted rather than cutting it off somewhere (benchmarks in #9950):

  • on data with short matches, everything below about -10 is dominated by lz4 / none
  • on data with long repeats (disk/VM images, database files), zstd,-50 reached 2.2x lz4's ratio at 2.2x lz4's speed, and even -128 beat lz4 on both axes

borg cannot know the user's data, so it does not impose a cutoff libzstd does not have.

Compatible with existing repositories

The clevel byte is now interpreted per compressor: an int8_t for zstd, unsigned as before for everything else. Levels 1..22 encode to the byte value they always had, and CompressionSpec never allowed anything outside 1..22 for zstd, so byte values 128..255 have never been written for ctype=0x03 by borg 1.x or 2.x. No repo format change.

Verified by building unmodified borg, writing a repo with it (zstd,3, zstd,22, lz4), then reading that repo with this branch: all archives extract byte-identical and check --verify-data passes.

Implementation

CompressorBase.encode_level / decode_level classmethods (identity by default, overridden by ZSTD). Doing it per compressor rather than globally means only zstd changes meaning and every other compressor keeps storing exactly the byte it did before. A global signed interpretation was considered and rejected: ObfuscateSize uses levels 110..123 and 250, which do not fit in an int8_t, and it would have changed the byte written for lz4/none.

Two spots in repo-compress needed fixing, as they compare or display the byte without knowing the type:

  • get_csettings returned the compressor object's level while transform compares it against the stored byte. With them diverging, every object gets parsed and recompressed on every run, only to then keep the old bytes - wasteful and silent. It now returns the stored byte.
  • format_compression_spec hid the level when the byte was 255 ("level not applicable"), which would print zstd,-1 as zstd. It now decodes before that check.

Tests

Level encoding, data round trips (normal and legacy mode), a guard that no other compressor's byte changed, and a repo-compress test that a second run recognises the objects as already compressed - the regression guard for the get_csettings fix (checking only "0 recompressed" would not catch it, since misread objects get recompressed to an identical result and are then counted as "kept as-is").

Full test suite passes.

🤖 Generated with Claude Code

https://claude.ai/code/session_01UowHQDcnmow3JnYBmaGDpN

@codecov

codecov Bot commented Aug 12, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 71.42857% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 86.77%. Comparing base (4f861a3) to head (bf6a449).
⚠️ Report is 9 commits behind head on master.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
src/borg/archiver/repo_compress_cmd.py 50.00% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master   #10084      +/-   ##
==========================================
+ Coverage   86.70%   86.77%   +0.07%     
==========================================
  Files          98       98              
  Lines       17174    17277     +103     
  Branches     2609     2622      +13     
==========================================
+ Hits        14890    14992     +102     
- Misses       1586     1587       +1     
  Partials      698      698              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

…9950

zstd levels below 1 trade compression ratio for speed. They are useful:
on data with long repeats (disk/VM images, database files), zstd,-50
reached more than twice lz4's compression ratio at more than twice its
speed. Which of them is worth using depends a lot on the data, so we
accept the full range the level byte can hold rather than cutting it off
somewhere: zstd,-128 .. zstd,22.

The clevel byte is now interpreted per compressor: for zstd it is an
int8_t, for everything else it stays unsigned as before. Levels 1..22
encode to the byte value they always had, so existing repositories keep
their meaning and no repo format change is needed.

The encoding lives in the new CompressorBase.encode_level/decode_level
classmethods (identity by default, overridden by ZSTD), so only zstd is
affected and every other compressor stores exactly the byte it did before.

repo-compress compares the compression settings it wants against the ones
found in the repo, so both sides have to use the same representation:
get_csettings now returns the stored byte. Otherwise every object would be
parsed and recompressed on every run, only to then keep the old bytes.
format_compression_spec decodes the byte before checking it against 255
("level not applicable"), as 255 is a valid zstd level (-1).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UowHQDcnmow3JnYBmaGDpN
@ThomasWaldmann
ThomasWaldmann force-pushed the zstd-negative-levels-9950 branch from eb6ccf5 to bf6a449 Compare August 12, 2026 08:06
@ThomasWaldmann
ThomasWaldmann merged commit 593698a into borgbackup:master Aug 12, 2026
19 of 20 checks passed
@ThomasWaldmann
ThomasWaldmann deleted the zstd-negative-levels-9950 branch August 12, 2026 08:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

borg2: support zstd negative (--fast) compression levels

1 participant