loxc

Train a compression model on your text. Ship it. Compress and decompress with table-driven decode.

A trainable, frequency-optimized text codec in pure C99. Made for embedded systems, log pipelines, and domain-specific text. loxc belongs to the Liquid Oxygen (LOX) ecosystem, and LOX means Liquid Oxygen.

At a glance

Format generations: v2 + matrix-based v3
V3 objectives:      payload / embedded / amortized
Demo corpus v3:     39.7% external payload ratio
Demo embedded v3:   47.9% self-contained ratio
JSON v3:            36.9% ratio, 43.2 MB/s encode, 60.0 MB/s decode
C-source v3:        41.0% ratio, 34.7 MB/s encode
Runtime deps:       standard C library only

The benchmark numbers above come from the checked-in BENCHMARKS.md report on an Intel Xeon E5-2690 v4 under WSL2 with GCC 13.3.0 and five timed iterations after warmup. They are host measurements, not MCU WCET claims.

Language agnostic

loxc has no built-in language assumptions. The included demo module is trained on a public-domain text sample (Pride and Prejudice) for testing purposes only. For your own data:

• Slovak, Czech, Polish, and other natural language corpora: train on your corpus
• JSON, XML, and log lines: train on representative samples of your format
• URLs and file paths: train on a representative set
• Source code: train on files from your codebase

The codec works on bytes. Any text-like data with repeated domain-specific patterns can benefit when its table is trained on representative data.

Three lines to compress text

loxc_ctx_t *ctx = loxc_open("modules/loxc_demo.loxctab");
if (ctx != NULL) {
    loxc_buffer_t out = loxc_compress_buffer(ctx, "Hello world!", 12, 0);
    if (out.error == LOXC_OK) {
        /* use out.data / out.size */
        loxc_buffer_free(&out);
    }
    loxc_close(ctx);
}

That's it. out.data now holds compressed bytes.

See full examples -> | 5-minute tutorial ->

Why loxc?

Built for

• Domain-specific text: JSON APIs, log lines, URL paths, localization files
• Embedded systems: small table-driven runtime path and standard-library dependencies
• Repeated payloads: train once and reuse a domain table many times
• Predictable decode structure: deterministic matrix traversal and explicit table identity
• Deployment choice: external tables, self-contained embedded tables, or amortized table cost

Not for

• Archival compression, where zstd, brotli, xz, or similar codecs will usually give better ratios
• One-shot compression of unknown text, where a general-purpose compressor is simpler
• Encryption or authenticated storage; LOXC is a codec, not a cryptographic primitive

How it works

TRAINING (offline, once per corpus)

your_corpus.txt  -->  loxc_train  -->  mytable.loxctab
    |
    +-- Counts byte frequencies
    +-- Extracts deterministic dictionary candidates
    +-- V3 evaluates deterministic dictionary prefixes against the selected objective
    +-- Builds/scans canonical HIER4 and HIER8 matrix layouts
    `-- Emits .loxctab plus optional generated/static C module

RUNTIME (online, many times)

input text  -->  [encode via lookup tables]  -->  .loxc file
.loxc file  -->  [decode via lookup tables]  -->  output text

V3 training objectives

V3 separates the compression objective from the deployment model:

payload
    minimize payload bits; table size is only a deterministic tie-break

embedded
    minimize payload bits + serialized table bits for one self-contained payload

amortized --messages N
    minimize serialized table bits + N * payload bits

Dictionary candidates are ordered deterministically by isolated gain, then candidate length, then original index. The trainer scores the empty dictionary and every prefix of that order using the complete v3 layout scorer. It chooses the best prefix with deterministic tie-breaks on objective cost, payload cost, table size, dictionary count, and matrix dimension.

This is exact over the deterministic prefix search space. It is deliberately not a combinatorial search over every possible dictionary subset.

The v3 matrix idea

V3 uses real nested 4x4 or 8x8 matrix nodes. Diagonal positions save one coordinate:

HIER8 normal cell:   3-bit X + 3-bit Y = 6 bits
HIER8 diagonal cell: 3-bit X             = 3 bits

HIER4 normal cell:   2-bit X + 2-bit Y = 4 bits
HIER4 diagonal cell: 2-bit X             = 2 bits

Cells may hold a direct symbol, RAW fallback, or a child matrix. The trainer places weighted symbols and child transitions deterministically and compares HIER4/HIER8 using the selected objective.

See docs/FORMAT_V3.md for the exact grammar and on-disk ABI.

Benchmarks

The current make bench-full report intentionally uses objective-matched v3 tables:

• loxc-ext(v3_*_payload) uses --objective payload
• loxc-emb(v3_*_embedded) uses a separate --objective embedded table

That avoids charging self-contained mode for a table deliberately optimized only for repeated external payloads.

Current reference measurements

File / domain	Mode	Ratio	Encode	Decode
trainings/demo_corpus.txt (734.9 KiB)	v3 external / payload	39.7%	3.3 MB/s	45.0 MB/s
trainings/demo_corpus.txt (734.9 KiB)	v3 embedded / embedded-objective	47.9%	7.2 MB/s	41.5 MB/s
benchmarks/corpora/json_test.json (86.4 KiB)	v3 external / payload	36.9%	43.2 MB/s	60.0 MB/s
benchmarks/corpora/json_test.json (86.4 KiB)	v3 embedded / embedded-objective	38.5%	44.1 MB/s	55.9 MB/s
benchmarks/corpora/csrc_test.c (16.3 KiB)	v3 external / payload	41.0%	34.7 MB/s	45.3 MB/s
benchmarks/corpora/logs_test.txt (325.0 KiB)	v3 external / payload	33.5%	1.2 MB/s	58.2 MB/s
benchmarks/corpora/logs_test.txt (325.0 KiB)	v3 embedded / embedded-objective	55.2%	2.3 MB/s	46.0 MB/s

Large dictionaries can improve payload ratio while reducing encode throughput. That trade-off is intentional: payload, embedded, and amortized let the caller choose what should be optimized.

Baseline tools in the benchmark report are invoked through their CLI, so their small-file timing includes process-startup overhead. LOXC measurements are in-process. Use larger inputs for throughput comparisons.

Full benchmark details ->

Quick start

Build

git clone https://github.com/Vanderhell/loxc
cd loxc && make

Use the CLI

./tools/loxc_cli compress \
    --table modules/loxc_demo.loxctab --embed \
    your_file.txt your_file.loxc

./tools/loxc_cli decompress your_file.loxc restored.txt

Use as a library

#include "loxc_simple.h"
#include <stdio.h>
#include <string.h>

int main(void) {
    loxc_ctx_t *ctx = loxc_open("modules/loxc_demo.loxctab");
    const char *text = "compress me";
    loxc_buffer_t out;

    if (ctx == NULL)
        return 1;

    out = loxc_compress_buffer(ctx, text, strlen(text), 0);
    if (out.error != LOXC_OK) {
        loxc_close(ctx);
        return 1;
    }

    printf("Original: %zu bytes, Compressed: %zu bytes\n",
           strlen(text), out.size);

    loxc_buffer_free(&out);
    loxc_close(ctx);
    return 0;
}

cc -Iinclude -Imodules myapp.c libloxc.a -o myapp && ./myapp

Train on your own data

For repeated external payloads:

./tools/loxc_train \
    --input your_data.txt \
    --output modules/loxc_mytable \
    --module-name mytable --module-id 50 \
    --format v3 --objective payload

For self-contained output where the table is embedded with each deployment:

./tools/loxc_train \
    --input your_data.txt \
    --output modules/loxc_mytable_embedded \
    --module-name mytable_embedded --module-id 51 \
    --format v3 --objective embedded

For an expected number of payloads sharing one table:

./tools/loxc_train \
    --input your_data.txt \
    --output modules/loxc_mytable_amortized \
    --module-name mytable_amortized --module-id 52 \
    --format v3 --objective amortized --messages 100

V2 training remains available for compatibility.

Full tutorial -> | Cookbook ->

Examples

Working code in examples/:

#	File	Shows
1	01_hello_world.c	Smallest possible usage
2	02_compress_file.c	File operations with timing
3	03_embedded_mode.c	Self-contained .loxc files
4	04_error_handling.c	Error handling paths
5	05_training_pipeline.c	Train and use a custom module
6	06_compare_modes.c	External vs embedded size tradeoff
7	07_streaming_chunks.c	Bounded framed v3 file streaming

Run them with make examples && ./examples/01_hello_world.

API

Simple API (recommended)

loxc_ctx_t *loxc_open(const char *table_path);
void        loxc_close(loxc_ctx_t *ctx);

int loxc_compress_file(loxc_ctx_t *ctx, const char *in_path,
                       const char *out_path, int embed_table);
int loxc_decompress_file(loxc_ctx_t *ctx, const char *in_path,
                         const char *out_path);

loxc_buffer_t loxc_compress_buffer(loxc_ctx_t *ctx,
                                   const void *data, size_t len,
                                   int embed_table);
loxc_buffer_t loxc_decompress_buffer(loxc_ctx_t *ctx,
                                     const void *data, size_t len);
void          loxc_buffer_free(loxc_buffer_t *buf);

const char   *loxc_strerror(int code);

Full API reference ->

Advanced API

For direct registry and buffer control, see docs/API.md#advanced-api.

Architecture

+-----------------------------------------------------------+
| Application                                               |
|  +-----------------------------------------------+        |
|  | loxc_simple.h  (recommended)                  |        |
|  | loxc.h         (low-level)                    |        |
|  +-----------------------------------------------+        |
+--------------------------+--------------------------------+
                           |
                           v
+-----------------------------------------------------------+
| libloxc.a                                                 |
|  +--------------+  +---------------+  +----------------+ |
|  | V2 strategy  |  | V3 matrix     |  | Stream Reader/ | |
|  | paths        |  | codec         |  | Writer         | |
|  +--------------+  +---------------+  +----------------+ |
|  +--------------+  +---------------+                     |
|  | Dictionary   |  | Module /      |                     |
|  | matching     |  | table loader  |                     |
|  +--------------+  +---------------+                     |
+--------------------------+--------------------------------+
                           |
                           v
+-----------------------------------------------------------+
| Module tables (.loxctab files)                            |
| Generated/static C modules or runtime-loaded tables       |
+-----------------------------------------------------------+

Full architecture ->

File structure

include/    public headers
src/        library implementation
tools/      loxc_train, loxc_cli, loxc_bench
tests/      unit tests
modules/    generated modules
benchmarks/ benchmark inputs and reports
trainings/  training data
examples/   runnable example programs
docs/       implementation documentation

Roadmap

• v0.1.0 - Initial release
• v0.2.0 - Benchmark suite + documentation overhaul
• v0.2.4 - Release workflow + documentation fixes
• v0.3.0 - Multi-module support, streaming API
• v0.4.1 - Matrix codec v3, framed streaming, deterministic training, generated/static v3 modules
• v0.4.2 - Objective-aware v3 dictionary training, runtime/codegen fixes, objective-matched benchmarks
• v1.0.0 - Production-stable release target

Related work

Detailed comparison with Dense Codes, FSST, zstd dictionary mode, and Shared Brotli ->

Briefly, loxc is not a new compression principle. It is a practical recombination of:

• Dense-code-like prefix structure
• Learned per-corpus symbol tables
• Trained dictionary deployment
• External or embedded packaging

License

MIT - see LICENSE

Contributing

PRs welcome. See CONTRIBUTING.md.

Questions or bug reports: open an issue.