Skip to content

fix(compiler): cap short-doc source text via configurable max_doc_chars (#73) - #218

Open
hudsonwa wants to merge 1 commit into
VectifyAI:mainfrom
hudsonwa:fix/markdown-content-budget
Open

fix(compiler): cap short-doc source text via configurable max_doc_chars (#73)#218
hudsonwa wants to merge 1 commit into
VectifyAI:mainfrom
hudsonwa:fix/markdown-content-budget

Conversation

@hudsonwa

Copy link
Copy Markdown

What

Fixes #73 (partially, the "graceful degradation" minimum it explicitly allows).

The short-doc compile path sends the entire markdown source file as a single LLM message (compile_short_doc reads the whole file and injects it into the summary prompt). An oversized markdown doc therefore overflows the model context and either fails or has its tail silently dropped — which is exactly what breaks ingesting large markdown files with local/private models.

This adds a configurable hard cap, max_doc_chars (default 500k chars ≈ 125k tokens — a pure safety ceiling that never affects normal docs), that truncates the payload and appends an explicit marker so the model knows the tail was cut and the user can see it happened. Local-model users lower max_doc_chars to fit their context window.

Why this scope

Issue #73 proposes a full heading-aware chunking + hierarchical-synthesis feature (medium/large). This PR ships the small, safe, "at minimum" increment the issue itself lists first: graceful degradation instead of a hard failure. It's a strict safety net; structured chunking can build on top later.

Changes

  • openkb/config.py — add max_doc_chars to DEFAULT_CONFIG and GLOBAL_SCALAR_KEYS (layers global → KB like pageindex_threshold).
  • openkb/api_models.py — expose max_doc_chars in the writable config schema (keeps GLOBAL_SCALAR_KEYS == _KB_CONFIG_WRITABLE_KEYS).
  • openkb/agent/compiler.py — new _maybe_truncate_doc(content, max_doc_chars, doc_name) helper; compile_short_doc applies it after reading the source.
  • docsconfig.yaml.example, README.md, examples/configuration/README.md.
  • teststest_compiler.py: 4 unit tests on the helper + 2 integration tests proving compile_short_doc truncates an oversized doc (marker present, tail gone) and passes through a doc within budget.

Verification

  • RED→GREEN proven: with the source fix reverted, _maybe_truncate_doc doesn't exist and the new tests fail at import (RED); with the fix, all 6 pass (GREEN).
  • pytest tests/test_compiler.py tests/test_config.py tests/test_api.py tests/test_api_documents.py tests/test_api_watch.py tests/test_indexer.py tests/test_file_size.py422 passed.
  • ruff check clean; ruff format clean.

No new dependencies. Diff: ~37 source lines + docs + tests.

The short-doc compile path sends the whole markdown source file as a single
LLM message, so an oversized doc overflows the model context and fails
(issue VectifyAI#73). Add max_doc_chars (default 500k chars, configurable per-KB/
global and via the API PATCH) that truncates the payload with an explicit
marker so the model knows the tail is missing.

- config.py: add max_doc_chars to DEFAULT_CONFIG + GLOBAL_SCALAR_KEYS
- api_models.py: expose max_doc_chars in the writable config schema
- compiler.py: _maybe_truncate_doc helper + use in compile_short_doc
- docs: config.yaml.example, README, examples/configuration
- tests: unit + integration (RED->GREEN proven); 6 new tests
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

No chunking for markdown files — entire document sent as single LLM message, fails on context overflow

1 participant