fix(compiler): cap short-doc source text via configurable max_doc_chars (#73) - #218
Open
hudsonwa wants to merge 1 commit into
Open
fix(compiler): cap short-doc source text via configurable max_doc_chars (#73)#218hudsonwa wants to merge 1 commit into
hudsonwa wants to merge 1 commit into
Conversation
The short-doc compile path sends the whole markdown source file as a single LLM message, so an oversized doc overflows the model context and fails (issue VectifyAI#73). Add max_doc_chars (default 500k chars, configurable per-KB/ global and via the API PATCH) that truncates the payload with an explicit marker so the model knows the tail is missing. - config.py: add max_doc_chars to DEFAULT_CONFIG + GLOBAL_SCALAR_KEYS - api_models.py: expose max_doc_chars in the writable config schema - compiler.py: _maybe_truncate_doc helper + use in compile_short_doc - docs: config.yaml.example, README, examples/configuration - tests: unit + integration (RED->GREEN proven); 6 new tests
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Fixes #73 (partially, the "graceful degradation" minimum it explicitly allows).
The short-doc compile path sends the entire markdown source file as a single LLM message (
compile_short_docreads the whole file and injects it into the summary prompt). An oversized markdown doc therefore overflows the model context and either fails or has its tail silently dropped — which is exactly what breaks ingesting large markdown files with local/private models.This adds a configurable hard cap,
max_doc_chars(default 500k chars ≈ 125k tokens — a pure safety ceiling that never affects normal docs), that truncates the payload and appends an explicit marker so the model knows the tail was cut and the user can see it happened. Local-model users lowermax_doc_charsto fit their context window.Why this scope
Issue #73 proposes a full heading-aware chunking + hierarchical-synthesis feature (medium/large). This PR ships the small, safe, "at minimum" increment the issue itself lists first: graceful degradation instead of a hard failure. It's a strict safety net; structured chunking can build on top later.
Changes
openkb/config.py— addmax_doc_charstoDEFAULT_CONFIGandGLOBAL_SCALAR_KEYS(layers global → KB likepageindex_threshold).openkb/api_models.py— exposemax_doc_charsin the writable config schema (keepsGLOBAL_SCALAR_KEYS == _KB_CONFIG_WRITABLE_KEYS).openkb/agent/compiler.py— new_maybe_truncate_doc(content, max_doc_chars, doc_name)helper;compile_short_docapplies it after reading the source.config.yaml.example,README.md,examples/configuration/README.md.test_compiler.py: 4 unit tests on the helper + 2 integration tests provingcompile_short_doctruncates an oversized doc (marker present, tail gone) and passes through a doc within budget.Verification
_maybe_truncate_docdoesn't exist and the new tests fail at import (RED); with the fix, all 6 pass (GREEN).pytest tests/test_compiler.py tests/test_config.py tests/test_api.py tests/test_api_documents.py tests/test_api_watch.py tests/test_indexer.py tests/test_file_size.py→ 422 passed.ruff checkclean;ruff formatclean.No new dependencies. Diff: ~37 source lines + docs + tests.