Preserve whitespace after inline document citations

[Paul Lizer (paullizer)]

Fixes #1289

What was wrong

Text following an inline document citation was jammed onto the end of the closing parenthesis instead of starting a new paragraph:

• ...support cited answers in chat. (Source: application_workflows.md, Page: 1)Admins can configure...
• ...warrants it (Source: document-intelligence.md, Page: 1)For best results...
• ...(Source: uploading_documents.md, Page: 1)Thank you, Paul.

Root cause

parseCitations() in chat-citations.js matches citations with:

/\(Source:\s*(...),\s*(Page(?:s)?|Sheet(?:s)?|Location):\s*(...)\)\s*((?:\[#.*?\]\s*)+)/gi

The trailing \s* inside the repeated bracket group is greedy and matches newlines, so it consumed the whitespace after the last [#citation-id] marker. The replacement callback returned only the rebuilt (Source: ...) string and never put that whitespace back.

parseCitations() runs on raw markdown before marked.parse() in renderAiMessageContent(), so a deleted \n\n didn't just remove a space — it changed how markdown parsed the remainder of the block. That's why the paragraph after the first citation was absorbed into numbered list item 5.

Reproduced in isolation against the production regex:

Input : "... in chat. (Source: application_workflows.md, Page: 1) [#181b54f7-..._1]\n\nAdmins can configure ..."
Output: "... in chat. (Source: application_workflows.md, Page: 1)Admins can configure ..."

The leftover [#guid] cleanup pass a few lines below had the same class of defect in the opposite direction — its leading greedy \s* could swallow the blank line before a stray marker that opened a paragraph.

The fix

• The replacement callback captures the trailing whitespace off the matched bracket group and re-emits it, exactly as the model wrote it. Nothing is invented, so a citation followed immediately by punctuation renders byte-identically to before.
• The [#guid] cleanup is now three ordered passes so line structure survives: a bracket run occupying a whole line is removed with its line, a run opening a line is removed with its trailing spacing, and any remaining inline run consumes only spaces and tabs. Consecutive runs like [#id-a] [#id-b] are handled as a unit.

These are whitespace-only changes. The emitted citation HTML is unchanged, so escaping, sanitization, and the XSS surface are untouched.

Before / after

Rendered HTML for the reported message, before:

<ol start="5">
<li><strong>Grounded chat:</strong> ... in chat. (Source: application_workflows.md, Page: <a ...>1</a>)Admins can configure the extraction approach ... The available modes are:</li>
</ol>

After:

<ol start="5">
<li><strong>Grounded chat:</strong> ... in chat. (Source: application_workflows.md, Page: <a ...>1</a>)</li>
</ol>
<p>Admins can configure the extraction approach for images and PDFs under <strong>Admin Settings &gt; Search &amp; Extract</strong>. The available modes are:</p>

Testing

functional_tests/test_chat_citation_whitespace_preservation.py executes the real parseCitations() in a Node sandbox and renders the result with the vendored marked bundle, so it asserts user-visible block structure rather than regex output.

Check	Result
Reported message keeps its paragraph breaks	Pass
Inline spacing preserved (same-line text, trailing punctuation, back-to-back citations, citation id on next line)	Pass
Stray [#guid] cleanup keeps line structure	Pass
Source guards for the whitespace restoration mechanism	Pass

All four fail against the pre-fix source. Each half of the fix was reverted independently to confirm both are covered.

ui_tests/test_chat_citation_paragraph_spacing.py seeds the reported message into the chat page and asserts the DOM: the numbered list item ends at its citation, the following text renders as its own <p>, and no sentence collides with a citation's closing parenthesis. Env-gated like the other UI tests.

Related suites re-run clean: test_agent_document_search_citations.py, test_markdown_citation_lookup_fallback.py, test_chat_cited_source_tracking.py, test_unicode_table_conversion.py, test_comprehensive_table_support.py, test_stored_xss_chat_workspace_rendering_fix.py, test_external_links_new_window.py, test_azure_maps_tile_token_refresh_fix.py.

Three unrelated tests fail (test_agent_citation_full_results_modal.py, test_stored_xss_chat_modal_filename_fix.py, test_file_upload_document_ingestion_security_audit.py), verified as already failing on the base branch before this change.

A review pass measured ReDoS behaviour on adversarial input (no new blowup; the pre-existing [^\]]*\] cost is unchanged) and probed CRLF, fenced code blocks, tables, blockquotes, headings, and list markers against the ^/m anchors. Every case where the new cleanup differs from the old is strictly less destructive — the old pass could break a fenced code block by pulling its closing fence onto the previous line.

Side effects

• Back-to-back citations no longer collide.
• copyMarkdown is built from the same parsed output, so copied and exported message text keeps its line breaks too.

Docs and version

• docs/explanation/fixes/CHAT_CITATION_WHITESPACE_COLLAPSE_FIX.md
• Release notes entry under v0.250.229
• config.py bumped to 0.250.229

Rebased onto Development after #1290 landed, which had claimed v0.250.228. The release notes conflict was resolved by moving this entry into a new v0.250.229 section above it; the diff against Development is these 7 files only.