Add BESC Engagement Checker: score posts using the real For You ranki… - #32
Open
BESCLLC wants to merge 55 commits into
Open
Add BESC Engagement Checker: score posts using the real For You ranki…#32BESCLLC wants to merge 55 commits into
BESCLLC wants to merge 55 commits into
Conversation
…ng weights New Next.js app (besc-engagement-checker/) that lets users paste a draft post and get a live 0-100 score, per-action signal breakdown, and visibility- filtering risk flags — all computed from the actual production values in this repo (home-mixer/params/param.rs RankingScorer weights, author-diversity decay, OON discount) plus rule citations from visibility-filtering/ and botmaker-rules/scarecrow/. Includes a polished, responsive glassmorphism UI with a live-updating composer and Railway deploy config.
|
Caution Review the following alerts detected in dependencies. According to your organization's Security Policy, you must resolve all "Block" alerts before proceeding. Learn more about Socket for GitHub.
|
Swap the placeholder green/purple theme for a gold/black palette sampled from the real BESC logo, use the actual logo as header/footer mark and favicon, add a metallic gradient score ring, and link @BESCLLC, dev @safudev0702, and bescfinancial.com in the header and footer. Also fixes a minor mobile horizontal-overflow issue from an absolutely positioned tooltip.
Paste a real x.com/.../status/... URL to pull live post text, media, and engagement metrics via Vee3's x-twitter MCP tools, prefill the composer, and compare the BESC predicted score against real performance. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A9oF1GYEbb9zvnk4nRzBmh
New rule-based optimizer (no external API/LLM) that mechanically fixes draft posts against the same weights lib/scoring.ts already scores against: collapses !!!/??? bursts, de-shouts ALL-CAPS, caps hashtag stuffing, strips boilerplate CTA phrasing, adds a genuine reply hook when missing, and trims to the 280-char limit. Runs multiple passes so later fixes can't reintroduce a pattern an earlier one already cleaned up, and only keeps a change if it provably raises the BESC Score via the existing analyzePost() — so "optimized" is never just vibes. Composer now has an "Optimize for the algorithm" button showing the before/after score, each applied fix with its reasoning, and a one-click "Use this version" to adopt the rewrite. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A9oF1GYEbb9zvnk4nRzBmh
Surveyed home-mixer/scorers, visibility-filtering/rules, safety-label- user-agg, and botmaker-rules/scarecrow/bot for real production weights/thresholds not yet reflected in the scorer, and added the ones derivable from a draft's own content plus optional public author stats: - Cold-start boost check (author_cold_start.rs, ColdStartFollowerCap= 1000, ColdStartMaxPostAgeSecs=86400): accounts under 1,000 followers posting within 24h can get force-boosted into a top slot, bypassing normal score-based ranking. New optional "follower count" field in the composer surfaces this, and it's now also threaded through from live tweet imports (previously fetched via Vee3 but discarded). - Repeated-NSFW account-label escalation risk (safety-label-user-agg/postToUserLabelRules.strato): flags that going NSFW repeatedly risks a 7-day account-level label, not just a per-post interstitial. - Enriched the URL-verdict risk to distinguish a LOW_QUALITY downrank from an UNSAFE verdict's hard MALICIOUS_URL_DROP (full non- distribution), per tweet_label_drops.rs. - Noted duplicate-text spam detection (BBQDuplicateTextRepliesProd.bot) applies to reply text too, not just original posts. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A9oF1GYEbb9zvnk4nRzBmh
Live-imported tweets are showing all-zero engagement metrics (likes, reposts, replies, quotes, bookmarks) even for posts that clearly have engagement — text/author fields work, so this is almost certainly a field-name mismatch in the defensive key-alias guessing in twitter-import.ts, built blind since this sandbox can't reach mcp.vee3.io to inspect the real response shape. Logs the raw payload server-side and also returns it as _rawDebug in the API response so the actual field names can be confirmed and the mapping fixed for real, instead of guessing again. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A9oF1GYEbb9zvnk4nRzBmh
No external LLM dependency — uses "compromise" (local, dependency-free
NLP library) plus a hand-built filler/hedge-word dictionary to extend
the deterministic in-house optimizer beyond regex pattern-matching:
- Detects filler/hedge words ("very", "just", "actually", "I think"),
passive-voice sentences, and weak sentence-openers, and factors them
into the action-probability estimates as a modest "wordiness"
penalty (clearly labeled as a general writing-craft heuristic, not a
repo-cited weight, matching the existing risk/tip distinction).
- New auto-fix rule strips filler words, handling "in order to" -> "to"
specially so grammar survives, and re-capitalizes the new sentence
start when a leading filler clause is removed.
- New tips for weak openers and heavy passive voice.
Also fixes a real bug the new rule exposed: the optimizer's per-step
gate previously used >=, which let a rule that didn't actually move
the (rounded, floor-clamped) score still get reported as "applied".
Now the walk stays permissive (>=, needed to cross plateaus where a
badly-spammy draft sits pinned at score 0 for several necessary
staging steps before a real jump) but the result is only surfaced if
the overall before/after score genuinely improved, or a hard
constraint (char limit) fired.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A9oF1GYEbb9zvnk4nRzBmh
Drop the "Positive signal"/"Negative signal" raw contribution numbers from the score panel — they're abstract sum-of-weights in arbitrary units with no clear action attached, and the same information (author diversity decay, OON discount) is already explained in plain language by the risk panel just below. Keep the two multipliers that are directly meaningful on their own. Signal breakdown (18 action rows) now shows only the top 3 by contribution, with the rest behind a "Show all N signals" toggle — this was the single longest section on the page for something most users only need to skim, not read start to finish. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A9oF1GYEbb9zvnk4nRzBmh
Got the actual raw x-twitter_tweet_info response from a live import (Railway logs) and fixed the two remaining mismatches it exposed: - authorVerified was checking is_blue_verified/verified, but Vee3's real field is blue_verified — was always false. - authorFollowers was checking followers_count/followers, but Vee3 only returns sub_count on the author object — was always 0, which also meant the cold-start boost check could never trigger for imports. The likes/retweets/replies/quotes/bookmarks aliases already matched (fixed in a prior commit) — confirmed against this real payload. Also capture "views" (present as a string in the raw payload) and surface it as a new stat in the live-performance panel, and remove the temporary debug logging/_rawDebug passthrough now that the real shape is confirmed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A9oF1GYEbb9zvnk4nRzBmh
Next.js recommends it for next/image (used for the logo/banner) in production and warns without it on every deploy. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A9oF1GYEbb9zvnk4nRzBmh
Import hanging forever: callVee3Tool had no timeout on client.connect() (the initial MCP handshake isn't covered by callTool's own request timeout), so a slow/unresponsive Vee3 endpoint could spin the UI indefinitely with no error and no way out. Added a hard 20s deadline on both connect() and callTool(), plus a 25s client-side AbortController on the import fetch as a backstop so the button always resolves to either a result or a clear "timed out, try again" error. Broken mobile signal breakdown: the weight/probability detail was a :hover-triggered absolute overlay, but touch has no reliable hover equivalent — mobile browsers can get stuck showing it after a tap, which is exactly what was in the screenshot: dozens of overlapping tooltip boxes stacked on top of the page. Replaced with always-visible inline text under each bar, which has no such failure mode. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A9oF1GYEbb9zvnk4nRzBmh
The previous timeout fix used an external Promise.race that stopped waiting on a hung client.connect()/callTool() but never actually cancelled it, then immediately called client.close() underneath the still-running operation in finally. Closing a live connection out from under an in-flight request is a plausible way to trigger an unhandled internal error in the transport's stream/reader handling — which crashes the Node process outright (unlike a promise rejection), dropping the connection mid-request. That would surface to the browser as exactly the connection-reset "Load failed" error reported, not a clean JSON error response. Switched to the SDK's own supported cancellation: a single AbortController's signal passed to both connect() and callTool()'s RequestOptions, which the SDK docs confirm raises a clean AbortError from request() instead of abandoning the operation. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A9oF1GYEbb9zvnk4nRzBmh
A post that hashtags its own brand (#BESC) and also writes it in caps
in the body ("BESC Exchange App Update...") isn't shouting — it's just
the name — but both the ALL-CAPS risk score and the auto-fixer treated
any all-caps word as shouting with no way to tell the two apart. Real
example: optimizing a BESC announcement silently rewrote it to "Besc
Exchange App Update".
Fix: a word echoed as a hashtag elsewhere in the same post (case/
punctuation-insensitive) is now protected from both the allCapsWordRatio
scoring penalty and the auto-fixer's title-casing, via new shared
getHashtagWordSet()/isShoutingCapsWord() helpers used by both scoring.ts
and optimize.ts (single source of truth, so what counts as a risk and
what gets auto-fixed can't drift apart again).
Also fixed a subtler bug this surfaced: the protected-word set was
being recomputed per-pass from whatever text survived so far, so if an
earlier pass's trim-to-limit truncated off the trailing hashtags, a
later pass's caps-fix would no longer see the brand as protected and
mangle it anyway. Now it's computed once from the original draft
before any pass runs, and threaded through as a fixed reference.
Verified against the real BESC announcement tweet that exposed this,
plus a regression check that genuine shouting ("AMAZING", "RIGHT NOW")
still gets corrected.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A9oF1GYEbb9zvnk4nRzBmh
Every text/number <input className="w-full"> in the composer sits in a flex row next to shrink-0 icon/button siblings. Flex items default to min-width: auto, and for a single-line text input that resolves to its full intrinsic preferred width (unlike wrappable text) — on a narrow phone viewport that intrinsic width exceeds the available space, forcing the whole row (and everything above it in the same glass-panel) wider than the screen. Combined with overflow-x: hidden on body, the overflow wasn't scrollable, it was just permanently clipped: the import button, follower-count field, and the post-count stepper's +/- controls were all invisible off-screen on mobile. Fixed the three w-full inputs with min-w-0, and hardened the label+control rows (mutual-reply/NSFW checkboxes, the post-count stepper) the same way: min-w-0 + truncate on the label so it wraps/ shrinks first, shrink-0 on the fixed-size control so it's never what gets pushed off-screen. Verified with a headless render at a 390px viewport (iPhone-width): document.scrollWidth now equals clientWidth (no overflow), and all controls are visibly present within the viewport. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A9oF1GYEbb9zvnk4nRzBmh
New standalone ollama-service/ (Dockerfile + setup README) meant to run as a second Railway service in the same project, with a model baked into the image at build time so restarts are instant instead of re-pulling multi-GB weights. Reached over Railway's private network, not exposed publicly. App side: lib/ollama.ts calls it (with a hard 45s timeout — no repeat of the earlier Vee3 "loads forever" bug) to generate creative rewrite candidates, fed a prompt describing the actual scoring signals (reply weight, spam penalties, char limit, "never invent facts" instruction to prevent hallucinated claims/numbers). /api/optimize now runs the existing free deterministic optimizer first as always, then optionally layers AI candidates on top — each one scored via the same analyzePost() everything else uses, and only surfaced if it scores strictly higher than the deterministic result. If OLLAMA_URL/OLLAMA_MODEL aren't set, or the call fails/times out, this silently falls back to the deterministic-only result — the AI layer is a bonus, never a dependency for the feature to work. Composer UI shows AI candidates in their own section with individual scores and a per-candidate "Use this version" button. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A9oF1GYEbb9zvnk4nRzBmh
… inputs The earlier input min-w-0 fix was real but incomplete — and worse, my own verification of it was wrong: I checked document.documentElement .scrollWidth, which this app's overflow-x:hidden setup keeps pinned to the viewport width regardless of actual overflow. document.body .scrollWidth (the number that actually reflects it) showed the composer was still ~40px too wide even right after that fix landed, which is exactly why it looked fixed then broke again in practice. Root cause, one layer up from the inputs: page.tsx's two-column results grid has no explicit grid-template-columns below the lg: breakpoint, so its two direct children (the composer wrapper, the results wrapper) are grid items with the standard min-width: auto default — same mechanism as the flex-item bug fixed earlier, just at the grid level. Any intrinsic content width bubbling up from either column could force the whole grid track, and with it the page, wider than the viewport. Added min-w-0 to both grid-item wrappers. Also found and fixed a second, separate overflow source while re-verifying: the footer's labeled SocialLinks row (3 pill links) had no flex-wrap, overflowing by itself on narrow phones. Added flex-wrap + justify-center. Verified properly this time with document.body.scrollWidth (not documentElement) at three points — before results load, ~3s after (matching the user's "shows and fits for a second then overflows" report), and scrolled to the footer — all now exactly 390 at a 390px viewport, with screenshots confirming visually. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A9oF1GYEbb9zvnk4nRzBmh
Given headroom on the paid Railway plan, swap the default from llama3.2:3b to qwen2.5:14b-instruct for meaningfully better rewrite quality. Updated alongside: - README sizing guidance: ~9GB model needs real RAM headroom for inference on top of it, now recommending 16GB (was 4GB for 3B) with fallback options noted if that's too much. - lib/ollama.ts request timeout: 45s -> 90s, since CPU inference on a 14B model is meaningfully slower per request than 3B. - Composer's optimize() call now has its own 100s client-side abort timeout (previously none), matching the same defensive pattern used for the Vee3 import after the "Load failed" incident — a slower backend path should degrade to a clear timeout message, not an indefinite spinner. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A9oF1GYEbb9zvnk4nRzBmh
Free accounts are capped at 280 characters, but X Premium subscribers and Verified Organizations can post far more (4,000 for Premium, up to 25,000 for Premium+/Verified Orgs). Everything here was hardcoded to 280 regardless of account status, so a verified user's legitimate long-form draft would get force-trimmed by the optimizer and flagged with a "too long, trim it down" tip that doesn't apply to them. Since available data can only tell us verified yes/no, not which paid tier, 4,000 is used as a floor that's accurate for every verified tier (never overstates what a given tier allows). New getCharLimit() in scoring.ts is the single source of truth, threaded through: - optimize.ts: trim-to-limit and add-reply-hook's length check both respect the dynamic limit instead of a hardcoded 280. - scoring.ts: the "too-long" tip is skipped for verified drafts (long-form is an intended, legitimate format for them), replaced by a much higher-threshold, differently-worded tip about front-loading the point rather than trimming length. - Composer: new "Verified / Premium checkmark" toggle, dynamic char counter, and — fixing the same "captured but discarded" pattern as the earlier follower-count bug — imported tweets' authorVerified now actually feeds this instead of being silently dropped. - request.ts: bumped the payload size cap above the new verified limit so a legitimate 4,000-char draft doesn't get truncated before it's even scored. Verified with real requests: a >280-char draft gets trimmed and tip-flagged when isVerified is false, passes through untouched with no false tip when true. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A9oF1GYEbb9zvnk4nRzBmh
"Can't tell if the AI is working": the Optimize response previously
just omitted aiCandidates whenever there was nothing to show, so
"not configured", "Ollama errored", and "checked but found nothing
better" were all visually identical, a blank space with no
explanation. Added an aiStatus field ("disabled" | "error" |
"no_improvement" | "found") that's always present and always
rendered as a small status line, so it's never ambiguous whether AI
was even attempted.
Also swept every user-facing string (tips, risks, labels, hero copy,
meta title/description) for em dashes and replaced each with
conventional punctuation, a period, comma, or colon depending on what
reads most naturally, for more professional, direct end-user wording.
Left dev-facing code comments alone since those aren't part of the
front end.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A9oF1GYEbb9zvnk4nRzBmh
Instead of only typing in your follower count and verified status manually, you can now paste your @handle, hit Fetch, and it pulls your real follower count + verified status live via Vee3's x-twitter_user_info tool, auto-filling both fields. New lookupAuthor() in twitter-import.ts (mirrors the existing tweet import's defensive field-alias picking), a new /api/lookup-author route, and a Composer row above the manual follower-count input with the same timeout/error-handling pattern used everywhere else in this app (25s client-side abort, clear error on failure). Manual entry still works as a fallback if the lookup fails or someone would rather just type a number. Verified at a 390px mobile viewport that the new row doesn't reintroduce the overflow bug fixed earlier in this session. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A9oF1GYEbb9zvnk4nRzBmh
lookupAuthor() failed on @safudev0702, a real, verified account confirmed working via the tweet-import path, so the bug is the same class as the earlier tweet-data mapping issue: x-twitter_user_info is a different Vee3 tool than x-twitter_tweet_info, and its response shape was guessed blind since there's no real sample of it. Logs the raw payload server-side so the actual field names can be confirmed from Railway logs and the mapping fixed for real. Also fixes an em dash the earlier front-end sweep missed: it skipped twitter-import.ts as "backend only," but Error() messages thrown here do surface to users via the import/lookup error text in the UI. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A9oF1GYEbb9zvnk4nRzBmh
…bake Your deploy log showed "total blobs: 0" right after startup, meaning the model never actually made it into the running container despite the build-time bake step appearing to complete. Baking multi-GB model weights into a Docker layer via a backgrounded "ollama serve &" during RUN is a known-fragile pattern (base image volume declarations, layer caching, or the pull simply not finishing before the RUN layer commits can all silently discard it) and that's clearly what happened here. Replaced it with the more reliable pattern: start.sh starts the real server, waits for it to accept connections, then pulls the model only if it isn't already present (checked via `ollama list`), so restarts against a populated volume are fast and restarts against an empty one still work, just slower. README now walks through attaching a Railway Volume at /root/.ollama so a successful pull actually persists across redeploys, plus a note on how to verify it worked (watch for "total blobs: N" with N > 0 in the logs, not 0). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A9oF1GYEbb9zvnk4nRzBmh
Got the actual raw x-twitter_user_info response from a live lookup (Railway logs) for @safudev0702. The handle field is "profile", not user_name/screen_name/username like I'd guessed (and unlike x-twitter_tweet_info's embedded author object, which really does use screen_name — the two Vee3 tools don't share a naming convention). That's why the lookup failed outright: authorHandle came back empty and the function threw before ever reading followers/verified. followers_count/verified aliases (sub_count, blue_verified) were already correct against this payload, confirmed here rather than assumed. Removed the temporary debug logging now that the real shape is known. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A9oF1GYEbb9zvnk4nRzBmh
Real deploy log showed a request still mid-generation (410 input tokens processed, sampler running) when it hit the 90s internal timeout — generation was unbounded, nothing capped how many tokens the model could produce before the timeout became the only thing stopping it. Added num_predict=250 (two short rewrite variants need well under that), which puts a real ceiling on worst-case latency instead of just hoping the model stops on its own in time. Also widened the timeout chain with a bit more margin: server-side 90s -> 100s, client-side 100s -> 115s (kept safely above the server timeout so a slow AI call resolves to a graceful aiStatus: "error" fallback with the deterministic result intact, rather than the client aborting the whole request first). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A9oF1GYEbb9zvnk4nRzBmh
The prompt previously only mentioned a hand-picked handful of rules (reply weight, ALL-CAPS, hashtags, boilerplate, char limit) written as prose, disconnected from the actual WEIGHTS in scoring.ts. Rewrote it to import WEIGHTS and BOILERPLATE_PHRASES directly and interpolate the real numbers, covering the full ranked hierarchy the deterministic scorer actually uses: - Every positive action ranked by value, including share-via-copy-link (20.0, the single highest weight in the whole model) and the bidirectional-reply boost, which the old prompt didn't mention at all despite reply being the headline "optimize for this" signal. - Every severe negative action ranked by severity (report -234, mute -58.8, not-interested -43.2, block -31.2), with what actually triggers them. - The filler-word/passive-voice/weak-opener craft signals from the in-house NLP layer. Pulling from the real constants (rather than hardcoding numbers in prose) means this can't quietly drift out of sync with the actual scorer as weights change. Also threads the correct char limit through based on verified status (280 vs 4,000), so the prompt tells the model the real constraint instead of always assuming 280. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A9oF1GYEbb9zvnk4nRzBmh
qwen2.5:14b-instruct repeatedly timed out on CPU-only inference even after bounding output length (num_predict) and widening the timeout to 100s+ — one deploy log showed a request still stuck mid-generation at that ceiling. That points at raw CPU throughput as the wall, not something further prompt or timeout tuning fixes. Switched the default to qwen2.5:7b-instruct: same weight-aware prompt, roughly 2x the tokens/sec, a tested tradeoff of some quality for actually finishing within a web request. Updated README sizing guidance to match (8GB RAM / 12GB volume instead of 16GB / 20GB) and left a note on how to size back up if real CPU headroom allows it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A9oF1GYEbb9zvnk4nRzBmh
…call Two consecutive timeouts at exactly 100s each, with no variance, is more consistent with the request never actually reaching a working Ollama endpoint than with genuine model slowness — but the previous error handling couldn't distinguish the two, "timed out after 100s" covers both a slow model and a request that silently hung the whole time for an unrelated reason. Added checkOllamaHealth(): hits Ollama's lightweight /api/tags first, with its own short 8s timeout, before ever attempting the real (possibly 100s) generate call. This surfaces a real, specific error fast for the actual failure modes that would otherwise masquerade as a slow model: OLLAMA_URL pointing at an unreachable host, private networking not enabled between the two services, the ollama service being down, or OLLAMA_MODEL not matching what's actually loaded there (reports the real available model list in that case). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A9oF1GYEbb9zvnk4nRzBmh
A real deploy log showed 683 input tokens for a single request under the previous, more verbose weight-aware prompt — on CPU, prefill time scales directly with prompt length, so that's real latency spent before generation even starts, on top of the up-to-250 output tokens already bounded by num_predict. Rewrote the same information (the full real weight hierarchy from scoring.ts, spam triggers, craft signals) in a terse, data-dense format instead of full prose sentences: same numbers, same coverage, roughly 60% fewer characters. Nothing about what the model is told to optimize for changed, only how many tokens it costs to say it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A9oF1GYEbb9zvnk4nRzBmh
Real example from testing: the 7b model rewrote a "heading into review, update once it's live" app-update post into "19 Smaller Fixes Now Live" — flipping a pending status into a false completed one, directly violating the "never invent facts, preserve every claim exactly" instruction in its own prompt. A higher BESC Score only means a candidate fits the algorithm's signals better; it says nothing about whether the AI preserved the actual facts, and this proves it can't be assumed. Added an explicit warning above the AI candidates list so "Use this version" isn't one click away from publishing something inaccurate without a second look. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A9oF1GYEbb9zvnk4nRzBmh
Three consecutive requests all timed out at exactly 100s with the 7b model, no variance, right after the health check confirmed connectivity and model-loading were both fine. Going 14b -> 7b already failed to fix this once; that's a strong signal the ceiling isn't model size, it's this Railway service's actual CPU allocation. Switching to llama3.2:3b, the smallest reasonable model, doubles as a real diagnostic: if this also times out, it conclusively rules out model size and points straight at Settings -> Resources on the ollama service needing more CPU, regardless of what the account's overall plan tier is. Documented that in the README directly. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A9oF1GYEbb9zvnk4nRzBmh
You confirmed 24 vCPUs allocated to the ollama service, which rules out CPU allocation as the bottleneck outright, three different model sizes all hit the same ~100s ceiling despite that. The likely explanation left standing: Ollama's automatic CPU thread detection is known-unreliable inside containers and can under-count what's actually schedulable, so it may have been running on far fewer threads than the 24 actually available regardless of model size. Set num_thread explicitly via /api/generate's documented per-request options (the same mechanism already used for num_predict), rather than guessing at a server-level env var name that's inconsistent across Ollama versions in the wild. Configurable via OLLAMA_NUM_THREAD if the real usable core count ever differs from what Railway shows. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A9oF1GYEbb9zvnk4nRzBmh
The 100s timeouts across 14b/7b/3b were cold-start, not a CPU/model-size ceiling: the first /api/generate call after boot pays to page weights into RAM, racing the app's request timeout. Confirmed 3b is fast and reliable once warm, so step up to qwen2.5:7b-instruct and eat the warm-up cost once at container start instead of on a live user's first request.
Gemini is preferred when GEMINI_API_KEY is set: no cold-start/CPU-quota risk since it's a hosted API, so it's a more reliable default now that billing supports it. Ollama stays as a free fallback when Gemini isn't configured. Shared prompt/parsing logic factored out to lib/aiPrompt.ts so both providers stay in sync with the real scoring weights.
gemini-2.5-flash just 404'd in production: "no longer available to new users." Dated model IDs can get retired out from under new API keys without warning. Google's -latest alias always resolves to their current recommended flash-tier model and gives a 2-week emailed notice before the target changes, instead of a cold 404.
…FW OON drop, AI-slop label, video duration floor
Full audit of the repo outside besc-engagement-checker turned up several
signals the scorer wasn't using or was describing imprecisely:
- Self-flagged NSFW media isn't just a blur for followers — TweetNsfwUserDropRule
hard-drops it from all out-of-network recommendation reach (registry.rs:145,
OON-only), while the interstitial blur applies everywhere. The risk detail
undersold this.
- The cold-start boost detail overstated the mechanism: a post must already be
ranking in the top 85% of candidates to even qualify, the boost lifts its
score to match whatever's sitting around rank 15 (not "a top slot"), and it
stops applying once the post crosses 1,000 views (ColdStartImpressionThreshold,
never previously cited).
- Added TopicOonWeightFactor=0.5 as an aside on the OON-discount risk — topic-
matched recommendation reach is discounted 50%, not the generic 25%.
- New tip: MinVideoDurationMs=10,000 — a video under 10s gets its VQV weight
forced to exactly 0.0, unconditional on quality.
- New tip: AI-slop phrasing ("delve into", "game-changer", "unlock the power
of"...) matches X's abuse-enforcement llm_slop_post classifier category,
which attaches a 30-day RiskyHighVizReply label. Cited honestly: nothing in
visibility-filtering consumes that label in this repo, so the exact
downstream effect isn't claimed, just that the label gets attached.
- Enriched the URL-verdict risk with two more scarecrow bots, including the
one that shows a bad domain verdict retroactively labels past posts too,
not just new ones.
Audited but deliberately left out: FOSNR hate/abuse/violence labels and
NSFW_TEXT (no real classifier here to back a heuristic without high false-
positive risk), RISKY_HIGH_VIZ_REPLY's reply-specific trigger (no reply-mode
context in the current UI), and all account-level/history-dependent labels
(user-cred-v2, abuse-enforcement-service account rules, AGATHA_SPAM) — none
of these are self-assessable from a single draft pre-publish.
Automates the last manual step in the optimize flow: when the top AI candidate scores >= AI_AUTO_APPLY_MARGIN (5) points higher than the deterministic result, it's applied to the draft automatically instead of requiring a "Use this version" click. Full silent auto-apply was rejected as too risky — this app has a real history of AI rewrites drifting a claim's meaning (see the fact-check warning already above the candidate list), so silently swapping text without the user looking at it first reintroduces that exact risk. Instead: auto-apply is opt-out, not silent. A prominent banner with the same fact-check warning plus a one-click Undo appears whenever it fires, and any manual edit to the text clears it. Below the margin, or with multiple candidates in play, the existing manual per-candidate review flow is unchanged. Verified end-to-end in a real browser (Playwright, mocked /api/optimize): a >=5pt margin auto-applies and Undo correctly reverts; a smaller margin falls back to the manual flow untouched.
… real-algorithm checks Read the actual repo README plus the source it links (ranking_scorer.rs, home-mixer/filters/) end to end to verify the checker matches X's real logic exactly, per request. Found a real bug, not just a missing citation: - The "Reply to a mutual follower" checkbox was backwards. Verified in ranking_scorer.rs:180-183, bidirectional_boost_eligible() requires in_reply_to_tweet_id.is_none() && retweeted_tweet_id.is_none() — the +15 reply-weight boost only ever applies to ORIGINAL posts shown to a mutual-follow viewer, and explicitly excludes replies. The old checkbox gave the boost to exactly the case the real algorithm excludes it from. Replaced with two correctly-scoped fields: isReply (post type) and hasMutualFollowAudience (audience relationship, disabled when isReply is checked since it's moot). Cold-start-boost eligibility also required original-post-only (author_cold_start.rs) and wasn't checking that either — fixed. - New risk: replies/reposts get hard-dropped from recommendations to non-followers entirely (OONRetweetReplyFilter), not just discounted, and still eat the standard 0.75x OON multiplier even when shown to followers (EnableOonRescoreForInNetworkRepliesRetweets, on by default) — verified in home-mixer/filters/oon_retweet_reply_filter.rs and ranking_scorer.rs. - New risk: AgeFilter hard-excludes any post older than 48h from For You ranking candidates entirely (home-mixer/params/config.rs:36, MAX_POST_AGE), separate from and stricter than the existing 24h cold-start eligibility window. Fires when checking an imported/older tweet via postedHoursAgo. Verified end-to-end in a real browser (Playwright): both checkboxes render correctly, the mutual-audience checkbox disables when isReply is checked, and the outgoing /api/analyze payload carries the new field names/values. Checked but correctly left out: NewUserMinEngagementFilter and the viewer-new-user OON weight variant are both gated on the VIEWER's account age, not the author's or the post's — nothing a pre-publish draft checker can act on, and the latter is inert in production anyway (NewUserAgeThresholdSecs defaults to 0).
… manual checkboxes TweetImportResult already fetches the raw Vee3 tweet object but was only reading a subset of it. Added detectIsReply() (checks in_reply_to_status_id in a few plausible v1.1/nested spellings, plus a "replied_to" entry in referenced_tweets for the v2 shape) and detectSensitive() (possibly_sensitive), following the same defensive multi-path pick() pattern already used for media type. Composer now sets isReply and nsfw straight from an import instead of leaving them at their unchecked defaults. hasMutualFollowAudience is deliberately left alone by import — it's an aggregate fact about overall follower composition, not something a single tweet lookup can tell us, so pretending otherwise would be dishonest rather than a real auto-fill. Verified the detection logic with unit tests against synthetic fixtures for both the v1.1-legacy and v2 tweet shapes (in_reply_to_status_id present/null/ absent, nested legacy.*, referenced_tweets with replied_to vs quoted, possibly_sensitive true/absent) — all passed. Couldn't verify against a live Vee3 response in this sandbox (mcp.vee3.io isn't in the network egress allowlist here), so this is worth a real import test once deployed.
…ring an existing draft New flow for when the user has an idea but no draft yet: describe it in loose notes, get back full posts already optimized for the same weighted- action ranking, pre-run through the deterministic optimizer, scored and sorted so the best option leads. - lib/aiPrompt.ts: buildGeneratePrompt(), a distinct prompt from the rewrite path — no ground truth text to preserve here, so the operative instruction is "only use facts/names/numbers given in the context, never invent specifics." Fabrication risk is real and higher than the rewrite case. - lib/gemini.ts, lib/ollama.ts: refactored each provider's HTTP-calling logic into a shared callXForVariants(prompt, n) so rewrite and generate reuse the same request/timeout/parsing code, differing only in which prompt builder feeds it. - lib/request.ts: parseSharedContextFields() factored out of parseAnalyzeRequest so the new parseGenerateRequest doesn't duplicate the mediaType/link/isReply/... validation logic. - app/api/generate/route.ts: new route. Every raw candidate gets run through the existing optimizePost() before being returned — generated content gets the same mechanical fixes and gating as everything else in this app, never a shortcut around the deterministic scorer. - components/Composer.tsx: collapsed-by-default "Don't have a draft? Generate one from an idea" panel above the main textarea, reusing all the same contextual toggles (media type, isReply, hasMutualFollowAudience, etc.) already set for the draft. Carries an explicit warning that this is fully AI-authored, not a rewrite of the user's own words, so there's no ground-truth text to check it against — verify names/numbers/claims before posting. Verified end-to-end in a real browser (Playwright, mocked /api/generate): payload carries context plus all contextual fields, fact-check warning renders, candidates display sorted by score, and picking one correctly populates the main draft textarea and collapses the panel.
…slop prompt) and add a copy-to-clipboard button
Quality fixes for both the rewrite and generate-from-idea AI paths:
- estimateMaxOutputTokens() replaces the flat 250-token cap in both
providers. That flat cap was already tight enough to risk truncating the
last of 2-3 short variants, and for a verified account's up-to-4,000-char
limit it wasn't remotely enough for even one full-length variant. Now
scales with charLimit x numVariants, floored at the old 250 so nothing
regresses, ceilinged (2000 shared, 500 on the Ollama path specifically)
so a verified draft times many variants can't blow generation time up
unboundedly — especially on Ollama's CPU path, which has a real
documented history of hitting its own timeout.
- generatePostsFromContext(Gemini) defaults bumped: 5 variants on Gemini
(hosted, cheap, no cold-start risk) for a wider best-of-N sample since
"write me the best tweet" has no single obviously-correct direction the
way tightening an existing draft does; kept at 3 on Ollama on purpose to
protect the timeout budget.
- Both prompts (lib/aiPrompt.ts) now explicitly warn off the same AI-slop
phrasing scoring.ts's own AI_SLOP_PHRASES list flags as a risk ("delve
into", "game-changer", ...) — would've been self-defeating for the
generator to produce exactly what the scorer penalizes. Generate prompt
also now asks for genuinely different angles per variant, not minor
rewordings of each other.
Also added a Copy button next to the character counter — writes the exact
current draft text to the clipboard with "Copied" feedback, so a finished
draft (typed, optimized, generated, or imported) can go straight to
pasting into X's own composer instead of manual text selection.
Verified in a real browser (Playwright, with clipboard permissions
granted): clipboard content matches the draft exactly, feedback toggles
correctly, and all previous typecheck/build passes still hold.
The toggle was plain text with no button chrome, easy to miss as interactive on mobile — now a full-width pill button matching the app's existing button style, with a chevron that flips to show expanded state. The idea textarea was rows=2 with a long example-laden placeholder that overlapped the Generate button on narrow viewports; bumped to rows=3 and trimmed the placeholder to a short prompt that fits without wrapping into the button below it. Verified visually at iPhone 13 viewport width via Playwright screenshots: toggle now reads clearly as tappable, textarea placeholder no longer overlaps or overflows its border.
Reported from production: the generator produced "...more views and engagement: https:// What do you think?" — a finished-looking post with a dead link in the middle. Reproduced it. The deterministic optimizer preserves real URLs correctly; the bug was upstream. The model's output was truncated mid-URL, parseVariants accepted the fragment as a complete variant, and the optimizer then appended its reply hook to the fragment, which is what made it read as finished. Three fixes, root cause outward: - Likely root cause: Gemini's flash-tier models spend part of maxOutputTokens on internal reasoning before emitting visible text, so a budget sized only for the visible post gets cut off mid-sentence. Added generous headroom on the Gemini path — unused budget is free (billing is on tokens actually produced). Ollama keeps its own hard cap; that path is CPU-time-bound and has its own timeout history. - parseVariants now drops any variant ending in a URL with no real dotted domain, so a truncated generation can never be surfaced as a finished post regardless of which provider or budget produced it. If every variant is dropped the route already returns generateStatus "empty", which the UI handles. - The generate prompt now forbids writing URLs outright, including placeholders. The model had invented a link to "the platform" it has no way to know — the same fabrication risk the existing "never invent specifics" rule covers, and this app already has a dedicated link field that gets scored separately for URL risk. Verified with unit tests over the exact production string plus bare http://, truncated mid-domain, and dangling www. — all dropped; full URLs, short URLs, URLs with paths, www URLs, mid-sentence URLs, and URLs with trailing punctuation all preserved; and a realistic multi-variant response keeps the two good variants while dropping only the truncated one.
The scorer has always been a heuristic proxy for Phoenix, not Phoenix — so
its central claim ("this draft will do better") went unverified, and the tool
was stateless, with nothing accumulating between sessions. This closes the
loop: score a draft, publish it, and the app pulls the post's real numbers and
grades its own prediction.
Flow: Track a draft -> publish on X -> Check for results. Syncing pulls the
handle's timeline, fuzzy-matches the draft against what was actually posted,
then fetches real views/engagement once the post has had time to accumulate
them. Once enough posts are measured it compares the higher-scoring half
against the lower-scoring half and reports which optimizer fixes correlated
with better numbers for that specific account.
Design decisions that matter more than the plumbing:
- Nothing is claimed below 6 measured posts; per-fix comparisons need 3+ on
each side. A "pattern" from three posts would be invented, not observed, and
this feature is worthless if its output can't be trusted.
- Medians, not averages, so one post that happens to go viral can't manufacture
a trend. The UI says outright that if the two halves look the same, the score
isn't predicting anything yet.
- Fixes are attributed only when the tracked text is exactly the optimizer's
output. Edit after optimizing and nothing is recorded, because a wrong
attribution corrupts the data this exists to produce.
- Matching is deliberately strict (0.72 Dice over character bigrams). A false
match would credit someone else's post's numbers to a draft; an unmatched
draft just stays pending.
- Tracking is strictly additive. With no DATABASE_URL every endpoint reports
enabled:false and the rest of the app is untouched.
- Sync is on-demand and idempotent — no cron or worker needed — and only spends
API calls on posts that are matched, old enough to measure, and stale.
Verified end to end rather than by inspection: fuzzy matching unit-tested
against t.co rewrites, pre-post edits, and near-miss same-topic posts; summary
gates unit-tested including the zero-baseline case that would otherwise
serialize Infinity to null through JSON; the DB layer exercised against a real
Postgres 16 (schema auto-creation, BIGINT coercion, per-handle scoping, the
partial unique index that stops two drafts claiming one tweet); sync
orchestration tested with injected fetchers covering match/no-match, the
too-new-to-measure gate, idempotent re-sync, and a failing metrics fetch not
aborting the run; and the UI driven in a real browser for the populated,
empty, and below-the-gate states.
Also fixes a packaging bug caught during that verification: pg had installed
into a stray package.json at the repo root, which resolved fine locally by
walking up to the root node_modules but would have failed the Railway build
with "Cannot find module 'pg'". It now lives in the app's own package.json.
The app had grown well past what its UI explains on its own — scoring, signal breakdown, risk flags, tips, the deterministic optimizer, AI rewrite, generate-from-idea, live import and tracking, each with its own conventions and gates. Users had no way to know what any of it actually meant. New /docs route covering every part of the tool: how the score is built and what the grade bands mean, what each input toggle actually changes and why (including the ones with outsized effects, like "This is a reply" costing all out-of-network reach), the real action weights, the visibility-filtering flags, what the optimizer does, how the AI layers are gated, and how the track record decides when it has enough data to say anything. Every number on the page is read from the implementation rather than written from memory, and the honest-limitation framing is kept front and centre — the page opens by explaining that the probabilities are a stand-in for Phoenix, not Phoenix. Also extracts a shared SiteHeader, which both pages now use instead of duplicating markup: - Below md, the nav collapses into a hamburger dropdown. On mobile the old header had a back button plus three social icons competing with the logo, which wrapped to two rows and crowded the title. - At md and up the links stay inline — there's room, so hiding them behind a menu would be worse, not cleaner. - The dropdown closes on link click, Escape, outside tap and route change, and carries aria-expanded/aria-haspopup/role=menu. Verified in a real browser at desktop and iPhone widths: hamburger open/close including Escape, outside-tap and post-navigation dismissal; the desktop inline nav and the Docs active state; every docs nav anchor resolving to a real section; the documented weights and thresholds matching the implementation; and zero horizontal overflow on mobile, including at the wide tables, which scroll inside their own container.
…reviews metadataBase was set to bescfinancial.com, the company site, but the OG and Twitter card image is a relative path to /besc-banner.png which lives in this app's public/ directory. Next resolves relative asset paths against metadataBase, so every shared link advertised a card image at bescfinancial.com/besc-banner.png — a URL that doesn't serve it. For a tool whose main distribution channel is being shared on X, that's a real cost. Points metadataBase at the app's own domain, and adds canonical URLs, og:url and og:site_name, plus dedicated OG metadata for /docs so it previews properly when linked directly rather than falling back to the root card. The domain is overridable via NEXT_PUBLIC_SITE_URL so preview deploys can advertise their own URL without a code change. bescfinancial.com is left alone where it belongs — the company link in the footer and social row. Verified against a real production build (dev mode uses the request origin, so this is only observable after next build): og:image and twitter:image now resolve to https://xalgo.beschyperchain.com/besc-banner.png on both routes, with correct per-page canonicals.
…ced "What do you think?" The complaint was that suggestions all end the same lazy way. The prompt was part of it, but the root cause was our own scorer rewarding exactly that. The bug: a generic closer scored +0.28 as a reply CTA *and* +0.18 for ending in a question mark = +0.46, while a genuinely specific question scored +0.18. The scorer preferred the laziest possible ending by 2.5x. Since the optimizer and the AI candidate gate both optimize for this scorer, the whole system was pulled toward "What do you think?" — and a better AI hook would have scored lower and been rejected by the gate. Fixing the incentive was a prerequisite for the AI improving anything, not a separate cleanup. - classifyReplyHook() now distinguishes a question carrying real content from a bolt-on closer, by stripping known closers and function words and checking what's left. Specific scores 0.40, generic 0.15, none 0. Verified: the same post scores 57.5 with a specific question vs 43.5 with "What do you think?", and a specific-hook rewrite now clears the AI gate it previously failed. - The deterministic optimizer no longer appends one identical string. It can't write a question about your specific post — only the AI can — so it picks a varied, content-anchored fallback (number-aware when the post has figures), stable per draft so multi-pass optimization stays idempotent. Beyond reading as filler, one tool appending the same tail to thousands of posts manufactures the templated-text pattern BBQDuplicateTextProd.bot looks for across accounts. Its reason text now says plainly that it's a fallback worth replacing. - New tip when a post has a generic closer, pointing at the specific-question upgrade rather than staying silent because a "reply CTA" was technically present. New lib/algorithmContext.ts builds the AI a real working brief from the same constants the scorer uses, so it can't drift: the full weight table and what the ratios mean in practice (one report ~= 468 likes; copy-link is 40x a like), the structural limits that cap reach before wording matters (reply/repost OON exclusion, the 0.75 discount, diversity decay, the 48h cutoff, the 10s video floor), the label risks that drop a post outright, and an explicit hook section with banned generic closers plus patterns that actually earn replies. Both prompts now also receive a per-post situation brief — media attached, reply or original, mutual-follow audience, how many posts already in this window — so suggestions are written for the actual situation instead of in the abstract. Tradeoff worth noting: the brief costs ~1,400 tokens of prefill. That's nothing for Gemini, which is the default path, but it is real CPU time on the Ollama fallback. Caught while testing: my first fallback set included a statement-form hook, which the multi-pass optimizer didn't recognise as a hook on the next pass and so appended a second one after it. All fallbacks are now questions, with a regression test asserting no draft ever gets two hooks and that every draft is idempotent under re-optimization. Docs updated to match — the optimizer fix table, a callout explaining why specific beats generic (including that this used to be backwards), and what the AI is actually given.
…uessing them
The scorer's per-action probabilities were hand-tuned heuristics — honest
guesses, labeled as such. This replaces them, where the data supports it, with
a model fitted to the author's real published outcomes.
The unlock is that we don't have to wait for tracked drafts to accumulate.
Views and per-action counts are both public on a published post, so
replies/views is a directly measured action rate. "Learn from my history"
walks the timeline (paginated, original posts only) and turns an existing
account into hundreds of real (post -> outcome) pairs in one call.
Ridge regression per action over the features the scorer already extracts,
then blended into the same Sigma(weight x P(action)) the real RankingScorer
uses. Same shape as Phoenix — predict per-action probabilities, combine with
production weights — learned at the level of "posts like this" rather than per
viewer, which is the version actually reachable: Phoenix takes a viewer plus
that viewer's private engagement history, an unpublished draft has no viewer,
and no API exposes that history. No trained weights ship either; phoenix/
ships training code and synthetic data generators.
Three guards, because a confidently wrong personalised score is worse than the
honest heuristic it replaces:
- Nothing fits below 40 posts.
- Every action must clear cross-validated R2 (5-fold) before it's used at all.
Verified: a fit on data with a real relationship reaches CV R2 0.93-0.98 and
recovers the correct coefficient signs, while pure noise produces no usable
model and falls back silently.
- The fit is shrunk toward the heuristic by n/(n+120), so 60 posts nudges the
score and 500 largely drives it.
Design correction found by testing rather than reasoning: my first version
substituted fitted rates as absolute probabilities, which dropped every
calibrated score ~13 points. Real reply rates are ~1% of views while the priors
sit near 0.4 — the priors were tuned to spread the 0-100 range, not to be
literal probabilities. So calibration now contributes relative structure
("~1.8x your typical reply rate") applied to the existing scale. It reorders
drafts instead of rebasing the number, which is what the score is actually for.
Also threads the model through the optimizer, AI rewrite and generate paths.
Without that the displayed score would be calibrated while the gate deciding
which AI candidates surface was still scoring on heuristics — two different
rulers on the same screen.
Share-family actions (copy-link, DM, generic share) are never exposed publicly
and can't be learned directly, so they ride the author's learned bookmark
ratio, which is the closest observable proxy for "worth saving or sending".
Verified end to end against a real Postgres: schema ALTER applies to an
existing table, the model round-trips through JSONB, an unknown handle scores
byte-identically to the heuristic path, and a fitted model correctly learns
that one author's specific-hook advantage is larger than the generic prior
assumed (20.3 vs 14.1 point gap) while keeping the score scale stable.
The UI still sold the original product — "draft on the left, watch the score
move on the right" — while the tool had grown generation, AI rewriting, live
import, tracking and per-account calibration. Worst of it: calibration had no
presence in the interface at all, so the score could silently change meaning
with nothing telling the user why.
- New CalibrationBadge sits with the score and says plainly which kind of
estimate it is: a generic one built on the real weights, or one fitted to
this author's own results with the fitted/heuristic split shown as a
percentage and a bar. That property matters more than any other number on
screen and was previously invisible.
- Hero rewritten for the current product, with capability chips so generation,
AI rewriting and calibration are discoverable rather than buried in a
collapsed panel. The empty state now points at the idea generator and live
import instead of only "start typing".
- The composer had grown to three screens: media, link, four toggles, handle,
follower count and post count all stacked under the textarea. Everything
secondary now collapses behind a "Post context & your account" disclosure
that summarises anything set away from its default ("· reply, @handle"), so
nothing affecting the score can hide while collapsed.
That collapse initially broke the calibration flow — a regression the
front-end tests caught. The badge tells a new user to add their handle, but
the handle field had just moved behind the disclosure. The track panel now
takes a handle directly, which is where someone is standing when they want to
calibrate anyway, so the flow no longer depends on finding a collapsed field
in another component.
Verified in a real browser at desktop and mobile widths: hero and chips
render, context collapses and re-expands with every control intact, the
collapsed summary reveals non-default settings, calibration status shows for
an uncalibrated score, entering a handle in the track panel switches it into
its active state and surfaces the calibration button, and there's no
horizontal overflow on mobile.
Production error on the first real "Learn from my history" run: "there is no unique or exclusion constraint matching the ON CONFLICT specification". tracked_posts_tweet_idx is a PARTIAL unique index — WHERE tweet_id IS NOT NULL, so the many unpublished drafts sitting on NULL don't collide with each other. Postgres will not infer a partial index for ON CONFLICT unless the statement repeats the index predicate, so the insert failed outright and took the whole backfill with it. Adding "WHERE tweet_id IS NOT NULL" to the conflict target fixes it. Why the tests missed it: the integration test seeded rows with its own direct INSERTs and only exercised refit/load/score, so backfill's actual insert statement never ran against a real database. backfillFromTimeline now takes an injectable page fetcher — the same pattern syncTrackedPosts already used — so the insert and pagination logic can be driven end to end without live network calls. Now covered against real Postgres: the insert succeeds against the partial index, a repeat insert of the same tweet is a silent no-op rather than an error, two NULL tweet_ids still coexist, backfill skips replies and zero-view posts, and a second backfill run inserts nothing and does not throw — which is the exact failure that shipped.
A real 212-post backfill surfaced problems that synthetic fixtures never would have. 1. Every post recorded 0 likes. A missing field is indistinguishable from a real zero, so the miss was silent — and it trained the calibration model on a metric that was always 0. Widened the candidate paths per metric (camelCase, British spelling, nested shapes), and added a diagnostic endpoint that reports the numeric fields a timeline entry actually carries next to what extraction produced. Guessing at field names from the outside is how this slipped through; the fix is being able to see the real response. 2. The calibration split compared "score ~0" against "score ~0". Backfilled rows stored predicted_score = 0, so the panel asking "does the score predict your reach?" was splitting a column of identical zeroes and reporting the resulting noise. Backfill now scores each post as the tool would have before publishing, which turns that panel into a genuine backtest against outcomes the scorer never saw. 3. A model explaining 6% of variance was driving 64% of a real user's score. Trust was weighted on sample size alone, so lots of data behind a barely predictive fit read as high confidence. It now scales by cross-validated R² as well: the same 212-post fit at R² 0.06 now carries 13% instead of 64%, while a genuinely predictive fit still earns real influence. calibrationStrength reports that same blended number, so the UI can't claim more confidence than the scorer actually applies. Backfill also now refreshes on re-run rather than skipping. Engagement keeps accruing after a post is first seen, and rows written by the older version still carry predicted_score = 0 — without this, existing data could only be repaired by wiping the table. A manually tracked draft's predicted_score is explicitly preserved, since that's the real pre-publish prediction and the whole point of the comparison; only its metrics refresh. Verified against real Postgres: backfilled posts get real predicted scores, a re-run refreshes metrics instead of erroring or skipping, a manual prediction survives a backfill that touches the same tweet, and weak fits no longer dominate the score.
…eld names Confirmed from the app itself: likes render correctly on imported posts, which read from tweet-info. So extractMetrics is fine — timeline entries just carry engagement in a different shape, and a field we can't find is indistinguishable from a real zero. That's how 212 backfilled posts all recorded 0 likes while replies and reposts came through. Rather than keep guessing key spellings from the outside, backfill now treats "zero likes alongside non-zero replies/reposts/bookmarks" as a failed read and re-fetches that post from tweet-info, the endpoint already known to return likes correctly. Runs at bounded concurrency, since a repair pass can mean a few hundred calls. Guards that matter as much as the fix: - A genuinely quiet post (zero engagement everywhere) is not mistaken for a failed read and costs no extra call. - Healthy timeline metrics are used as-is, so the repair pass is free when nothing is wrong. - If tweet-info returns the same broken shape, the original values are kept rather than overwritten, and nothing claims to have been repaired. - A failing tweet-info never aborts the run. The reported counts now separate inserted / refreshed / repaired, so the next real backfill says plainly how much data was actually salvaged. Two test bugs found and fixed along the way, both mine rather than the code's: fixtures reused tweet ids across handles (tweet_id is globally unique, so those inserts were correctly becoming updates), and a debug script left rows behind that collided with a later run.
A 212-post backfill of a real account made these obvious: half the history scored exactly 0.0, including posts with 600+ views and 30 likes, while a two-word status update scored 35. 1. t.co was in the shortener blocklist. X rewrites EVERY link posted to it into a t.co shortlink, so this penalised every post containing any link at all — the opposite of a spam signal. The URL-verdict rules being modelled are about the reputation of the destination domain, and X plainly doesn't distrust its own wrapper. Removed; a t.co link now reads as "destination unknowable", which is neutral. 2. The negative action probabilities were wildly inflated. report could reach 3%, and paired with its -234 weight that alone was -7 points; notInterested added -9.9. Real report rates are minuscule — that's precisely why the real model can afford such a large weight. Pairing huge weights with invented probabilities meant a normal product update lost ~10 points to imagined abuse and floored at 0. Rescaled to plausible magnitudes: ordering is unchanged, but a clean post now loses ~0.1 instead of ~10. 3. Absence of problems was being rewarded as if it were substance. "Almost done" tripped no penalties, so it outscored informative posts. Copy-link share is the heaviest action in the model and requires something concrete worth sending someone; share-family and quote probabilities now scale with how much post there actually is. Checked against the real timeline that exposed this. Nothing is pinned at 0 any more: an informative post with a specific question scores 66.7, a normal update with a link 48.6 (was 0.0), content-free 20.7, while ALL-CAPS bursts (4.3), engagement bait (5.8) and hashtag-stuffing with a genuine shortener (4.6) all stay High Risk. Worth stating plainly: this makes the score sane, not yet predictive. On that account the fitted models still land at R² 0.06-0.11, because its engagement rate is fairly stable across posts while total reach varies for reasons the text doesn't explain. The quality-weighted shrinkage already keeps weak fits near the heuristics, and the calibration panel keeps reporting the flat comparison honestly rather than dressing it up.
…ooter Two things the data and the layout were both asking for. Timing analysis, because of what a real 212-post history actually showed: the fitted content models topped out at R² 0.06-0.11 since that account's engagement *rate* barely moves between posts (5-10% likes per view almost regardless of wording), while total views swing from 47 to 1,400. Reach is being driven by distribution, not phrasing. Author-diversity decay and the 48-hour eligibility window are both cited mechanics that operate on timing, so that's where the remaining explainable variance plausibly lives. Buckets median views by time of day, day of week, and gap since the previous post, and surfaces the single strongest well-supported pattern. Same discipline as the rest of the calibration work: nothing below 30 measured posts, no bucket reported under 6 samples, medians not means, and a lift under 1.25x is treated as ordinary variation rather than a finding. When nothing stands out it says so outright — "for your account, when you post isn't the lever" is a real answer, not a failure. Hour buckets use the browser's timezone, since "post in the morning" is only actionable where the author actually lives. Labelled as correlation, with the obvious confound called out: announcements at 9am and idle notes at midnight will read as a time pattern when it's really content. Layout: the track record was at the very bottom of the page, below the full signal breakdown, so the calibration state and insights — the most valuable output the tool has — were the last thing anyone would ever see. It now sits directly under the score, right where the calibration badge points at it, and the 100-post list that made it bulky is collapsed behind a toggle. The signal was buried under its own raw data. Verified end to end against real Postgres: a seeded history with a genuine morning advantage is found at 2.97x through the full stack, flat data yields no claim, thin buckets are dropped, the timezone offset changes the answer, the panel now renders above the tips panel instead of the footer, the post list is collapsed by default, and there's no mobile overflow.
…tcomes Groundwork for judging any new probability estimator, and useful on its own. Context: a proposed plan suggested running the open-source Phoenix ranking path against drafts. Checked the repo — phoenix/README.md states twice that "there is no checkpoint or corpus bundle to fetch" and there's no run_pipeline.py, so training your own is the only path, and it would learn the synthetic generator's rules rather than human behaviour. Combined with the ranking head needing a specific viewer's engagement history (a draft has no viewer), that route produces confident numbers with no connection to real reach. The salvageable idea is the other one: estimate P(action) some better way and feed the same weighted scorer. But that's only worth doing if the new estimator can be shown to beat the current one. So: lib/backtest.ts scores an author's entire measured history and reports Spearman rank correlation between predicted score and actual outcomes. Rank correlation rather than R² because the score's job is ordering drafts, not forecasting view counts — a badly calibrated scorer that ranks correctly is still useful, and rank correlation shrugs off the heavy tail one viral post creates. compareEstimators() runs two approaches head to head on the same posts, so a proposed replacement has to actually win. The verdict now sits at the top of the track record, above everything else. If the score isn't predicting an author's outcomes, that changes how much weight the panels below it deserve, and putting it anywhere else would be self-serving. It distinguishes "not predicting" from "consistently backwards", which is a more actionable finding — it means the scorer is rewarding something that author's audience actively dislikes. Thresholds are deliberately unambitious: social engagement has enormous irreducible variance, so r=0.3 is treated as a genuinely useful signal rather than a weak one, and nothing is claimed below 25 measured posts. Verified against real Postgres end to end: a seeded history where score truly drives views reports "predictive" (r=+1.00), an unrelated one reports "not predicting" (r=-0.09), a backwards one is caught as inverted rather than noise, constant predictions report no signal instead of r=0, and ties are handled with averaged ranks.
Gives the scorer a second source for its action probabilities: a model that has been handed the real weight table, the visibility-filtering rules and the structural limits, and asked how a draft compares to a typical post from the same account. It answers in multipliers rather than probabilities. Language models are good at "this will get roughly twice the replies of a typical post" and bad at "P(reply) = 0.031" — absolute rates depend on follower count, timing and audience, none of which are in the text. Since the 0-100 score is a squashed sum, a wrong scale would shift every grade, whereas multipliers slot into the exact mechanism the fitted regression already uses. The multipliers are damped by how much of the score already comes from a model fitted to real outcomes. Both estimate the same thing from overlapping evidence, so stacking them at full strength would double-count, and where measured outcomes exist they are the better evidence. The larger point is the trial, not the head. Anyone can attach an LLM and emit confident numbers; nobody can tell from reading them whether they predict anything. /api/track/evaluate scores the author's published posts twice — heuristic probabilities and AI probabilities, same posts, same weights — and reports which ordered the real results better. A tie goes to the incumbent, and the draft-level panel says out loud when the estimator is untested or has lost. Fixes a bug that trial exposed: compareEstimators scored a null correlation as -1, so a challenger ranking outcomes *backwards* beat a baseline that had simply found no signal. Backwards is worse than nothing. Null now scores 0, a challenger must clear the weak-signal threshold, must not be inverted, and must win by more than noise. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A9oF1GYEbb9zvnk4nRzBmh
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.