Activity

What & why

DevTrends is being refocused as a private research/content archive (a free public MCP is a later newsletter hook) — not a SaaS. This PR guts everything that isn't that: the prediction bet that failed its experiment, the commodity feed tools, the dashboard UI, and the dead/vanity/low-signal sources. What's left is clean scraping of high-signal sources + the few lookup tools that actually work, made trustworthy.

Approved plan, executed as 5 atomic, individually-green commits. main is untouched — this is for review.

⚠️ Deploy safety (read before merging)

This branch contains a migration that drops 4 tables (divergences, divergence_topics, predictions, domain_reputations) and runs on deploy via AUTO_MIGRATE. pg_dump those 4 tables on prod before merging/deploying. The source-deactivation migration is safe (only flips is_active=false; raw_items history untouched).

Commits

1. Remove the web UI — routes/controllers/app/Reports/app/Actions/Dashboard (the last also retires the Action↔tool query duplication). Kept /health, /scheduler-health, a bare JSON / (the named home route Fortify needs). Frontend left orphaned.
2. Strip MCP to one authed server, 5 tools — search, compare, neighborhood, stack_radar, security_digest. Deleted the 9 feed/prediction/raw-SQL tools + the interactive "app" resource/prompt. Public MCP removed.
3. Drop the prediction/divergence feature entirely — models, actions, commands + schedules, seeder, factories, tests, and the drop-tables migration. Decoupled two kept features (comment-harvest, content-extraction) that only used predictions as an optional booster.
4. Strip dead/vanity/low-signal sources + fetchers — vanity repos, Twitter, LinkedIn, YouTube, gh_trending, Google Trends, Adzuna, Bluesky, Mastodon (deactivated, history kept; removed from seeder; 8 fetcher classes deleted).
5. Fix tool matching — stack_radar + security_digest built unanchored term1|term2 regexes (react matched reactive); both now use the existing SearchPattern::wordBoundary() helper. compare now returns matched-title receipts so counts are self-verifying. Regression tests: evals≠medieval, arize≠summarize, proven end-to-end via a new CompareToolTest.

Kept

HN, Reddit, Lobsters, dev blogs, gh_rel_* releases, package registries, GHSA/NVD/CISA, arXiv/HF, remoteok; the comments pipeline (used by search/stack_radar); auth/Fortify.

Out of scope (flagged, not done here)

• REST API (/api/v1/*, ~20 endpoints) — left intact; separate strip decision. Only its /diagnostics/predictions* endpoints were removed (they referenced deleted code).
• Telescope — already filters to failures-only in prod; fully disabling is a Coolify TELESCOPE_ENABLED=false toggle, not a code change (would also kill local dev).
• Frontend teardown, public/newsletter MCP, entity-dictionary/derived layer.

Verification

• ✅ 137 tests pass on Postgres CI; pint clean.
• ✅ route:list: only MCP + health + settings/auth + (kept) API routes; stripped routes gone; /mcp/public gone.
• ⏳ Post-deploy: live-call the 5 MCP tools to confirm the word-boundary fix (arize no longer returns summarize-pollution) + sample receipts; scheduler:health stays HEALTHY with stripped sources inactive.