Case Study: Building an Automated Competitive Intelligence Engine, From Signal Detection to Weekly Strategic Briefings

As a Director of Product, I need to know what competitors are shipping, whether they're building things my product doesn't have, and whether anything strategically important happened this week. Doing that manually means hours spent reading changelogs, release notes, and documentation pages across dozens of competitor sites, and most of it is noise.

So I designed and built Product Intelligence OS: a self-hosted automation system running on n8n that watches competitor websites daily, extracts what actually matters, compares it against a structured map of my own product's capabilities, and delivers a five-minute weekly briefing. No manual research required.

This wasn't a weekend script. It's a 9-workflow production pipeline with its own data layer, its own hallucination-prevention system, and six rounds of hardening driven by real operational issues after it went live.

The Challenge

Competitive intelligence, done properly, is tedious and constant. It means checking dozens of changelog pages, developer docs, release notes, and GitHub repos every day, most of which haven't changed, and separating the handful of genuinely important updates from routine noise and marketing language.

Doing this by hand doesn't scale, and doing it with AI introduces a different problem: language models are confident even when they're wrong. An AI system asked to compare a competitor's roadmap against your own product can just as easily hallucinate a capability gap as it can correctly identify a real one, especially when the evidence is a vague blog post or a GitHub repo with some sample code in it.

The real challenge wasn't "can AI read competitor websites." It was building a system disciplined enough that its conclusions could actually be trusted.

What I set out to build

The system needed to run unattended, on a schedule, without turning into another dashboard nobody checks. From the start, I treated it as a full intelligence pipeline, not a single automation: source monitoring, evidence extraction, internal capability mapping, gap analysis, and a digestible weekly output, each as its own accountable stage.

Just as important as what it found was how it reasoned. Every finding needed to separate observed fact from AI interpretation, every gap claim needed to be checked against what my own product actually does, and anything uncertain needed to be flagged for a human, not stated as settled fact.

Cost and reliability mattered too. The system had to run on real budget (well under $50/month) and had to fail loudly rather than silently, since a competitive intelligence system nobody trusts is worse than no system at all.

My Role: Product Leader & Builder

Defined the problem, the intelligence model, and the 6 strategic lanes used to classify every finding
Designed the 9-workflow architecture and the full data schema end to end
Built, configured, and prompt-engineered every workflow personally, from source monitoring through newsletter generation
Designed the 4-layer hallucination-prevention system that keeps AI-generated findings trustworthy
Iterated through 6 production-hardening stages driven by real operational issues after launch
Operates and maintains the system in production today

Designing an AI system disciplined enough to trust

The intelligence model: lanes, tiers, and scores

Before writing a single workflow, I designed the classification system everything else would run on. Every finding gets assigned to one of six strategic lanes (identity experience, developer platform, AI agent identity, UI architecture, design workflows, strategic signals) so intelligence gets organized into buckets that actually map to product decisions, not a flat, undifferentiated feed.

Internal capabilities are tracked on a 5-tier maturity scale, from production_mature (shipped, versioned, in release notes) down to sample_only (exists only as a code sample). This distinction turned out to matter enormously: a competitor being "documented" while my product is only "sample_only" for the same capability is a very different finding than both being production-mature, and the system's language has to reflect that difference precisely.

Every finding also carries two independent scores: an importance score (how much this matters to product strategy) and a confidence score (how certain the evidence actually is). Gap claims and competitive recommendations require confidence of 4 or higher out of 5. Nothing gets stated as a confirmed gap on weak evidence.

The automation pipeline: 9 workflows, 2 models, 1 schedule

The system runs on n8n, self-hosted, orchestrated by a single Master Orchestrator that fires every weekday morning and calls each sub-workflow in sequence. Market Watch checks roughly 114 competitor URLs daily, fingerprinting each page and only flagging real content changes. A relevance gate scores every change before it's allowed anywhere near an AI model, and low-signal pages (marketing homepages, generic blog posts) are dropped immediately, which keeps both cost and noise down.

Changes that pass the gate go to Claude Haiku, the fast, cheap model, for classification and summarization. Once a week, a separate Internal Product Specialist workflow reads roughly 120 of my own product's sources and rebuilds a tiered capability map from scratch. Competitor Intelligence then takes fresh evidence and compares it against that map using Claude Sonnet, the more capable model, reserved for the judgment calls that actually need it.

Every Friday, a Newsletter workflow reads the week's structured findings and writes a five-minute briefing: what's genuinely new this week, what's still being watched, and what (if anything) needs action. It's emailed directly and published to Notion as an archived page. A Source Discovery workflow runs monthly, crawling known documentation hubs to find new sources to monitor on its own, so the system's coverage grows without manual upkeep.

Two workflows exist purely for reliability: an Error Handler that catches failures anywhere in the system and pushes an instant phone alert for anything critical, and a Credential Health Check that runs an hour before everything else to catch expired API connections before they silently break a run.

Engineering for trust: preventing AI hallucination

This is the part of the build I care about most, because it's the difference between a system people actually use and one they quietly stop trusting after the first wrong conclusion.

Every prompt carries explicit rules: marketing language like "AI-powered" or "seamless" is never treated as evidence; code sitting in a GitHub repo is not a shipped feature; when evidence is weak, the model is instructed to abstain rather than guess. Every finding separates observed fact (only what the source explicitly states) from analyst interpretation (what it might mean) as two distinct fields, never merged into one confident-sounding sentence.

Behind the AI sits a second, non-negotiable layer: deterministic parser rules that override the model when needed. A finding sourced from a sample app is hard-capped at the lowest maturity tier, regardless of what Claude concludes. A GitHub repo claiming production-grade maturity gets checked for real release history before that claim is allowed to stand. When the parser overrides the model, the record is marked so the override is always visible, never silent.

Anything that's uncertain but potentially important gets flagged for human review rather than presented as settled. The system is designed to make a wrong "I don't know" far more likely than a confident, wrong answer.

Outcome and impact

The system has been in continuous production use since early 2026, evolving through six hardening stages as real operational issues surfaced: a newsletter that was recycling old findings got rebuilt around a strict 7-day window with a novelty check; critical alerts moved off Gmail's expiring OAuth tokens onto a channel that doesn't expire; a timezone misconfiguration that was silently firing the schedule at 2:30 AM instead of 9:00 AM got caught and fixed.

What used to take hours of manual research every week now arrives as a five-minute Friday briefing, generated and delivered without anyone visiting a single competitor website. The system watches its own health, alerts on failure before a human would notice, and grows its own source coverage monthly, all for roughly $12–19 a month in AI API costs.

More than the automation itself, the project validated an approach: AI agents can be trusted with judgment-heavy work when the system around them, not just the prompt, is engineered to catch and contain their mistakes.

✓ Replaced hours of manual competitive research each week with a five-minute Friday briefing
✓ Built a 9-workflow automation pipeline monitoring roughly 234 combined external and internal sources
✓ Designed a 4-layer hallucination-prevention system: prompt rules, deterministic parser caps, a relevance gate, and human review flags
✓ Delivered a self-healing system with automatic error alerting and pre-run credential health checks
✓ Iterated through 6 production-hardening stages driven entirely by real operational data, not hypotheticals
✓ Held ongoing cost to roughly $12–19/month by filtering low-signal content before it ever reaches the AI