Product Management
AI Automation
Competitive Intelligence
Data Systems
Product Intelligence OS Case Study
From fragmented market signals to evidence-based weekly product briefings.
Competitive intelligence rarely fails because teams lack information. It fails because useful signals are scattered across release notes, documentation, developer repositories, product announcements, pricing pages, and changelogs. Most updates are routine. A small number may affect product direction, investment decisions, or competitive positioning.
Reviewing those sources manually takes time and produces inconsistent results. Automating the work creates a different problem. AI can summarize information quickly, but it can also turn weak evidence into confident conclusions, mistake sample code for a shipped capability, or present an old observation as something new.
Product Intelligence OS was created to address both problems. It is a self-hosted intelligence system that monitors market activity, detects meaningful changes, compares external evidence with an internal capability model, and produces a concise weekly product briefing.
The system combines scheduled automation, AI-assisted analysis, deterministic rules, structured data, and human review. It was built as a working operational product, not simply as a prototype or isolated AI experiment.
Key Metrics
9
Connected workflows
128
n8n nodes
234
External and internal sources configured
$12–19
Approximate monthly AI cost
How It Works, At a Glance
Nine workflows, one orchestrator
A single Master Orchestrator runs on a weekly schedule and calls each sub-workflow in sequence. Every workflow can also be triggered and tested independently, and any crash is caught by a dedicated Error Handler rather than breaking the chain.
- Credential health check — tests critical Google connections before the main run begins.
- Master orchestrator — determines the day of the week and calls the appropriate workflows.
- Market watch — retrieves monitored pages, fingerprints changes, and calls the AI model on qualified updates.
- Internal product specialist — rebuilds the internal capability map used for gap and parity analysis.
- Competitor intelligence — compares new evidence against the internal capability map by strategic lane.
- Newsletter — suppresses repeated findings, builds the weekly briefing, and delivers it by email.
- Memory and audit — flags risky or low-confidence conclusions for review.
- Source discovery — finds and qualifies new sources for the registry monthly.
- Error handler — classifies failures, logs them, and alerts on anything critical.
Keeping the Model Honest
Six rules included verbatim in every AI prompt
- Marketing language ("seamless," "AI-powered") is not evidence of a capability — ignore it.
- GitHub code ≠ a shipped feature — only README usage instructions, changelog entries, or official docs count.
- Abstain by default — if evidence is weak, return
not_enough_evidence: true. - Observed fact and analyst interpretation are always kept in separate fields.
- No gap claims without reading the internal capability map first.
- Gap or requirement claims require confidence ≥ 4.
See what actually broke in production, and how it was fixed, in six stages of improvement under real conditions below.
The Product Intelligence Problem
Turning information into decisions
Product teams have access to more competitive information than ever, but the information is fragmented and difficult to evaluate consistently.
A meaningful capability may appear in a release note, a documentation update, or the changelog of a developer repository. A marketing page may use similar language without proving that anything has actually shipped. Even when a change is real, understanding its importance requires comparing it with existing product capabilities and the strength of the available evidence.
Manual monitoring does not scale well. It depends on someone remembering which sources to check, recognizing what changed, and separating meaningful developments from routine content updates. Important signals can be missed, while low-value information consumes time and attention.
A traditional dashboard would not solve the complete problem. It could collect links and changes, but it would still leave the reader responsible for interpreting every update.
The product opportunity was broader: create a system that could continuously monitor the market, preserve the supporting evidence, evaluate what changed, compare it with an internal capability model, and deliver only the intelligence that deserved attention.
Defining the Product Direction
Designing for decisions, not information volume
The product was designed backward from a practical question: What would a product leader need to understand at the end of the week to make a better decision?
The answer was not another feed of competitor announcements. The useful output needed to explain what changed, why it mattered, how reliable the evidence was, how it compared with existing capabilities, and whether any action was justified.
That led to several product principles.
Signal over volume
The system should remove routine noise before spending time or AI resources analyzing it.
Evidence before interpretation
Every conclusion should remain connected to the source material that supports it.
Visible confidence
The strength of the evidence should be presented separately from the strategic importance of the finding.
Human judgment at the right moments
The system should automate routine monitoring and synthesis while escalating uncertain or consequential conclusions for review.
Delivery in the flow of work
The final intelligence should arrive as a concise weekly briefing, not wait inside another dashboard that someone has to remember to visit.
Operational reliability
The system should monitor its own health, expose failures, and make problems visible before they silently affect the output.
My Role Across the Product Lifecycle
Product leader and hands-on builder
I led the work from initial problem framing through product design, architecture, implementation, launch, and continued improvement.
- Framing the product opportunity and defining the decisions the system needed to support
- Designing the weekly intelligence briefing and working backward into the required data, evidence, and workflows
- Creating the strategic classification model, maturity levels, scoring system, and source taxonomy
- Designing the nine-workflow architecture and the supporting data model
- Building and connecting 128 n8n nodes, including 46 custom Code nodes
- Writing more than 1,100 lines of JavaScript for filtering, transformation, validation, classification, parsing, and error handling
- Designing the AI prompts, model responsibilities, deterministic safeguards, and human-review rules
- Configuring the data layer, API integrations, service credentials, delivery channels, and operating schedules
- Testing the system with real sources and improving it through six development and operational-hardening stages
- Operating and maintaining the working system after launch
Building the Product End to End
From the intended outcome down to individual nodes
Designing the outcome first
Before building the automation, I designed the final experience: a styled weekly newsletter that could be read in approximately five minutes.
The briefing was structured around three questions:
- What is genuinely new this week?
- What remains important but has not materially changed?
- Is there anything that deserves validation, deeper investigation, or product action?
This decision shaped the rest of the product. If the final briefing needed to explain evidence, confidence, competitive implications, and recommended actions, those fields had to exist throughout the data model and intelligence pipeline.
The newsletter includes a specific weekly date range, clearly separated sections, source links, system-health information, and repeat-suppression rules. The same briefing is delivered by email and archived in Notion for future reference.
Creating the intelligence model
A useful intelligence system needs more than summaries. It needs a consistent way to classify information and compare evidence.
I created six strategic lanes for the initial implementation:
- Identity Experience
- Developer Platform
- AI Agent Identity
- UI Architecture
- Design Workflows
- Strategic Signals
These lanes organize findings around areas that can influence product direction. They prevent the system from producing an undifferentiated stream of updates and make it possible to compare related capabilities.
Each finding receives two independent scores. The importance score measures how relevant the development may be to product strategy. The confidence score measures how strongly the available source supports the conclusion.
Separating the two matters. A development can be strategically important but supported by weak evidence. Another may be unquestionably real but have little effect on product direction. The system should not treat those situations as equivalent.
Modeling capability maturity
Competitive comparison becomes unreliable when every capability is treated as either present or absent.
A product may have a working sample, partial documentation, a preview implementation, or a mature production release. Those are different levels of evidence and should lead to different conclusions.
I created a five-level maturity model:
Production mature
Supported by release history, release notes, versioning, or a general-availability announcement.
Documented
Confirmed through official documentation, but without enough evidence to establish production maturity.
Partial
Some support exists, but availability, coverage, or completeness remains limited.
Unclear
Evidence exists, but the maturity level cannot be determined reliably.
Sample only
The capability appears in sample code or a demonstration but is not confirmed as a shipped product capability.
The system uses these levels when comparing competitor evidence with the internal capability map. A documented competitor capability compared with an internal sample should not be described the same way as two mature production capabilities.
This model allowed the system to use more precise language such as “possible gap,” “requires validation,” “public evidence suggests parity,” or “no gap detected.”
Designing the data layer
The system needed structured memory across daily and weekly runs. Google Sheets was selected as a lightweight and transparent data layer that could support the first version without introducing unnecessary infrastructure.
The complete data model includes functional tables for:
- Competitor configuration
- Page fingerprints and change snapshots
- Raw evidence
- Structured findings
- Internal capability maturity
- Source discovery and classification
- Workflow execution history
- Error records
- Audit and review activity
Notion provides the presentation and archive layer. Competitor findings are stored as structured cards, while completed weekly briefings are published into a searchable newsletter archive.
Service credentials are managed through n8n. Google Sheets and Gmail use OAuth, Notion uses an integration credential, and AI requests use protected API credentials. Secrets are kept out of the workflow logic.
Translating the product into nine workflows
The product was separated into nine workflows so each stage could have a clear responsibility, input, output, and failure path.
Credential health check
Runs before the primary workflow and tests critical Google connections. If a credential is expired or unavailable, the system produces an alert before the main run begins.
Master orchestrator
Starts every weekday, creates the run context, determines the day of the week, calls the appropriate workflows, and records the beginning and completion of each run.
Market watch
Reads the competitor configuration, retrieves monitored pages, calculates content fingerprints, identifies changes, filters low-value updates, calls the appropriate AI model, and writes qualified evidence into the data layer.
Internal product specialist
Reads active internal product sources on a priority-based schedule and rebuilds the capability map using the defined maturity model. This creates the internal evidence required for responsible gap and parity analysis.
Competitor intelligence
Takes each new evidence item, identifies the relevant strategic lane, retrieves the corresponding internal capability information, and produces a structured comparison.
Newsletter
Reads the week’s findings, checks whether each conclusion is genuinely new, suppresses repeated observations, generates the briefing, formats it as a styled HTML email, publishes it to Notion, and delivers it through Gmail.
Memory and audit
Reviews the week’s structured findings, identifies risky or low-confidence conclusions, and writes an audit summary for future review.
Source discovery
Runs monthly, reads known documentation hubs, extracts candidate links, classifies them using rules and AI, removes duplicates, and adds qualified sources to the registry for review.
Error handler
Receives workflow failures, creates a structured error record, assigns severity, logs the failure, and sends a push notification when immediate attention is required.
Building the system node by node
Each workflow was built as a connected sequence of small, accountable operations. Across the complete system, this resulted in 128 nodes and 128 workflow connections.
The implementation includes:
- Schedule and workflow triggers
- Google Sheets reads, writes, updates, and data clearing
- API requests for source retrieval, AI analysis, and Notion publishing
- Page fingerprinting and change detection
- Conditional branches and relevance gates
- Content cleanup and transformation
- Prompt construction
- Structured JSON parsing
- Source classification and deduplication
- Evidence and confidence validation
- Model-output safety caps
- HTML newsletter formatting
- Workflow health checks
- Error logging and push alerts
Forty-six of the nodes contain custom JavaScript. These nodes handle the product-specific logic that standard integrations could not provide, including URL normalization, page hashing, relevance scoring, source-role enforcement, prompt assembly, maturity calibration, novelty detection, evidence parsing, and failure classification.
This was an important part of the build. The product intelligence model did not live only inside an AI prompt. It was implemented throughout the workflows, data structures, code, and decision paths.
Using Different AI Models for Different Jobs
Matching model capability to the task
Not every task requires the same level of AI reasoning.
The system uses a faster, lower-cost model for high-volume classification tasks such as relevance evaluation and source discovery. A more capable model is reserved for work that requires deeper judgment, including capability extraction, competitive comparison, and newsletter generation.
This division improves both cost and quality. Routine filtering does not consume expensive model capacity, while strategically important conclusions receive more careful analysis.
The current system operates for approximately $12 to $19 per month in AI API costs. Low-signal content is filtered before it reaches the model, which keeps the cost manageable as the source registry grows.
Designing Trust Into the System
Four layers that keep conclusions trustworthy
Trust was treated as a product requirement, not a prompt-engineering detail. The system uses four complementary layers of control.
Layer 1: Explicit evidence rules
Every relevant prompt defines what the model can and cannot treat as evidence. Marketing language such as “AI-powered” or “seamless” does not prove a capability exists. Code in a repository does not automatically prove that a feature has shipped. Weak or ambiguous material should produce an abstention rather than a confident conclusion.
Layer 2: Relevance filtering
Changed pages are scored before deeper analysis. Low-value pages, generic marketing content, and unrelated changes are removed early. This reduces noise, lowers operating cost, and limits the amount of weak material presented to the AI.
Layer 3: Deterministic safety rules
Custom parser logic reviews the model’s structured output and applies non-negotiable caps. For example, sample applications cannot be classified as mature production capabilities. Repository content cannot support a production claim without evidence such as release history. Short or vague evidence cannot receive a high confidence score. When a deterministic rule overrides an AI conclusion, the record is marked so the intervention remains visible.
Layer 4: Human review
High-impact findings with uncertainty are flagged for review rather than presented as settled conclusions. The system deliberately prefers a cautious “requires validation” over an unsupported competitive claim. Human judgment remains responsible for consequential product decisions.
Observed Facts and Interpretation Remain Separate
Keeping evidence and analysis in distinct fields
Every finding contains distinct fields for observed facts and analyst interpretation.
Observed facts are limited to what the source explicitly supports.
Analyst interpretation explains what the evidence may mean for product direction, competitive positioning, or future investigation.
Keeping these fields separate makes it easier to challenge a conclusion without losing the original evidence. It also reduces the risk that an interpretation will later be repeated as an established fact.
From Daily Monitoring to a Weekly Briefing
The operating rhythm
The product follows a defined operating rhythm.
Every weekday, the system checks critical connections and runs Market Watch across the configured competitor sources.
On Mondays, the Internal Product Specialist refreshes the internal capability map using active sources and their assigned priority cadence.
On Tuesdays and Thursdays, Competitor Intelligence processes new evidence and produces structured comparisons.
On Fridays, the Newsletter workflow creates and delivers the weekly briefing. Memory and Audit reviews the findings and records anything that may require additional attention.
Once a month, Source Discovery examines known documentation hubs and identifies potential new sources. This allows coverage to grow without maintaining every URL manually.
Launch, Learning, and Operational Hardening
Six stages of improvement under real conditions
The first working version was not treated as the end of the project. Running the system under real conditions exposed problems that were difficult to predict during initial design. The product evolved through six stages.
Schema strengthening
The internal capability schema expanded from a simple capability list into a more precise evidence model. New fields captured source strength, support maturity, capability tags, freshness, and the reason behind each maturity decision.
Source-weighted extraction
Different source types received different evidence rules. Release notes, documentation, repositories, sample applications, and press announcements could no longer support the same level of conclusion.
Deterministic parser caps were added to prevent weak sources from being promoted into mature capability claims.
Maturity-tiered comparison
The competitive analysis moved from a flat capability comparison to a five-tier maturity model.
This allowed the system to distinguish a likely gap from a possible gap, a validation requirement, or evidence of parity.
Market-watch relevance gate
The initial workflow sent too many changed pages into AI analysis.
A deterministic relevance gate was introduced to score content using source type, page title, body content, and product-specific keywords. Low-signal changes are now removed before an AI request is made.
Source discovery
The first internal source list was manually maintained. It was replaced with a structured source registry and a monthly discovery workflow.
The discovery process combines deterministic URL rules with AI-assisted classification. New sources enter the registry as candidates, preserving human control over what becomes an active intelligence source.
Production hardening
Live operation exposed several reliability and quality issues.
The newsletter was initially recycling older findings. It was rebuilt around a strict seven-day window, novelty evaluation, and a clear distinction between new intelligence and ongoing observations.
Critical failure notifications were moved to a push-alert channel that did not depend on the same email connection it was expected to monitor.
A timezone configuration problem caused scheduled workflows to run at the wrong hour. The runtime configuration was corrected and the deployment was updated to restart reliably.
Additional protections were added for AI API errors, unexpected response structures, empty results, and workflow-output failures.
Outcome and Impact
A fragmented research process becomes a weekly operating rhythm
Product Intelligence OS now turns a fragmented and repetitive research process into a structured weekly operating rhythm.
The system is configured around approximately 114 external competitor sources and 120 internal product sources. It monitors changes, removes routine noise, extracts evidence, compares capability maturity, and produces a briefing that can be reviewed in approximately five minutes.
Routine monitoring and synthesis run automatically. Human attention is reserved for findings that are uncertain, strategically important, or strong enough to justify deeper investigation.
The project also demonstrated an important principle for AI product development: trustworthy output depends on the complete system surrounding the model.
Prompts matter, but so do the data model, source hierarchy, deterministic rules, confidence thresholds, failure handling, human-review paths, and continued learning after launch.
Evidence
- ✓ Designed and built nine connected product intelligence workflows
- ✓ Created 128 n8n nodes, including 46 custom Code nodes
- ✓ Wrote more than 1,100 lines of JavaScript for product-specific workflow logic
- ✓ Configured monitoring across approximately 234 external and internal sources
- ✓ Created six strategic intelligence lanes and five capability-maturity levels
- ✓ Designed four complementary layers of AI trust and evidence control
- ✓ Delivered a styled five-minute weekly briefing through email and Notion
- ✓ Evolved the system through six development and operational-hardening stages
- ✓ Added structured error logging, credential checks, and critical failure alerts
- ✓ Maintained an approximate monthly AI operating cost of $12 to $19