A product manager can now build an agent that emails customers who dropped out of a trial, answers their replies within minutes, and checks its own answers against company documentation before any automated reply goes out. The build takes roughly 30 to 40 minutes in Claude Code, Anthropic's command-line coding agent, plus a markdown file of rules and an email connection. A former chief product officer at Bolt who now runs a conversational AI startup laid out this exact build step by step. The process doubles as a concrete test of a bigger claim he makes: that one PM with three to five engineers and AI tooling can match the output that once required 20 to 25 engineers.

Why a PM would want thousands of conversations

At Bolt, talking to 7 or 8 customers a month counted as a strong research cadence for a product leader. Automated two-way conversations change the scale entirely: an agent can run back-and-forth interviews with tens of thousands of users each month, resolve concerns on the spot, and hand synthesized themes back to the product team on a schedule.

The clearest numbers come from an A/B test his startup ran with Ancestry.com on shoppers who abandoned a DNA kit checkout. Customers were split into an outreach group of 86,400 and a no-contact holdout of 85,900.

Group Customers Conversions Conversion rate
AI outreach 86,400 15,898 18.4%
No-contact holdout 85,900 11,511 13.4%

That works out to a 37.3 percent relative lift (p < 0.001) and about 4,300 incremental orders in this single case. The agent also tagged the reasons people hesitated across 3,470 conversations, averaging 1.5 themes each:

Objection theme Share of conversations
Privacy and unexpected DNA discoveries 42.1%
Gifting versus personal-use timing 38.6%
Subscription and hidden-fee concerns 31.2%
Skepticism about scientific accuracy 24.5%
Competitive alternatives 19.8%

The more useful cut is conversion by objection. Shoppers who asked about shipping and delivery timing converted at 41.3 percent once the agent answered, against a 31.5 percent benchmark for all users who replied. Shoppers confused about the subscription converted at only 19.8 percent, and those anxious about unexpected relatives at 14.2 percent. The interpretation offered is that objections a fact can fix, such as a holiday shipping cutoff, convert well through conversation, while objections that persist despite a good answer point to pricing or UX problems that belong on the roadmap.

The check-in loop, piece by piece

The prototype is called a check-in loop, and it has five parts. The worked example is Cadence, a fictional AI fitness coaching app with a 14-day trial, where outreach fires on day 12 for users who signed up but barely used the app.

Whiteboard flow of spreadsheets feeding a rules document, outgoing email, reply loop and a Slack-style alert

▲ The customer check-in loop architecture

  1. Customer data. A CSV export works to start; a warehouse such as Snowflake or a customer data platform such as Segment works at scale. The key fields are the trial drop-off timestamp and usage history, which the agent uses to personalize each message.
  2. A rules file. A markdown file named CLAUDE.md sets message structure, tone, escalation paths and the limits of the agent's authority. For Cadence that means rules on workout customization, an injury policy, refund handling and a cap on discounts. If a company has no such documentation, Claude can be pointed at the public website to draft a first version.
  3. An outreach skill. The script finds dropouts who have not been contacted, writes a short email using each person's metadata, and sends it. Subject lines in the sample run read like a human wrote them, for example "How's HIIT Express feeling so far?" and "4 days left on your Cadence trial, Riley."
  4. Email transport. Sending runs through Model Context Protocol, or MCP, a standard that lets an AI agent call outside tools. A Gmail MCP server is enough for early tests and small cohorts; Gmail's daily sending quotas mean larger volumes need a transactional service such as Postmark, which also has an MCP server, or coordination with the marketing team. Either way, a PM can run the experiment without touching enterprise survey platforms such as Medallia or Qualtrics.
  5. Reply handling and alerts. When a user answers, the agent classifies the message as a normal reply or an escalation, responds within minutes, and posts to Slack when something needs a human. In one sample thread, a trial user raised both workout intensity and budget concerns, and the agent handled both inside the rules it was given.

The first test passes exposed ordinary bugs: replies that did not attach to the original Gmail thread, and the occasional inappropriate phrase. Iterating on those is part of the 30-to-40-minute estimate.

Outreach rules that actually get replies

The mechanics matter less than how the emails read and when they arrive.

  • Plain text, one question. Emails should look like one-to-one correspondence, not a campaign, and open with a single direct question tied to where the person is in the funnel. Cramming a survey into the first message is the most common mistake.
  • Answer fast, but not instantly. Delays injected into automated replies caused reply rates to fall sharply, and conversion fell with them, because people leave their inbox and the momentum breaks. At the same time, a reply within seconds feels robotic. A few minutes is the sweet spot.
  • Use email for people who left. In-app chat suits active users, but abandoned users are by definition not in the app, and research prompts during a live session can distract people from finishing the task. Well-targeted research emails typically draw 20 to 40 percent response rates; a fitness company's trial-abandonment test drew about 35 percent. Response rises with funnel depth: people who used the product reply far more than top-of-funnel dropouts.
  • Target lifecycle milestones. The recommended touchpoints are onboarding drop-off (lowest reply rate, highest volume, reveals activation barriers), a mid-trial check-in, trial end without conversion, users who explored an upsell without buying, lapsed users whose activity fell over recent weeks, and subscription cancellation.
  • Pair the trial-end email with an incentive. Offering something like 20 percent off a subscription in exchange for feedback lifts replies from undecided prospects and gives the agent a chance to resolve the objection and recover the sale in the same thread.

The threads also become a reusable dataset. A PM can later ask Claude to scan past conversations for everyone who complained about a missing capability and invite them to test the feature once it ships.

Adding an eval loop so the agent can reply unattended

An eval, short for evaluation, is a check written into the pipeline that scores the model's output against explicit criteria and runs every time. For a customer-facing agent the most important eval is groundedness: does every claim in the reply trace back to approved documentation such as the help center and the rules file?

Envelopes pass a magnifying-glass inspection gate; one is diverted to be rewritten and filed as a draft under a bell

▲ Groundedness eval with retry and draft fallback

The loop was added to the Cadence pipeline with a single prompt to Claude Code, and the resulting design is simple enough to copy:

  1. Insert the eval after classification and before sending. The check runs only on replies configured for auto-send mode. Escalations already go to a human.
  2. Judge groundedness against the system documentation. A model acts as the judge, comparing the draft reply with the source files rather than its own sense of plausibility.
  3. On failure, regenerate once with the reason attached. The eval's explanation of what went wrong is fed back into the generation prompt as context. In the large majority of cases the second attempt passes.
  4. If the retry also fails, save a Gmail draft instead of sending. A human reviews the draft; nothing ungrounded reaches the customer.
  5. Alert the team. Claude Code added a Slack notification with a failure marker for every eval failure without being asked. Agentic coding tools often anticipate adjacent needs this way, which is useful but also a reason to read every diff.
Groundedness eval loop before an auto-sent reply goes out yes no yes no Customer reply arrives Classify as normal replyor escalation Generate reply Groundedness eval onauto-send replies only Grounded? Send Post Slack alert, regenerateonce with failure reason Retry passes? Save as Gmail draftfor human review
▲ Groundedness eval loop before an auto-sent reply goes out

Two design points stand out. First, email makes this loop possible. Chat users expect responses in a few hundred milliseconds and will not tolerate a three-minute wait, while a three-minute email reply is completely normal, so the asynchronous channel leaves room for multi-step evaluation and regeneration. Second, the eval should guide, not constrain. Overfitting prompts to a rigid rubric produces stilted messages; treating a failure as context for a full regeneration preserves natural reasoning while fixing the specific defect.

The startup's enterprise platform ships platform-wide guardrails that double as a checklist for what else to evaluate: AI disclosure and transparency, conciseness under 250 words, clean email formatting, grounding in verified context, and bans on clichés, embedded images, unapproved coupons, empty link parameters, serial numbers and unsolicited advice. Each draft is checked against these and against custom domain rules, and is rerun until it complies. Before going live, batches of simulated customer conversations let operators review outputs and turn their feedback into playbook and guardrail updates. On the enterprise platform, a new engagement point takes 5 to 10 minutes to configure once a team knows the workflow, and large customers run hundreds of them.

Testing a probabilistic system

The fundamentals of product management do not change here, but the testing paradigm does. Traditional software has deterministic tests that assert a given input produces a given output. A conversational agent faces the nth permutation of user intent, so teams need LLM-as-a-judge frameworks capable of judging subjective properties such as groundedness and intent adherence, combined with deterministic rules for structured parts of a response.

Several practices follow:

  • Keep a trusted sample dataset outside production and require any model upgrade or prompt change to pass the guardrail benchmarks at parity before deployment. Even bumping the underlying model version can shift behavior in user-facing ways, and teams iterating hundreds of times a week cannot check this by hand.
  • Hold customer-facing agents to a much higher bar than internal copilots. Internal tools have an employee reviewing outputs before anything is published; a customer-facing agent has no such net.
  • Inspect intermediate traces, not just final text. Language models are inclined to please the user, so the eval must confirm that every discrete action was explicitly permitted by the system prompt and business rules. The oft-cited case of a car dealer chatbot agreeing to sell a truck for $1 is the failure mode to design against.
  • Study the successes too. When the agent converts a customer in a scenario where conversions almost never happen, dissect the trace and fold the approach back into prompts and guidelines.

Does one PM really equal 25 engineers?

At Uber the typical ratio was one PM to three or four engineers; at his most productive stretch, before teams grew, the same leader ran one PM to 25 engineers and found that the ratio burned away process and forced everyone to focus on shipping. His claim is that a pod of one PM and three to five engineers using AI tools now delivers what 25 engineers once did.

The build above supports a narrower version of that claim. One person working alone in Claude Code stood up data ingestion, personalized outreach, reply handling, an eval and retry loop, draft routing and Slack alerting in under an hour. Arguably, that scope of work would previously have been a multi-person project. The claim about team ratios itself comes from one leader's experience rather than a measured benchmark, and it carries conditions he spells out: PMs must switch context quickly, take on adjacent work such as design and product marketing, and hold the product together so that faster shipping does not produce a pile of disconnected features. They also have to decline alignment meetings that exist only as busywork and let AI draft the status reports, planning briefs and meeting prep that used to fill their calendars.

His own four-person company shows the operating model. The team spends most of its human time debating problem spaces and mapping customer journeys on a whiteboard or in FigJam. Meeting transcripts and diagrams feed custom AI skills that draft product requirements documents, break them into Linear tickets, and hand the tickets to coding agents for first implementations. Deciding what to build, not building it, is the bottleneck. In a large company with mandated PRD templates and multi-department review, the same approach still works: brainstorm at high bandwidth, then have AI map the transcript into the required format for sign-off.

When not to automate

The strongest caveat is about timing. Before product-market fit, or at low customer volume, automated outreach is a mistake. Early discovery depends on video calls and direct relationships that build the intuition a team needs, and delegating it to an agent isolates the PM from exactly the signal that matters. Automation belongs after a product has shown baseline value and scale demands systematic engagement.

Even then, automation sits alongside human contact rather than replacing it. The suggested rhythm is at least two direct customer conversations a week over Zoom, Slack or text, supported by three AI-driven pillars: synthetic avatar conversations, automated email touchpoints like the loop above, and automated synthesis of sales calls and social feedback. Automated voice interviews through platforms such as Listen Labs can add scale for companies with large customer bases, though they are overkill for very early-stage startups that still need direct founder contact. Email agents work well as a screening funnel, surfacing the respondents with the most compelling feedback for a deeper human interview, and the background research means the PM arrives at that interview already past the basic questions.

What to do with this

The practical path is short. Export a CSV of trial dropouts with drop-off dates and usage. Write a CLAUDE.md that sets tone, escalation triggers, refund and discount limits, or have Claude draft one from your website. Connect a Gmail MCP server for a small cohort and send plain-text emails with one question each. Before letting the agent reply on its own, add a groundedness eval that regenerates once with the failure reason and otherwise saves a draft for human review, and read every diff the coding agent produces. Watch which objections still fail to convert after a good answer, because those are roadmap items, not messaging problems. And if you are still searching for product-market fit, keep making the calls yourself.