A company using the onboarding AI Nimbia says its first-week activation rate rose from 49% to 70%. It’s an eye-catching result, but the 1.4× figure should be read as a hypothesis to test, not a reason to buy.
49% to 70% is not a “21% increase”
First, let’s read the numbers correctly. Nimbia is an onboarding agent that converses with new software users in real time by voice, sees their screen, clicks buttons, and fills in fields. It says it learns from product documentation, recorded onboarding calls, and test accounts, then adds a script snippet to the app.
On Product Hunt, Nimbia says its first customer ran an A/B test against its existing onboarding and achieved results 1.4× higher for first-week activation and trial-to-paid conversion. The figures for WeMind, a customer published on Nimbia’s website, are slightly more specific: first-week activation rose from 49% to 70%, while paid conversion became 1.4× higher.
The wording matters here. Going from 49% to 70% is a 21-percentage-point increase; calculated as relative lift, it is about 42.9%, or roughly 1.43×. Saying “activation increased by 21%” understates the actual change, while saying “it increased by 70%” means something entirely different.
What changed isn’t the chatbot—it’s the execution distance to first value
Nimbia’s core advantage is not writing better explanations. It is reducing the actions users must take themselves before experiencing initial value. A typical chatbot might answer, “Open Settings and click the integration button,” while Nimbia says it continues the voice conversation, navigates to that page, and fills in the fields.
This difference can grow in complex B2B SaaS. In products where value appears only after connecting data sources, creating a project, and generating a first output, there is a long gap between reading help and actually completing the work. Conversely, in simple products where value is apparent with one click after signup, an AI call may become an unnecessary step.
| Approach | Help users receive | Best fit | Risk to check |
|---|---|---|---|
| Checklists · tooltips | Guidance on a defined sequence and location | Short, predictable flows | UI changes and exception paths |
| Support chatbot | Text answers to questions | When users can explain their issue | Doesn’t know the current screen or whether the task was truly completed |
| Human onboarding call | Contextual explanation and demonstration | High-value, high-complexity contracts | Scaling is constrained by schedules and staffing |
| Screen-operating AI | Completes real tasks together through conversation | Self-serve trials with long setup steps | Mistakes, personal data, user discomfort |
Your experiment hypothesis should therefore be specific, not simply “adding AI increases conversion.” For example: “When new trial users who need five steps to create their first report receive screen-operating guidance, the share that creates a first report within seven days increases.” Amplitude also recommends defining the behavior where users first experience value before improving activation, then validating with data whether that behavior relates to long-term retention.
Why you should not make a buying decision on 1.4× alone
The public results omit experimental information needed to make a decision. They do not disclose the control and treatment sample sizes, how users were assigned, experiment duration, original paid-conversion rate, or statistical uncertainty. So outsiders cannot determine whether 1.4× is an effect beyond chance or a result seen only in a particular customer segment.
In particular, don’t confuse voluntary participation with random assignment.
If only people who clicked a help button and joined an AI call are grouped as treatment and compared with everyone else, the result may simply select users who were already more motivated. Randomly assign all eligible users at signup, and assess results by their originally assigned group regardless of whether they actually joined a call.
It is also important to determine the required sample size and minimum detectable effect before the experiment. LaunchDarkly explains that larger samples increase confidence in results and recommends calculating sample size and duration before starting when using a fixed-duration approach. Adobe likewise warns that stopping early because an advantage briefly appears in limited data makes it more likely that chance will be mistaken for a winner.
You also need to confirm that random assignment worked properly. A sample-ratio mismatch in which the proportions of users in the two groups diverge unexpectedly, assignment errors that mix users and workspaces, or missing events can invalidate the entire experiment. LaunchDarkly’s experiment checks likewise make alignment among the exposure unit, sample ratio, and metric analysis unit a core condition.
Finally, activation can rise while long-term outcomes worsen. If the AI completes the first task for users, the activation event is recorded, but users may be unable to repeat the task independently the following week. That’s why you should review downstream and safety metrics together, including paid conversion, 30-day retention, refunds, support requests, and AI misoperations.
Four steps to validate it in your product
- Lock the activation contract into one sentence first.
Specify the audience, action, and deadline, such as “new workspaces that connect a real data source and create their first report within seven days of signup.” Exclude events distant from value, such as signup completion or finishing a tour, and first use existing cohorts to check whether this action is associated with 30-day retention. - Validate the boundaries of what the AI can do in a test account.
Try not only the happy path, but also empty data, insufficient permissions, slow screens, and users’ mid-flow questions. Prohibit payments, deletion, and external sending, or require human approval for them. Confirm with the vendor how sensitive fields are masked, how calls are disclosed and consented to, log-retention periods, an immediate stop button, and handoff to human support. NIST recommends assessing data privacy and information-security risks in third-party generative AI integrations and documenting pre-deployment testing. - Randomly assign newly eligible cohorts at the workspace level.
For B2B products, the workspace level is safer than the account level so multiple users at one company do not see different versions. Give the control group the current onboarding, and the treatment group the current flow plus an entry point to AI guidance. Apply acquisition channel, plan, country, and device conditions equally to both. - Do not change definitions until results mature.
Set one primary metric: the seven-day activation rate, and make paid conversion and 30-day retention downstream metrics. Record AI call start and completion rates as diagnostic metrics, and misoperations, stops, human handoffs, and support tickets as safety metrics. After the trial and payment-observation periods end, review the absolute difference, relative lift, sample size, and uncertainty interval together before deciding whether to adopt it.
The conclusion is simple. The Nimbia case suggests that AI which understands screens and takes action may reduce the execution distance in complex onboarding. But the public 1.4× figure alone cannot establish the certainty or reproducibility of that effect. Before buying the product, first align internally on the activation definition and experiment design.
If you want to dig deeper
Nimbia: AI screen-sharing calls for user onboarding — This is the page where you can first verify the product introduction and 1.4× A/B-test claim. producthunt.com
Nimbia onboards your users in live screen-sharing calls — You can verify the 49%→70% activation figure and the product’s voice and screen-operating approach. nimbia.ai
Getting Started: Driving Product Engagement by Obsessing Over Activation — Explains how to define product-specific activation behaviors and validate them with real behavioral data. amplitude.com
Experiment sample size and run time — Official documentation to consult when determining the sample size and runtime needed for an A/B test. launchdarkly.com
How long should you run an A/B test? — Covers the problems that arise when an experiment is ended early simply because the results look favorable. experienceleague.adobe.com
Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile — NIST material for reviewing third-party AI adoption, privacy, and pre-deployment evaluation. nist.gov
.png)
