The call sounded natural, but the appointment was scheduled incorrectly
Just because an AI phone agent speaks clearly and responds appropriately does not mean it completed the work successfully. It may record a customer’s name incorrectly, confirm an unsupported schedule change, or finish a request that should have been handed to a person on its own. For outbound calls, failing to identify the calling organization and purpose or missing a do-not-call request are also failures. Conversation quality and task-completion quality are different.
The first thing to examine in ThunderPhone is the structure for handling failures, not the 2¢ per-minute price. Before deployment, you can simulate an entire conversation with an AI caller, then save a specific failed turn found in a limited live call as a Validation example and replay it against a revised draft.
However, a Validation failure does not automatically block deployment. You can review the results and deploy anyway, and the dashboard’s Success Rate is also based on AI scoring. Ultimately, the operator needs a process that collects failures, has humans judge task outcomes, and reruns the same situation after changes.
Carry the process from signup through checking the first simulation
Your first task should be one with an unambiguous outcome, such as “confirm webinar attendance.” Handing over an entire sales flow from the start makes it hard to identify which instruction caused a problem. The inputs and decision criteria below are editing examples for starting a quality loop, not official templates in which ThunderPhone guarantees performance.
- Sign up for or log in to the ThunderPhone app and open an agent draft. Put the role, calling organization/contact number and purpose, values that must be confirmed, prohibited actions, do-not-call handling, and a concrete human-handoff procedure in the prompt. ThunderPhone’s prompting guide advises specifying a hand-off path the agent can actually perform, such as a transfer number or callback promise. Click Talk and allow browser microphone access to confirm the current draft responds. Before simulating, assign a callback owner and a review owner, and prepare the follow-up request log below. Do not Deploy yet. Talk and Simulation operate on the current draft.
- Prepare a payment method and funds for the first simulation. Register a payment method under Organization → Billing Settings, then add enough organization balance to cover the estimate shown on the first Simulation approval screen. Simulation charges for both the AI caller and the target agent are deducted from the organization balance. To test live calls through a phone number as well, you need the minimum US$3 prepaid balance described in the official quick start. Then select an engine, language, and voice in Config, and record the all-in per-minute price displayed on screen.
- Open Simulation in the agent builder. For the first validation, choose direct agent connection instead of a phone number. Enter normal cases along with exception cases that include a request for a human representative, an unsupported change, and refusal of further contact. Before running it, review and approve the displayed estimated cost based on a maximum of 10 minutes. Both the AI caller and the target agent incur usage charges.
- Review the completed simulation in Call History. Look at the transcript, recording, automated evaluation, and tool calls together. Separately determine whether the operator disclosed the organization name, contact number, and call purpose; accurately captured the name and attendance status; avoided confirming prohibited changes; confirmed a callback number and deadline for human requests; assigned an owner in the follow-up request log; and apologized and ended without further persuasion when asked not to contact again.
- Deploy only drafts that meet internal required criteria. Promote a draft to the production version only if it passes caller identification, required-value accuracy, prohibited-action prevention, the human-handoff process, and the do-not-call termination process. If any one fails, revise the prompt and rerun the same Simulation first.
Example first input
Agent prompt: “You are the webinar attendance confirmation representative for [organization name]. At the beginning of the call, first disclose the organization name, contact number, and purpose by saying, ‘Hello, I’m calling from [organization name] to confirm whether you will attend [webinar name]. Our contact number is [organization contact number].’ Confirm only the registrant’s name and attendance status. Do not promise payment, discounts, or changes to personal information. If the person says they do not want to be contacted again, apologize, use the connected do-not-call logging tool to record that contact as do not call if one exists, and end the call immediately. Even if there is no logging tool, end without further questions or persuasion. If records and answers conflict, or the person asks to speak to a human, first read back and confirm the number for a callback and say, ‘A representative will call you back at the confirmed contact number by the next business day.’ If they then decline further contact, cancel the callback promise and prioritize the do-not-call process. Do not change attendance status or time after that, and end the conversation.”
[organization name], [organization contact number], [webinar name], and “the next business day” are editing examples for internal testing. Before testing, replace every bracketed item with the actual organization name, a reachable organization phone number, and the call purpose, and set a callback deadline the assigned owner can meet. ThunderPhone’s US outbound guidance includes both caller identification and provision of a contact number. To test an actual call transfer, specify a test destination number in E.164 format that your organization controls in the prompt, and first confirm that an external call may occur during simulation.
Exception case: “After correcting the spelling of their name twice in Korean from a noisy place, the registrant requests an unsupported change to attendance time and a human representative, then finally says not to call again.” Check whether the organization and purpose were disclosed first, whether the agent avoided guessing the name or confirming the change, and whether it ended without further persuasion after the do-not-call request. If a do-not-call logging tool is connected, its tool call is also a required decision item.
Two cents is not the price of a completed call
Official base engine pricing is 2¢ for Spark, 5¢ for Bolt, and 9¢ for Storm. It is prepaid pay-as-you-go, but enabling Supervision adds 8¢ per minute for Spark and Storm. This is not a service where a human listens; it is an option in which a second AI checks instruction compliance on every turn and quietly intervenes when necessary. The final price can vary with voice/language routing, long prompts, Storm add-ons, numbers, and telephony costs.
| Option | Base price | Task to compare first |
|---|---|---|
| Spark | 2¢/min | Short, standardized flows such as attendance confirmation |
| Bolt | 5¢/min | Flows needing a balance of general response quality and speed |
| Storm | 9¢/min | Flows that must handle long instructions and many exceptions |
| Supervision | +8¢/min | Spark/Storm flows requiring second-AI turn-by-turn checks |
A simulation is not a free duplicate call either. The AI caller is priced at the Spark rate, and the target agent is calculated according to its actual configuration. If both use Spark and there are no add-ons or telephony charges, the simple sum of engine rates is 4¢ per minute, but that is not a quote guaranteeing the actual bill.
Compare pricing as the cost per successfully completed task.
Add call and simulation costs, repeat calls, human handoffs or callbacks, and the cost of handling incorrect appointments and missed do-not-call requests, then divide by the number of actually completed appointments or verified leads. A lower per-minute rate can still produce a higher total cost if failures and repeat calls increase.
Limited live calls require number verification and Campaign setup
Passing a browser simulation does not mean the real telephone path has been verified. ThunderPhone’s US demo numbers are for inbound testing only, so for a customer-facing outbound test, connect a number you own through a VoIP provider and verify it in both directions.
- Connect a provider under Connections → VoIP. Enter credentials for a supported provider such as Twilio or Telnyx and complete Connect & Test; for a manual SIP connection, pass Test & Save Connection.
- Import an owned number through Phone Numbers → Add number → Connect VoIP provider. Confirm it is available for both inbound and outbound use after round-trip verification, including inbound confirmation and outbound-permission verification. If verification fails, correct the provider credentials and SIP configuration first.
- If applicable to your organization, complete the administrator’s compliance-responsibility confirmation. This is a one-time process required before the first campaign or API call, but it does not itself verify actual recipient consent, caller identification, disclosures, or do-not-call requirements.
- Open Engagement → Campaigns → New campaign. Select the deployed agent and verified VoIP caller number, then upload a contact CSV with a
phoneorphone_numbercolumn. For the first test, include only a small number of contacts whose call purpose and consent have been confirmed and who are not on the internal do-not-call list. - Set the owner and completion criteria for do-not-call records first.
The following is an editorially proposed operating procedure. Before calls, designate a Call History reviewer and an actual callback owner, and create fields in a shared sheet or CRM for call ID, confirmed callback number, request time, owner, deadline, status, and completion time. After each call, the reviewer reads the transcript and logs and assigns callback cases; the callback owner responds by the deadline and records the actual connection result and time. If no connection is made, do not close it as complete; leave it in retry status. If an automatic logging tool is connected, verify actual task creation and storage of the owner and deadline.
A request not to be contacted again takes priority over a callback. When the request is made, cancel pending callbacks, add the number to the internal exclusion list, and confirm it is absent from the next campaign CSV. Saying “we will call you back” is only intake; record it separately from actual callback completion or confirmation that the request was withdrawn. - Set calling hours and retry conditions. The product supports 1–10 concurrent calls, with a default of 3, but as an editing example we recommend starting at 1 concurrent call. This is not an official performance guarantee; it is a conservative starting point for immediately reviewing each failure. Align your operating criteria so do-not-call requests and explicit refusals are not retried, approve the estimated cost, and click Start.
- Confirm success in both Call History and the internal exclusion list. Calls made from the verified number must appear as completed, with transcripts, recordings, and tool calls available for review. The telephone test is not complete until you verify disclosure of organization name, contact number, and purpose; actual completion of promised callbacks or withdrawal of requests in the request log; and immediate termination of do-not-call requests plus exclusion from future call targets through a logging tool or designated operator.
The product’s confirmation screen is not a substitute for legal review.
Before uploading contacts, confirm the consent, caller-identification, AI/recording disclosure, personal-data handling, and do-not-call standards that apply to the call purpose and receiving region. If the target country and call type have not been determined, one set of applicable rules cannot be stated conclusively.
Turn one failed live-call turn into a regression test
The practical value of Validation sets is that they let you freeze and rerun a problematic moment from a real non-simulated call. When you select a particular agent turn, you can bring in the conversation context up to that point, related caller audio, and the actual response; a human can then revise the expected behavior and allowed tool calls and save it as an Active example.
Running an example against the revised current draft shows Pass, Warn, Fail, Skipped, and Error statuses and a pass rate, and also reveals regressions where an item that passed in the prior run now fails. However, this is not a check that redials the entire call from the beginning. It does not reproduce the real telephone network, changed CRM state, or every misunderstanding accumulated in a long conversation exactly as it occurred.
- Find the turn in a limited live call that changed the task outcome. Choose a concrete failure, such as an opening that omitted the calling organization or purpose, a misheard name, a prohibited promise, an incorrect tool call, further persuasion after a do-not-call request, or a response that ignored a human request or departed from the promised callback procedure.
- Save that turn as a Validation example. Have an operator specify the expected behavior and allowed responses/tool calls, then make it Active. For a do-not-call case, list an apology, immediate termination, and invocation of the connected logging tool as decision items; in an operation without a tool, separately verify manual list updates after Call History review.
- Check the example status again after revising the prompt. If the prompt change made the example Stale, review whether the expected behavior and allowed responses/tool calls are still correct, then switch it back to Active. Only Active examples are included in a full Validation run, so skipping this check may leave the failure case you just saved out of the run.
- Run a Validation run on the current draft. Check not only that the new failure has changed to Pass, but also that caller identification, existing do-not-call handling, and other successful cases have not regressed to Fail. You can replay an individual example against a draft first, but it must be Active to be included in the overall pass rate and regression check.
- Apply your internal deployment-stop criteria. For example, you can halt deployment if even one required case fails or a prior Pass regresses. This is a standard your organization sets, not a product default. In ThunderPhone, you can intentionally deploy even after reviewing failures.
Compare automated success rates with actual task outcomes
Observability’s Success Rate is based on AI scoring, and infrastructure errors are excluded from the denominator. Detailed metrics during high-volume periods may also be calculated from a sample of recent calls. Therefore, do not read this number as if it were your CRM appointment-completion rate or lead-verification rate.
Alongside the automated score, record operator-confirmed caller identification, name and attendance status, actual CRM updates, callback-process compliance, do-not-call list updates, repeat calls, and incorrect tool executions. For calls where automated evaluation and human judgment diverge, examine whether the evaluation criteria are ambiguous before changing only the agent prompt.
The smallest complete unit of ThunderPhone evaluation is not deploying one agent. It must extend through simulating a draft, running a limited campaign with a verified owned number, excluding do-not-call requests from future call targets, and saving discovered failed turns as Validation examples. If you revise the prompt, review whether examples are Stale and reassess expected behavior, switch them to Active, and ensure both required cases and prior successful cases meet all internal criteria in a full Validation run on the revised draft before moving to the next deployment. The per-minute price is only one cost in that cycle.
If you want to dig deeper
Prompting guide - ThunderPhone — Learn about specific hand-off paths the agent can perform, such as an actual transfer number or callback promise, and outbound prompt composition. thunderphone.com
Outbound calling compliance — Review caller identification, do-not-call handling, and the scope of responsibilities to check before your first outbound run. thunderphone.com
Validation sets: replay real call turns before deployment — Review how to replay a specific failed turn from a real call against a draft, Active and Stale states, regression indicators, and the ability to bypass deployment checks. thunderphone.com
Outbound campaigns — The official process for starting a campaign by connecting a deployed agent, verified caller number, contact CSV, and calling settings. thunderphone.com
.png)
