Role: Lead Product Designer
Tools: Figma, FigJam, AI-assisted synthesis & prototyping
Team: Design, Product, Engineering
Timeline: End of June – Mid Jul 2026 (GA)
Simulations: Testing Voice Agents Before They Talk to Customers
Vapi is a voice AI platform for developers — companies use it to build and run the AI agents that answer and make phone calls for support, scheduling, and sales. Simulations is Vapi's end-to-end testing surface: teams define scenarios and personalities, run their assistants (or squads of them) against simulated callers, and evaluate the results. The positioning we aligned on with customers: evals are unit tests; simulations are end-to-end tests.
Responsibilities
I'm the lead designer on Simulations, owned the dashboard experience end to end. I ran the design/eng/PM working sessions, drove the information architecture and creation-flow decisions, and scoped the final GA spec with engineering.
PROBLEM STATEMENT
The problem
Customers building production voice agents need reliable, fast, realistic ways to test agent behavior before and after changes, because a voice agent that fails does it out loud, on a live call, with a real customer.
The alpha had significant reliability and usability gaps. As a result, our most sophisticated customers were building their own testing infrastructure outside Vapi rather than relying on Simulations.
Alpha Release
Current alpha experience
Challenges
01
Trust is the product
Testing tool that produces false positives. Reliability issues: mocking, tester realism, determinism, were the core design problem, because every unreliable run taught users to ignore results.
02
Two audiences, one surface
Product managers needed a simple, reliable way to run tests, while the existing alpha users (developer-centric) tend to test primarily through the API and never touching the dashboard. We had to make the PM path primary without ripping out the workflow power users already depended on.
03
Redesigning mid-flight
The project was a classic fixing the plane while in it scenario, with the team pivoting several times during engineering work as the team worked up against a hard GA date.
Success metrics
Adoption
Reduced reliance on external and custom testing workarounds built by customers.
Task efficiency
Scenario creation in 1–2 clicks, decoupled from the generative flow.
Personas
Based on our team’s user research, there are two key personas we are focused on: Product Manager (primary persona for GA, limited technical depth, frequent in-product), and the developer (alpha cohort — advanced, often API-first users).
Product Manager
Responsibilities:
Define what "working" means for the agent; run tests before and after changes; share results with the team
Motivations:
Confidence that the agent behaves before it reaches customers
Pain Points:
Creation flow required detouring through a separate generative tool; unclear errors; results hard to parse; no way to tell which part of a simulation failed
GOals:
A simple, reliable way to define a test, run it, and read the verdict
Developer / Power User
Responsibilities:
Regression-test agents at scale, often dozens of simulations concurrently; debug failures down to the specific iteration
Motivations:
Deterministic results they can wire into their own workflows
Pain Points:
No deep links to individual runs or iterations; broken transient configurations; concurrency bottlenecks; chat transcripts clunky
GOals:
Shareable URLs for every run item, API parity with the dashboard, and results they can trust enough to act on
User Flows
The information architecture didn't match how people actually worked and was a confusing mental model for users to understand.
The run history was buried in tabs even though many users go straight to it. The GA structure names the layers explicitly: what you're testing (Suite → Simulation), how you run it, and what you're measuring (Run → Iteration).
Result: Simulations and Runs became separate navigation items rather than tabs, and a suite with a single simulation behaves like a standalone simulation, so the simple case stays simple.
Investigate
Research and Synthesis
UX Punchlist, gather customer pains, and synthesize feedback
UX Session with Applied AI and FDEs to create a punch-list of UX issues. Reviewed current painpoints from engineers who work directly with customers or are embeded within customer teams.
AI-assisted synthesis over months of meeting records. Synthesized feedback from slack customer channels, Notion and granola notes all together.
MVP
Create a simulation a few clicks and being able to demonstrate that it just works. From the runs list, suite list, or directly from the assistant page, users can configure evaluation plans, tool mocks, and variables per simulation, with proactive warnings for missing mocks or variable values before a run starts. Watch the evaluation rubric while the run executes instead of a blank loading state. Land on a dedicated full-page run detail with transcript and evaluations side by side, every run item deep-linkable for sharing and debugging.
Takeaways
01
In AI products, trust is a design surface. Every false positive teaches users to ignore your tool. The most important "features" we shipped were warnings, determinism, and honest states.
02
Make the simple path primary without punishing power. PMs got a two-click flow; power users kept run history, deep links, and API parity. Neither audience paid for the other's needs.
03
A pivot survives on decision hygiene. Logging every call as aligned, shelved, or cut, and revisiting nothing without new information, is how a mid-project redesign hit a hard GA date.
04
AI accelerated the process; it didn't run it. LLM-assisted synthesis over months of meetings and feedback kept the full decision history at hand, and prototyping moved at code speed. The judgment calls, what to cut, which audience wins, what trust requires, stayed human.
Role: Lead Product Designer
Tools: Figma, FigJam, AI-assisted synthesis & prototyping
Team: Design, Product, Engineering
Timeline: End of June – Mid Jul 2026 (GA)
Simulations: Testing Voice Agents Before They Talk to Customers
Vapi is a voice AI platform for developers — companies use it to build and run the AI agents that answer and make phone calls for support, scheduling, and sales. Simulations is Vapi's end-to-end testing surface: teams define scenarios and personalities, run their assistants (or squads of them) against simulated callers, and evaluate the results. The positioning we aligned on with customers: evals are unit tests; simulations are end-to-end tests.
Responsibilities
I'm the lead designer on Simulations, owned the dashboard experience end to end. I ran the design/eng/PM working sessions, drove the information architecture and creation-flow decisions, and scoped the final GA spec with engineering.
PROBLEM STATEMENT
The problem
Customers building production voice agents need reliable, fast, realistic ways to test agent behavior before and after changes, because a voice agent that fails does it out loud, on a live call, with a real customer.
The alpha had significant reliability and usability gaps. As a result, our most sophisticated customers were building their own testing infrastructure outside Vapi rather than relying on Simulations.
Alpha Release
Current alpha experience
Challenges
01
Trust is the product
Testing tool that produces false positives. Reliability issues: mocking, tester realism, determinism, were the core design problem, because every unreliable run taught users to ignore results.
02
Two audiences, one surface
Product managers needed a simple, reliable way to run tests, while the existing alpha users (developer-centric) tend to test primarily through the API and never touching the dashboard. We had to make the PM path primary without ripping out the workflow power users already depended on.
03
Redesigning mid-flight
The project was a classic fixing the plane while in it scenario, with the team pivoting several times during engineering work as the team worked up against a hard GA date.
Success metrics
Adoption
Reduced reliance on external and custom testing workarounds built by customers.
Task efficiency
Scenario creation in 1–2 clicks, decoupled from the generative flow.
Personas
Based on our team’s user research, there are two key personas we are focused on: Product Manager (primary persona for GA, limited technical depth, frequent in-product), and the developer (alpha cohort — advanced, often API-first users).
Product Manager
Responsibilities:
Define what "working" means for the agent; run tests before and after changes; share results with the team
Motivations:
Confidence that the agent behaves before it reaches customers
Pain Points:
Creation flow required detouring through a separate generative tool; unclear errors; results hard to parse; no way to tell which part of a simulation failed
GOals:
A simple, reliable way to define a test, run it, and read the verdict
Developer / Power User
Responsibilities:
Regression-test agents at scale, often dozens of simulations concurrently; debug failures down to the specific iteration
Motivations:
Deterministic results they can wire into their own workflows
Pain Points:
No deep links to individual runs or iterations; broken transient configurations; concurrency bottlenecks; chat transcripts clunky
GOals:
Shareable URLs for every run item, API parity with the dashboard, and results they can trust enough to act on
User Flows
The information architecture didn't match how people actually worked and was a confusing mental model for users to understand.
The run history was buried in tabs even though many users go straight to it. The GA structure names the layers explicitly: what you're testing (Suite → Simulation), how you run it, and what you're measuring (Run → Iteration).
Result: Simulations and Runs became separate navigation items rather than tabs, and a suite with a single simulation behaves like a standalone simulation, so the simple case stays simple.
Investigate
Research and Synthesis
UX Punchlist, gather customer pains, and synthesize feedback
UX Session with Applied AI and FDEs to create a punch-list of UX issues. Reviewed current painpoints from engineers who work directly with customers or are embeded within customer teams.
AI-assisted synthesis over months of meeting records. Synthesized feedback from slack customer channels, Notion and granola notes all together.
MVP
Create a simulation a few clicks and being able to demonstrate that it just works. From the runs list, suite list, or directly from the assistant page, users can configure evaluation plans, tool mocks, and variables per simulation, with proactive warnings for missing mocks or variable values before a run starts. Watch the evaluation rubric while the run executes instead of a blank loading state. Land on a dedicated full-page run detail with transcript and evaluations side by side, every run item deep-linkable for sharing and debugging.
Takeaways
01
In AI products, trust is a design surface. Every false positive teaches users to ignore your tool. The most important "features" we shipped were warnings, determinism, and honest states.
02
Make the simple path primary without punishing power. PMs got a two-click flow; power users kept run history, deep links, and API parity. Neither audience paid for the other's needs.
03
A pivot survives on decision hygiene. Logging every call as aligned, shelved, or cut, and revisiting nothing without new information, is how a mid-project redesign hit a hard GA date.
04
AI accelerated the process; it didn't run it. LLM-assisted synthesis over months of meetings and feedback kept the full decision history at hand, and prototyping moved at code speed. The judgment calls, what to cut, which audience wins, what trust requires, stayed human.
Simulations: Testing Voice Agents Before They Talk to Customers
Vapi is a voice AI platform for developers — companies use it to build and run the AI agents that answer and make phone calls for support, scheduling, and sales. Simulations is Vapi's end-to-end testing surface: teams define scenarios and personalities, run their assistants (or squads of them) against simulated callers, and evaluate the results. The positioning we aligned on with customers: evals are unit tests; simulations are end-to-end tests.
Responsibilities
I'm the lead designer on Simulations, owned the dashboard experience end to end. I ran the design/eng/PM working sessions, drove the information architecture and creation-flow decisions, and scoped the final GA spec with engineering.
Role: Lead Product Designer
Tools: Figma, FigJam, AI-assisted synthesis & prototyping
Team: Design, Product, Engineering
Timeline: End of June – Mid Jul 2026 (GA)
PROBLEM STATEMENT
The problem
Customers building production voice agents need reliable, fast, realistic ways to test agent behavior before and after changes, because a voice agent that fails does it out loud, on a live call, with a real customer.
The alpha had significant reliability and usability gaps. As a result, our most sophisticated customers were building their own testing infrastructure outside Vapi rather than relying on Simulations.
Alpha Release
Current alpha experience
Challenges
01
Trust is the product
Testing tool that produces false positives. Reliability issues: mocking, tester realism, determinism, were the core design problem, because every unreliable run taught users to ignore results.
02
Two audiences, one surface
Product managers needed a simple, reliable way to run tests, while the existing alpha users (developer-centric) tend to test primarily through the API and never touching the dashboard. We had to make the PM path primary without ripping out the workflow power users already depended on.
03
Redesigning mid-flight
The project was a classic fixing the plane while in it scenario, with the team pivoting several times during engineering work as the team worked up against a hard GA date.
Success metrics
Adoption
Reduced reliance on external and custom testing workarounds built by customers.
Task efficiency
Scenario creation in 1–2 clicks, decoupled from the generative flow.
Personas
Based on our team’s user research, there are two key personas we are focused on: Product Manager (primary persona for GA, limited technical depth, frequent in-product), and the developer (alpha cohort — advanced, often API-first users).
Product Manager
Responsibilities:
Define what "working" means for the agent; run tests before and after changes; share results with the team
Motivations:
Confidence that the agent behaves before it reaches customers
Pain Points:
Creation flow required detouring through a separate generative tool; unclear errors; results hard to parse; no way to tell which part of a simulation failed
GOals:
A simple, reliable way to define a test, run it, and read the verdict
Developer / Power User
Responsibilities:
Regression-test agents at scale, often dozens of simulations concurrently; debug failures down to the specific iteration
Motivations:
Deterministic results they can wire into their own workflows
Pain Points:
No deep links to individual runs or iterations; broken transient configurations; concurrency bottlenecks; chat transcripts clunky
GOals:
Shareable URLs for every run item, API parity with the dashboard, and results they can trust enough to act on
User Flows
The information architecture didn't match how people actually worked and was a confusing mental model for users to understand.
The run history was buried in tabs even though many users go straight to it. The GA structure names the layers explicitly: what you're testing (Suite → Simulation), how you run it, and what you're measuring (Run → Iteration).
Result: Simulations and Runs became separate navigation items rather than tabs, and a suite with a single simulation behaves like a standalone simulation, so the simple case stays simple.
Investigate
Research and Synthesis
UX Punchlist, gather customer pains, and synthesize feedback
UX Session with Applied AI and FDEs to create a punch-list of UX issues. Reviewed current painpoints from engineers who work directly with customers or are embeded within customer teams.
AI-assisted synthesis over months of meeting records. Synthesized feedback from slack customer channels, Notion and granola notes all together.
MVP
Create a simulation a few clicks and being able to demonstrate that it just works. From the runs list, suite list, or directly from the assistant page, users can configure evaluation plans, tool mocks, and variables per simulation, with proactive warnings for missing mocks or variable values before a run starts. Watch the evaluation rubric while the run executes instead of a blank loading state. Land on a dedicated full-page run detail with transcript and evaluations side by side, every run item deep-linkable for sharing and debugging.
Takeaways
01
In AI products, trust is a design surface. Every false positive teaches users to ignore your tool. The most important "features" we shipped were warnings, determinism, and honest states.
02
Make the simple path primary without punishing power. PMs got a two-click flow; power users kept run history, deep links, and API parity. Neither audience paid for the other's needs.
03
A pivot survives on decision hygiene. Logging every call as aligned, shelved, or cut, and revisiting nothing without new information, is how a mid-project redesign hit a hard GA date.
04
AI accelerated the process; it didn't run it. LLM-assisted synthesis over months of meetings and feedback kept the full decision history at hand, and prototyping moved at code speed. The judgment calls, what to cut, which audience wins, what trust requires, stayed human.