CallsQA Experiment Dashboard

CallsQA nine-call matrix · available recordings/transcripts may be cut off · run 20260715T123224197384Z
9unique calls
43speech pipelines
86end-to-end results
0execution errors
$1.3868batch known cost
1951.276ssummed step time
Group by

Where time and money went

Totals across 9 calls. Speech-to-text is counted once per pipeline; evaluator phases are shown separately, without duplicating shared STT.

Speech-to-textGemma 4 26BGPT-5.5

Total processing time

summed phase time
ElevenLabs Scribe v2highest9 calls
493.3s
MLX Whisper large-v39 calls
480.4s
OpenAI 4o transcribe9 calls
379.6s
OpenAI 4o mini9 calls
364.7s
Existing transcript7 calls
233.2s

Total known cost

estimated and provider-reported
OpenAI 4o transcribehighest9 calls
$0.3999
ElevenLabs Scribe v29 calls
$0.3224
OpenAI 4o mini9 calls
$0.3089
MLX Whisper large-v39 calls
$0.1987
Existing transcript7 calls
$0.1569

Sample 01

5 speech pipelines · 10 results

MLX Whisper large-v3

sample-01-mlx-known
252 wordsSTT 14.4sSTT $0.0000
Gemma 4 26B
Tone 2 · Clarity 2 · Efficiency 2
valid · fenced_json · 1 candidate(s)
outcome unresolved · unable to evaluate false

The agent's tone was somewhat dismissive and unprofessional (e.g., saying 'I don't know, okay'). While the agent correctly informed the customer that orders must be placed via the website, they were unable to assist with the specific product inquiry and ended the call abruptly once they could not find the product number in their system.

  • Unprofessional and dismissive tone
  • Failure to assist with product identification
  • Abrupt termination of the call without resolution
Pipeline time & cost (STT + analysis) · 70.5s · $0.0195
PhaseLatencyFinancial cost
Speech-to-textMLX Whisper large-v3
14.4stime
$0.0000runtime_hours
AnalysisGemma 4 26B
56.1stime
$0.0195runtime_hours
Pipeline total70.5s$0.0195
GPT-5.5
Tone 4 · Clarity 3 · Efficiency 3
valid · direct_json · 1 candidate(s)
outcome partial · unable to evaluate false

The agent was generally polite and explained that orders must be placed through the website, but the conversation was somewhat confusing and repetitive. The customer’s request to identify or order a specific perfume was not fully resolved.

  • Agent could not identify the product from the number provided by the customer.
  • Agent did not offer an alternative way to search for the product, such as asking for more details or guiding the customer step by step on the website.
  • Conversation lacked structure and required repeated clarification.
Pipeline time & cost (STT + analysis) · 23.5s · $0.0124
PhaseLatencyFinancial cost
Speech-to-textMLX Whisper large-v3
14.4stime
$0.0000runtime_hours
AnalysisGPT-5.5
9.1stime
$0.0124tokens
Pipeline total23.5s$0.0124

OpenAI gpt-4o-transcribe

sample-01-openai-full
240 wordsSTT 15.7sSTT $0.0192
Gemma 4 26B
Tone 4 · Clarity 4 · Efficiency 3
valid · fenced_json · 1 candidate(s)
outcome unresolved · unable to evaluate false

The agent was polite and clearly explained the company's policy regarding website-only orders. However, the customer's primary need (placing an order) was not met because they were unable to use the website, and the agent could not assist manually. Additionally, the agent was unable to identify the specific product the customer was referring to.

  • Customer is unable to use the online ordering system
  • Agent cannot process orders over the phone
  • Difficulty identifying the specific product requested by the customer
Pipeline time & cost (STT + analysis) · 35.9s · $0.0262
PhaseLatencyFinancial cost
Speech-to-textOpenAI gpt-4o-transcribe
15.7stime
$0.0192audio_minutes
AnalysisGemma 4 26B
20.2stime
$0.0070runtime_hours
Pipeline total35.9s$0.0262
GPT-5.5
Tone 4 · Clarity 3 · Efficiency 3
valid · direct_json · 1 candidate(s)
outcome unresolved · unable to evaluate false

The agent was generally polite and explained that phone orders are not accepted, directing the customer to the website. However, the interaction was somewhat confusing, and the agent did not provide much additional assistance for a customer who stated they had difficulty ordering online. The requested product could not be identified or ordered, so the customer's issue remained unresolved.

  • Agent could not place an order by phone and offered limited alternative support.
  • Product identification was unsuccessful based on the customer's number and description.
  • The call ended without a clear next step beyond using the website.
Pipeline time & cost (STT + analysis) · 22.3s · $0.0304
PhaseLatencyFinancial cost
Speech-to-textOpenAI gpt-4o-transcribe
15.7stime
$0.0192audio_minutes
AnalysisGPT-5.5
6.6stime
$0.0112tokens
Pipeline total22.3s$0.0304

OpenAI gpt-4o-mini-transcribe

sample-01-openai-mini
110 wordsSTT 3.8sSTT $0.0096
Gemma 4 26B
Tone 4 · Clarity 3 · Efficiency 3
valid · fenced_json · 1 candidate(s)
outcome unresolved · unable to evaluate false

The agent was polite and professional, but the call ended without resolving the customer's inquiry because the agent could not locate the order number provided in the system.

  • Inability to locate the order based on the information provided
  • Conflicting or unclear order information provided by the customer
Pipeline time & cost (STT + analysis) · 28.6s · $0.0182
PhaseLatencyFinancial cost
Speech-to-textOpenAI gpt-4o-mini-transcribe
3.8stime
$0.0096audio_minutes
AnalysisGemma 4 26B
24.8stime
$0.0086runtime_hours
Pipeline total28.6s$0.0182
GPT-5.5
Tone 4 · Clarity 3 · Efficiency 3
valid · direct_json · 1 candidate(s)
outcome unresolved · unable to evaluate false

The agent was polite and attempted to locate the order, but the issue was not resolved because the provided number could not be found and no alternative lookup method was offered.

  • Order could not be identified from the provided information.
  • Agent did not offer alternative ways to search for the order, such as customer name, email, phone number, or full order number.
  • Call ended without a clear next step for the customer.
Pipeline time & cost (STT + analysis) · 9.9s · $0.0189
PhaseLatencyFinancial cost
Speech-to-textOpenAI gpt-4o-mini-transcribe
3.8stime
$0.0096audio_minutes
AnalysisGPT-5.5
6.1stime
$0.0093tokens
Pipeline total9.9s$0.0189

ElevenLabs Scribe v2

sample-01-elevenlabs
283 wordsSTT 16.2sSTT $0.0117
Gemma 4 26B
Tone 3 · Clarity 3 · Efficiency 2
valid · fenced_json · 1 candidate(s)
outcome unresolved · unable to evaluate false

The agent correctly informed the customer that orders cannot be placed over the phone and directed them to the website. However, the call was unproductive because the customer could not provide specific product details, and the agent's tone was somewhat blunt and abrupt during the questioning.

  • Customer was unable to identify the specific product
  • Agent's tone was somewhat curt/abrupt
  • The customer's goal of placing an order was not achieved
Pipeline time & cost (STT + analysis) · 43.3s · $0.0211
PhaseLatencyFinancial cost
Speech-to-textElevenLabs Scribe v2
16.2stime
$0.0117audio_minutes
AnalysisGemma 4 26B
27.2stime
$0.0094runtime_hours
Pipeline total43.3s$0.0211
GPT-5.5
Tone 4 · Clarity 4 · Efficiency 4
valid · direct_json · 1 candidate(s)
outcome unresolved · unable to evaluate false

The agent was polite and explained that orders cannot be placed by phone, directing the customer to the website. The agent also attempted to look up the product code, but the item could not be identified and the customer’s request was not resolved.

  • Customer was unable to place the order by phone.
  • Product could not be identified from the number provided.
  • Agent did not offer an alternative support option for helping the customer complete the online order.
Pipeline time & cost (STT + analysis) · 22.2s · $0.0232
PhaseLatencyFinancial cost
Speech-to-textElevenLabs Scribe v2
16.2stime
$0.0117audio_minutes
AnalysisGPT-5.5
6.0stime
$0.0114tokens
Pipeline total22.2s$0.0232

Existing transcript

sample-01-received
257 wordsSTT 0.0sSTT $0.0000
Gemma 4 26B
Tone 2 · Clarity 3 · Efficiency 2
valid · fenced_json · 1 candidate(s)
outcome unresolved · unable to evaluate false

The operator's tone was somewhat blunt and lacked professional warmth, using phrases like 'I don't know' and 'Why are you calling now?'. While the instructions on how to use the website were clear, the operator was unable to find the specific product the customer was looking for, leaving the customer's request to purchase the item unresolved.

  • Abrupt and unprofessional tone
  • Inability to locate the requested product in the system
  • Failure to resolve the customer's intent to purchase
Pipeline time & cost (STT + analysis) · 30.8s · $0.0107
PhaseLatencyFinancial cost
Speech-to-textExisting transcript
0.0stime
n/aunpriced
AnalysisGemma 4 26B
30.8stime
$0.0107runtime_hours
Pipeline total30.8s$0.0107
GPT-5.5
Tone 3 · Clarity 3 · Efficiency 3
valid · direct_json · 1 candidate(s)
outcome partial · unable to evaluate false

The operator remained generally polite and explained that orders must be placed through the website, and also checked the requested perfume number. However, the interaction was somewhat repetitive and abrupt, and the operator did not offer much additional assistance when the item number could not be found.

  • Some confusion around whether the customer was referring to an order number or perfume number.
  • The operator could have asked clearer follow-up questions earlier to identify the customer's need.
  • The call ended without offering alternatives or guidance for finding the [brand redacted] perfume on the website.
Pipeline time & cost (STT + analysis) · 6.9s · $0.0125
PhaseLatencyFinancial cost
Speech-to-textExisting transcript
0.0stime
n/aunpriced
AnalysisGPT-5.5
6.9stime
$0.0125tokens
Pipeline total6.9s$0.0125

Sample 02

4 speech pipelines · 8 results

MLX Whisper large-v3

sample-02-mlx-auto
265 wordsSTT 17.0sSTT $0.0000
Gemma 4 26B
Tone 5 · Clarity 5 · Efficiency 5
valid · fenced_json · 1 candidate(s)
outcome resolved · unable to evaluate false

The agent was polite and professional. They clearly addressed the customer's questions regarding custom sizes, the lack of installation services, and the necessity of ordering via the website.

  • The transcript begins with a long sequence of repetitive 'Áno' (Yes/No) entries, which may indicate connection noise or pre-call audio artifacts.
Pipeline time & cost (STT + analysis) · 40.8s · $0.0083
PhaseLatencyFinancial cost
Speech-to-textMLX Whisper large-v3
17.0stime
$0.0000runtime_hours
AnalysisGemma 4 26B
23.8stime
$0.0083runtime_hours
Pipeline total40.8s$0.0083
GPT-5.5
Tone 5 · Clarity 4 · Efficiency 4
valid · direct_json · 1 candidate(s)
outcome partial · unable to evaluate false

The agent was polite and provided clear information about product availability, ordering through the website, and the lack of assembly service. The customer's questions were answered, but the customer could not complete an order by phone or get assembly support, so the outcome was only partial.

  • Phone ordering is not available.
  • Assembly service is not provided.
Pipeline time & cost (STT + analysis) · 25.5s · $0.0131
PhaseLatencyFinancial cost
Speech-to-textMLX Whisper large-v3
17.0stime
$0.0000runtime_hours
AnalysisGPT-5.5
8.6stime
$0.0131tokens
Pipeline total25.5s$0.0131

OpenAI gpt-4o-transcribe

sample-02-openai-full
231 wordsSTT 7.7sSTT $0.0178
Gemma 4 26B
Tone 4 · Clarity 4 · Efficiency 5
valid · fenced_json · 1 candidate(s)
outcome resolved · unable to evaluate false

The agent effectively addressed all the customer's inquiries regarding product availability (no custom sizes), assembly services (not provided), and the ordering process (website only). The agent maintained a polite and professional tone throughout the call.

Pipeline time & cost (STT + analysis) · 40.2s · $0.0291
PhaseLatencyFinancial cost
Speech-to-textOpenAI gpt-4o-transcribe
7.7stime
$0.0178audio_minutes
AnalysisGemma 4 26B
32.5stime
$0.0113runtime_hours
Pipeline total40.2s$0.0291
GPT-5.5
Tone 5 · Clarity 4 · Efficiency 4
valid · direct_json · 1 candidate(s)
outcome resolved · unable to evaluate false

The agent was polite and professional, clearly explained that only website-listed sizes are available, custom production and assembly are not offered, and orders must be placed through the website. The customer's questions were answered, though some explanations were repetitive due to the customer's follow-up questions.

  • No phone ordering option was available, and the agent did not offer additional guidance beyond directing the customer to the website.
Pipeline time & cost (STT + analysis) · 15.2s · $0.0305
PhaseLatencyFinancial cost
Speech-to-textOpenAI gpt-4o-transcribe
7.7stime
$0.0178audio_minutes
AnalysisGPT-5.5
7.5stime
$0.0127tokens
Pipeline total15.2s$0.0305

OpenAI gpt-4o-mini-transcribe

sample-02-openai-mini
290 wordsSTT 4.9sSTT $0.0089
Gemma 4 26B
Tone 5 · Clarity 5 · Efficiency 5
valid · fenced_json · 1 candidate(s)
outcome resolved · unable to evaluate false

The agent was professional and polite. They clearly addressed all of the customer's inquiries regarding custom sizes, assembly services, delivery fees, and the specific procedure for placing an order online.

Pipeline time & cost (STT + analysis) · 34.1s · $0.0190
PhaseLatencyFinancial cost
Speech-to-textOpenAI gpt-4o-mini-transcribe
4.9stime
$0.0089audio_minutes
AnalysisGemma 4 26B
29.2stime
$0.0101runtime_hours
Pipeline total34.1s$0.0190
GPT-5.5
Tone 5 · Clarity 4 · Efficiency 4
valid · direct_json · 1 candidate(s)
outcome resolved · unable to evaluate false

The agent was polite and answered the customer's questions about product availability, shipping cost, cash-on-delivery fee, assembly service, and ordering method. The conversation was somewhat repetitive and the transcript is partially unclear, but the main information was communicated effectively.

  • The agent could not place an order by phone and directed the customer to the website.
  • Assembly service is not provided, which may not fully meet the customer's needs.
Pipeline time & cost (STT + analysis) · 12.2s · $0.0211
PhaseLatencyFinancial cost
Speech-to-textOpenAI gpt-4o-mini-transcribe
4.9stime
$0.0089audio_minutes
AnalysisGPT-5.5
7.3stime
$0.0123tokens
Pipeline total12.2s$0.0211

ElevenLabs Scribe v2

sample-02-elevenlabs
306 wordsSTT 17.7sSTT $0.0109
Gemma 4 26B
Tone 5 · Clarity 5 · Efficiency 5
valid · fenced_json · 1 candidate(s)
outcome resolved · unable to evaluate false

The agent was polite and professional. They clearly answered all the customer's questions regarding product availability, shipping costs (noting the free shipping but 10 EUR COD fee), and the lack of assembly services. The agent also correctly directed the customer to the website for ordering.

Pipeline time & cost (STT + analysis) · 35.7s · $0.0171
PhaseLatencyFinancial cost
Speech-to-textElevenLabs Scribe v2
17.7stime
$0.0109audio_minutes
AnalysisGemma 4 26B
18.0stime
$0.0062runtime_hours
Pipeline total35.7s$0.0171
GPT-5.5
Tone 5 · Clarity 4 · Efficiency 4
valid · direct_json · 1 candidate(s)
outcome partial · unable to evaluate false

The agent was polite, patient, and provided clear answers about availability, free delivery, cash-on-delivery fee, assembly, and online ordering. The customer did not place an order during the call and planned to check assembly options first, so the outcome was partial.

Pipeline time & cost (STT + analysis) · 24.0s · $0.0216
PhaseLatencyFinancial cost
Speech-to-textElevenLabs Scribe v2
17.7stime
$0.0109audio_minutes
AnalysisGPT-5.5
6.3stime
$0.0108tokens
Pipeline total24.0s$0.0216

Sample 03

4 speech pipelines · 8 results

MLX Whisper large-v3

sample-03-mlx-auto
277 wordsSTT 14.4sSTT $0.0000
Gemma 4 26B
Tone 5 · Clarity 5 · Efficiency 5
valid · fenced_json · 1 candidate(s)
outcome resolved · unable to evaluate false

The agent handled the complaint regarding a damaged product with empathy and provided very clear, actionable instructions on how to proceed with the claim (emailing specific details, providing photos, and stating the desired resolution).

Pipeline time & cost (STT + analysis) · 31.9s · $0.0061
PhaseLatencyFinancial cost
Speech-to-textMLX Whisper large-v3
14.4stime
$0.0000runtime_hours
AnalysisGemma 4 26B
17.4stime
$0.0061runtime_hours
Pipeline total31.9s$0.0061
GPT-5.5
Tone 4 · Clarity 4 · Efficiency 4
valid · direct_json · 1 candidate(s)
outcome partial · unable to evaluate false

The agent was polite and empathetic, apologized for the damaged product, and clearly explained the next steps for submitting a claim with photos and refund/replacement preference. The issue was not resolved during the call, but the customer was given actionable instructions.

  • Claim still needs to be submitted by email before resolution can proceed.
  • No specific resolution timeframe was provided beyond a general statement that they would try to handle it quickly.
Pipeline time & cost (STT + analysis) · 20.3s · $0.0102
PhaseLatencyFinancial cost
Speech-to-textMLX Whisper large-v3
14.4stime
$0.0000runtime_hours
AnalysisGPT-5.5
5.9stime
$0.0102tokens
Pipeline total20.3s$0.0102

OpenAI gpt-4o-transcribe

sample-03-openai-full
295 wordsSTT 7.7sSTT $0.0139
Gemma 4 26B
Tone 5 · Clarity 5 · Efficiency 5
valid · fenced_json · 1 candidate(s)
outcome resolved · unable to evaluate false

The agent handled the complaint professionally and with empathy. They provided clear, actionable instructions on exactly what the customer needs to include in their email (description, photos of damage, photo of the whole item, and preference for replacement or refund) to initiate the reclamation process.

Pipeline time & cost (STT + analysis) · 22.4s · $0.0190
PhaseLatencyFinancial cost
Speech-to-textOpenAI gpt-4o-transcribe
7.7stime
$0.0139audio_minutes
AnalysisGemma 4 26B
14.7stime
$0.0051runtime_hours
Pipeline total22.4s$0.0190
GPT-5.5
Tone 5 · Clarity 4 · Efficiency 4
valid · direct_json · 1 candidate(s)
outcome partial · unable to evaluate false

The agent was polite, empathetic, and explained the complaint process clearly enough, including what photos and details the customer should send by email. The issue was not resolved during the call because the customer still needs to submit documentation for the claim.

  • No claim or case number was created during the call.
  • Resolution depends on the customer sending follow-up information by email.
Pipeline time & cost (STT + analysis) · 15.5s · $0.0259
PhaseLatencyFinancial cost
Speech-to-textOpenAI gpt-4o-transcribe
7.7stime
$0.0139audio_minutes
AnalysisGPT-5.5
7.8stime
$0.0120tokens
Pipeline total15.5s$0.0259

OpenAI gpt-4o-mini-transcribe

sample-03-openai-mini
292 wordsSTT 4.7sSTT $0.0070
Gemma 4 26B
Tone 5 · Clarity 5 · Efficiency 5
valid · fenced_json · 1 candidate(s)
outcome resolved · unable to evaluate false

The agent handled the complaint with empathy and provided very clear, actionable instructions on how to proceed with the claim, including exactly what information and photos are required in the follow-up email.

Pipeline time & cost (STT + analysis) · 24.4s · $0.0138
PhaseLatencyFinancial cost
Speech-to-textOpenAI gpt-4o-mini-transcribe
4.7stime
$0.0070audio_minutes
AnalysisGemma 4 26B
19.7stime
$0.0068runtime_hours
Pipeline total24.4s$0.0138
GPT-5.5
Tone 5 · Clarity 4 · Efficiency 4
valid · direct_json · 1 candidate(s)
outcome partial · unable to evaluate false

The agent was polite, empathetic, and provided clear next steps for submitting a complaint with required photos and information. The issue was not resolved during the call, but the customer understood what to do next.

  • Resolution depends on the customer sending an email and the complaints department reviewing it.
  • The agent did not provide a specific expected timeframe for the complaint resolution.
Pipeline time & cost (STT + analysis) · 10.1s · $0.0167
PhaseLatencyFinancial cost
Speech-to-textOpenAI gpt-4o-mini-transcribe
4.7stime
$0.0070audio_minutes
AnalysisGPT-5.5
5.4stime
$0.0097tokens
Pipeline total10.1s$0.0167

ElevenLabs Scribe v2

sample-03-elevenlabs
334 wordsSTT 18.8sSTT $0.0085
Gemma 4 26B
Tone 5 · Clarity 5 · Efficiency 5
valid · fenced_json · 1 candidate(s)
outcome resolved · unable to evaluate false

The agent handled the complaint professionally, expressing empathy for the damaged product and providing clear, actionable instructions on how to proceed with the claim via email, including the specific details and documentation required.

Pipeline time & cost (STT + analysis) · 39.1s · $0.0155
PhaseLatencyFinancial cost
Speech-to-textElevenLabs Scribe v2
18.8stime
$0.0085audio_minutes
AnalysisGemma 4 26B
20.3stime
$0.0070runtime_hours
Pipeline total39.1s$0.0155
GPT-5.5
Tone 5 · Clarity 4 · Efficiency 4
valid · direct_json · 1 candidate(s)
outcome partial · unable to evaluate false

The agent was polite, empathetic, and provided clear next steps for submitting a damage claim, including what information and photos to send. The issue was not fully resolved during the call because the customer still needs to email the documentation and wait for claims processing.

  • Resolution depends on follow-up by email and claims department review.
  • Some instructions were slightly repetitive due to interruptions, but overall understandable.
Pipeline time & cost (STT + analysis) · 25.2s · $0.0192
PhaseLatencyFinancial cost
Speech-to-textElevenLabs Scribe v2
18.8stime
$0.0085audio_minutes
AnalysisGPT-5.5
6.5stime
$0.0107tokens
Pipeline total25.2s$0.0192

Sample 04

5 speech pipelines · 10 results

MLX Whisper large-v3

sample-04-mlx-known
302 wordsSTT 16.0sSTT $0.0000
Gemma 4 26B
Tone 4 · Clarity 3 · Efficiency 4
valid · fenced_json · 1 candidate(s)
outcome resolved · unable to evaluate false

The agent successfully provided the manufacturer's name and proactively spelled it out when the customer reported difficulty hearing due to poor audio quality. Although the agent could not provide the manufacturer's location or specific restock dates, the customer's primary goal of identifying the manufacturer was achieved.

  • Poor audio quality/microphone fuzziness
Pipeline time & cost (STT + analysis) · 41.8s · $0.0090
PhaseLatencyFinancial cost
Speech-to-textMLX Whisper large-v3
16.0stime
$0.0000runtime_hours
AnalysisGemma 4 26B
25.8stime
$0.0090runtime_hours
Pipeline total41.8s$0.0090
GPT-5.5
Tone 4 · Clarity 4 · Efficiency 4
valid · direct_json · 1 candidate(s)
outcome partial · unable to evaluate false

The agent was polite and provided the manufacturer name, spelling it out when the customer had trouble hearing. The agent also explained the stock notification process and limitations. However, the customer’s broader issue of repeatedly being unable to obtain the product was not fully resolved, and no alternative solution or escalation was offered.

  • No clear alternative or escalation was offered for the repeated out-of-stock issue.
  • The agent could not provide manufacturer location or direct sourcing information.
  • Limited empathy was shown regarding the customer's repeated failed orders.
Pipeline time & cost (STT + analysis) · 26.3s · $0.0130
PhaseLatencyFinancial cost
Speech-to-textMLX Whisper large-v3
16.0stime
$0.0000runtime_hours
AnalysisGPT-5.5
10.4stime
$0.0130tokens
Pipeline total26.3s$0.0130

OpenAI gpt-4o-transcribe

sample-04-openai-full
307 wordsSTT 9.8sSTT $0.0157
Gemma 4 26B
Tone 4 · Clarity 2 · Efficiency 3
valid · fenced_json · 1 candidate(s)
outcome resolved · unable to evaluate false

The agent successfully provided the manufacturer's name and explained the stock notification process. However, the call suffered from poor clarity when the agent first stated the manufacturer's name, which the customer could not understand, necessitating the agent to spell it out letter by letter.

  • Poor clarity regarding the manufacturer's name
  • Communication breakdown requiring spelling out of information
Pipeline time & cost (STT + analysis) · 30.0s · $0.0227
PhaseLatencyFinancial cost
Speech-to-textOpenAI gpt-4o-transcribe
9.8stime
$0.0157audio_minutes
AnalysisGemma 4 26B
20.3stime
$0.0070runtime_hours
Pipeline total30.0s$0.0227
GPT-5.5
Tone 4 · Clarity 4 · Efficiency 3
valid · direct_json · 1 candidate(s)
outcome partial · unable to evaluate false

The agent was polite and provided the manufacturer's name, including spelling it out to resolve an audio/understanding issue. However, the agent could not provide more detailed sourcing information or a concrete solution for obtaining the out-of-stock item. The explanation about limited stock and notifications was clear but did not fully address the customer's recurring issue.

  • No concrete solution offered for the customer's repeated failed attempts to purchase the item.
  • Agent could not provide manufacturer location or direct purchasing guidance.
  • Limited empathy shown regarding the customer's frustration with repeated cancellations.
Pipeline time & cost (STT + analysis) · 16.8s · $0.0276
PhaseLatencyFinancial cost
Speech-to-textOpenAI gpt-4o-transcribe
9.8stime
$0.0157audio_minutes
AnalysisGPT-5.5
7.1stime
$0.0118tokens
Pipeline total16.8s$0.0276

OpenAI gpt-4o-mini-transcribe

sample-04-openai-mini
918 wordsSTT 11.5sSTT $0.0079
Gemma 4 26B
Tone 3 · Clarity 1 · Efficiency 1
valid · fenced_json · 1 candidate(s)
outcome unresolved · unable to evaluate true

The transcript is insufficient for a full evaluation because the customer's dialogue consists of an extreme, repetitive loop of the same sentence, likely due to a transcription error. The conversation ends abruptly without any interaction from the agent following the customer's initial statement.

  • Extreme repetition/looping in the transcript
  • Incomplete conversation
  • Transcript ends abruptly
Pipeline time & cost (STT + analysis) · 38.7s · $0.0173
PhaseLatencyFinancial cost
Speech-to-textOpenAI gpt-4o-mini-transcribe
11.5stime
$0.0079audio_minutes
AnalysisGemma 4 26B
27.2stime
$0.0094runtime_hours
Pipeline total38.7s$0.0173
GPT-5.5
Tone 0 · Clarity 0 · Efficiency 0
valid · direct_json · 1 candidate(s)
outcome unknown · unable to evaluate true

The transcript is insufficient for QA evaluation because it appears corrupted and highly repetitive, with only the opening and the customer's repeated issue visible. There is no meaningful agent response, troubleshooting, or call outcome available.

  • Transcript appears duplicated or corrupted
  • Agent handling cannot be assessed beyond the greeting
  • No resolution or next steps are captured
Pipeline time & cost (STT + analysis) · 16.3s · $0.0258
PhaseLatencyFinancial cost
Speech-to-textOpenAI gpt-4o-mini-transcribe
11.5stime
$0.0079audio_minutes
AnalysisGPT-5.5
4.8stime
$0.0179tokens
Pipeline total16.3s$0.0258

ElevenLabs Scribe v2

sample-04-elevenlabs
340 wordsSTT 20.9sSTT $0.0096
Gemma 4 26B
Tone 5 · Clarity 4 · Efficiency 5
valid · fenced_json · 1 candidate(s)
outcome resolved · unable to evaluate false

The agent was professional, polite, and helpful. When the customer had difficulty hearing the manufacturer's name due to audio issues, the agent proactively spelled it out. The agent answered the customer's questions regarding the manufacturer and the notification process, even though they could not provide the manufacturer's location.

  • The customer reported poor audio/microphone quality, making it difficult to hear certain words.
Pipeline time & cost (STT + analysis) · 55.0s · $0.0214
PhaseLatencyFinancial cost
Speech-to-textElevenLabs Scribe v2
20.9stime
$0.0096audio_minutes
AnalysisGemma 4 26B
34.1stime
$0.0118runtime_hours
Pipeline total55.0s$0.0214
GPT-5.5
Tone 5 · Clarity 4 · Efficiency 4
valid · direct_json · 1 candidate(s)
outcome partial · unable to evaluate false

The agent was polite and provided the manufacturer name, spelling it out when the customer had difficulty hearing. The agent also clearly explained that stock is limited and that reservations or additional orders are not possible. The customer received some useful information but the main issue of obtaining the product was not fully resolved.

  • Audio or microphone quality made the manufacturer name hard for the customer to understand initially.
  • The agent could not provide details about where the manufacturer is located or where else the product might be available.
  • No alternative solution beyond signing up for stock notifications was available.
Pipeline time & cost (STT + analysis) · 27.1s · $0.0216
PhaseLatencyFinancial cost
Speech-to-textElevenLabs Scribe v2
20.9stime
$0.0096audio_minutes
AnalysisGPT-5.5
6.2stime
$0.0120tokens
Pipeline total27.1s$0.0216

Existing transcript

sample-04-received
292 wordsSTT 0.0sSTT $0.0000
Gemma 4 26B
Tone 3 · Clarity 2 · Efficiency 3
valid · fenced_json · 1 candidate(s)
outcome partial · unable to evaluate false

The operator was polite but the call had a confusing and unprofessional opening regarding call recordings. The operator also struggled to clearly communicate the manufacturer's name, necessitating spelling it out. While the customer's specific question about the manufacturer was answered, their primary issue regarding product availability remains unresolved.

  • Unprofessional/confusing opening statement
  • Poor clarity when stating the manufacturer's name
  • Lack of helpful information regarding stock replenishment or manufacturer location
Pipeline time & cost (STT + analysis) · 26.2s · $0.0091
PhaseLatencyFinancial cost
Speech-to-textExisting transcript
0.0stime
n/aunpriced
AnalysisGemma 4 26B
26.2stime
$0.0091runtime_hours
Pipeline total26.2s$0.0091
GPT-5.5
Tone 4 · Clarity 4 · Efficiency 4
valid · direct_json · 1 candidate(s)
outcome partial · unable to evaluate false

The operator was polite and provided the manufacturer name, spelling it out when the customer had difficulty hearing. The explanation about limited availability and notifications was clear, but the operator could not provide the country of manufacture or a concrete restock date, so the customer's broader issue was only partially addressed.

  • Audio or microphone quality made it difficult for the customer to hear the manufacturer name.
  • No concrete restock timeline or alternative purchasing path was provided.
  • Limited empathy was shown regarding the customer's repeated failed orders.
Pipeline time & cost (STT + analysis) · 9.0s · $0.0109
PhaseLatencyFinancial cost
Speech-to-textExisting transcript
0.0stime
n/aunpriced
AnalysisGPT-5.5
9.0stime
$0.0109tokens
Pipeline total9.0s$0.0109

Sample 05

5 speech pipelines · 10 results

MLX Whisper large-v3

sample-05-mlx-known
345 wordsSTT 17.2sSTT $0.0000
Gemma 4 26B
Tone 5 · Clarity 5 · Efficiency 5
valid · fenced_json · 1 candidate(s)
outcome resolved · unable to evaluate false

The agent successfully addressed the customer's concern regarding a delayed delivery by providing the courier's customer service phone number, enabling the customer to follow up directly.

Pipeline time & cost (STT + analysis) · 36.3s · $0.0066
PhaseLatencyFinancial cost
Speech-to-textMLX Whisper large-v3
17.2stime
$0.0000runtime_hours
AnalysisGemma 4 26B
19.1stime
$0.0066runtime_hours
Pipeline total36.3s$0.0066
GPT-5.5
Tone 5 · Clarity 4 · Efficiency 4
valid · direct_json · 1 candidate(s)
outcome partial · unable to evaluate false

The agent was polite and helpful, checked the order status, explained that the shipment was with the courier and delayed, and provided the courier's customer service phone number. The customer's underlying delivery concern was not resolved during the call, but a reasonable next step was provided.

  • The agent could not provide a definitive delivery date.
  • The explanation of the delay was somewhat general and based on broad courier delays rather than a confirmed carrier update.
Pipeline time & cost (STT + analysis) · 23.5s · $0.0122
PhaseLatencyFinancial cost
Speech-to-textMLX Whisper large-v3
17.2stime
$0.0000runtime_hours
AnalysisGPT-5.5
6.3stime
$0.0122tokens
Pipeline total23.5s$0.0122

OpenAI gpt-4o-transcribe

sample-05-openai-full
327 wordsSTT 8.8sSTT $0.0192
Gemma 4 26B
Tone 5 · Clarity 4 · Efficiency 5
valid · fenced_json · 1 candidate(s)
outcome resolved · unable to evaluate false

The agent successfully identified the customer's order, confirmed the shipping status, and provided the requested contact information for the courier to help the customer manage the delivery timing.

Pipeline time & cost (STT + analysis) · 33.3s · $0.0277
PhaseLatencyFinancial cost
Speech-to-textOpenAI gpt-4o-transcribe
8.8stime
$0.0192audio_minutes
AnalysisGemma 4 26B
24.6stime
$0.0085runtime_hours
Pipeline total33.3s$0.0277
GPT-5.5
Tone 5 · Clarity 4 · Efficiency 4
valid · direct_json · 1 candidate(s)
outcome partial · unable to evaluate false

The agent was polite and helpful, checked the order status, explained that the package was with the courier and likely delayed, and provided the courier service phone number. However, the delivery issue itself was not resolved during the call, and the customer still needed to contact the courier separately.

  • No direct resolution or confirmed delivery date was provided.
  • The agent did not offer to contact or escalate with the courier on the customer's behalf.
  • The explanation for the delay was somewhat general and not specific to the customer's shipment.
Pipeline time & cost (STT + analysis) · 15.4s · $0.0313
PhaseLatencyFinancial cost
Speech-to-textOpenAI gpt-4o-transcribe
8.8stime
$0.0192audio_minutes
AnalysisGPT-5.5
6.6stime
$0.0121tokens
Pipeline total15.4s$0.0313

OpenAI gpt-4o-mini-transcribe

sample-05-openai-mini
307 wordsSTT 8.0sSTT $0.0096
Gemma 4 26B
Tone 5 · Clarity 5 · Efficiency 5
valid · fenced_json · 1 candidate(s)
outcome resolved · unable to evaluate false

The agent was polite and professional. They successfully addressed the customer's concern regarding the delayed delivery by providing the courier's customer service phone number, enabling the customer to take the next step themselves.

Pipeline time & cost (STT + analysis) · 26.2s · $0.0159
PhaseLatencyFinancial cost
Speech-to-textOpenAI gpt-4o-mini-transcribe
8.0stime
$0.0096audio_minutes
AnalysisGemma 4 26B
18.2stime
$0.0063runtime_hours
Pipeline total26.2s$0.0159
GPT-5.5
Tone 4 · Clarity 4 · Efficiency 3
valid · direct_json · 1 candidate(s)
outcome partial · unable to evaluate false

The agent was polite and provided basic shipment information and the courier's phone number. However, the delivery issue itself was not resolved, and the customer had to contact the courier directly.

  • Agent did not offer to contact the courier or escalate the delayed delivery.
  • Explanation for the delay was general and not specific to the customer's shipment.
  • Resolution depended on the customer taking further action.
Pipeline time & cost (STT + analysis) · 14.4s · $0.0215
PhaseLatencyFinancial cost
Speech-to-textOpenAI gpt-4o-mini-transcribe
8.0stime
$0.0096audio_minutes
AnalysisGPT-5.5
6.4stime
$0.0119tokens
Pipeline total14.4s$0.0215

ElevenLabs Scribe v2

sample-05-elevenlabs
363 wordsSTT 18.6sSTT $0.0117
Gemma 4 26B
Tone 5 · Clarity 5 · Efficiency 5
valid · fenced_json · 1 candidate(s)
outcome resolved · unable to evaluate false

The agent was polite and helpful. They successfully identified the delivery status and provided the customer with the courier's contact number as requested, which addressed the customer's immediate need to track the package themselves.

Pipeline time & cost (STT + analysis) · 39.7s · $0.0191
PhaseLatencyFinancial cost
Speech-to-textElevenLabs Scribe v2
18.6stime
$0.0117audio_minutes
AnalysisGemma 4 26B
21.1stime
$0.0073runtime_hours
Pipeline total39.7s$0.0191
GPT-5.5
Tone 5 · Clarity 4 · Efficiency 4
valid · direct_json · 1 candidate(s)
outcome partial · unable to evaluate false

The agent was polite and helpful, checked the order status, explained that the package was with the courier and experiencing delays, and provided the courier's phone number. The shipment issue itself was not resolved during the call, but the customer received the next step to follow up.

  • The agent did not directly contact or escalate with the courier on the customer's behalf.
  • The explanation of the delay was somewhat general and not very definitive.
Pipeline time & cost (STT + analysis) · 24.8s · $0.0237
PhaseLatencyFinancial cost
Speech-to-textElevenLabs Scribe v2
18.6stime
$0.0117audio_minutes
AnalysisGPT-5.5
6.2stime
$0.0120tokens
Pipeline total24.8s$0.0237

Existing transcript

sample-05-received
329 wordsSTT 0.0sSTT $0.0000
Gemma 4 26B
Tone 5 · Clarity 5 · Efficiency 5
valid · fenced_json · 1 candidate(s)
outcome resolved · unable to evaluate false

The operator was very helpful and polite. They successfully identified the shipment status, explained the current delays in the courier network, and proactively offered to dictate the courier's customer service number to the customer to help them track the package.

Pipeline time & cost (STT + analysis) · 23.6s · $0.0082
PhaseLatencyFinancial cost
Speech-to-textExisting transcript
0.0stime
n/aunpriced
AnalysisGemma 4 26B
23.6stime
$0.0082runtime_hours
Pipeline total23.6s$0.0082
GPT-5.5
Tone 5 · Clarity 4 · Efficiency 4
valid · direct_json · 1 candidate(s)
outcome partial · unable to evaluate false

The operator was polite and helpful, checked the order status, explained that the shipment appeared to be in transit, and provided the courier customer service phone number. The delivery issue itself was not resolved during the call, but the customer received next-step information.

  • The operator used some speculative language about the reason for delays instead of providing a confirmed cause.
  • No proactive escalation or follow-up with the courier was offered.
Pipeline time & cost (STT + analysis) · 6.5s · $0.0136
PhaseLatencyFinancial cost
Speech-to-textExisting transcript
0.0stime
n/aunpriced
AnalysisGPT-5.5
6.5stime
$0.0136tokens
Pipeline total6.5s$0.0136

Sample 06

5 speech pipelines · 10 results

MLX Whisper large-v3

sample-06-mlx-known
134 wordsSTT 8.5sSTT $0.0000
Gemma 4 26B
Tone 5 · Clarity 5 · Efficiency 5
valid · fenced_json · 1 candidate(s)
outcome resolved · unable to evaluate false

The agent was polite and provided a clear update regarding the shipment's current location (in transit from Poland) and the next steps for notification (email alert upon arrival in Hungary).

Pipeline time & cost (STT + analysis) · 33.3s · $0.0086
PhaseLatencyFinancial cost
Speech-to-textMLX Whisper large-v3
8.5stime
$0.0000runtime_hours
AnalysisGemma 4 26B
24.8stime
$0.0086runtime_hours
Pipeline total33.3s$0.0086
GPT-5.5
Tone 5 · Clarity 4 · Efficiency 5
valid · direct_json · 1 candidate(s)
outcome partial · unable to evaluate false

The agent was polite, quickly located the order, and explained the current shipping status and expected timeframe. However, no firm delivery date could be confirmed, so the customer's question was only partially resolved.

  • No exact delivery date was provided.
Pipeline time & cost (STT + analysis) · 14.5s · $0.0087
PhaseLatencyFinancial cost
Speech-to-textMLX Whisper large-v3
8.5stime
$0.0000runtime_hours
AnalysisGPT-5.5
6.0stime
$0.0087tokens
Pipeline total14.5s$0.0087

OpenAI gpt-4o-transcribe

sample-06-openai-full
135 wordsSTT 4.1sSTT $0.0072
Gemma 4 26B
Tone 5 · Clarity 5 · Efficiency 5
valid · fenced_json · 1 candidate(s)
outcome resolved · unable to evaluate false

The agent was polite and professional. They clearly explained the current status of the shipment (departed from Poland, not yet in Hungary) and provided a realistic expectation for the delivery timeline, while also managing expectations regarding potential courier delays.

Pipeline time & cost (STT + analysis) · 27.4s · $0.0152
PhaseLatencyFinancial cost
Speech-to-textOpenAI gpt-4o-transcribe
4.1stime
$0.0072audio_minutes
AnalysisGemma 4 26B
23.3stime
$0.0081runtime_hours
Pipeline total27.4s$0.0152
GPT-5.5
Tone 5 · Clarity 5 · Efficiency 5
valid · direct_json · 1 candidate(s)
outcome resolved · unable to evaluate false

The agent was polite, quickly verified the order, provided the current shipment status, explained the expected timeframe, and clarified that an email notification will be sent once the courier receives the package in Hungary.

Pipeline time & cost (STT + analysis) · 9.1s · $0.0151
PhaseLatencyFinancial cost
Speech-to-textOpenAI gpt-4o-transcribe
4.1stime
$0.0072audio_minutes
AnalysisGPT-5.5
5.0stime
$0.0080tokens
Pipeline total9.1s$0.0151

OpenAI gpt-4o-mini-transcribe

sample-06-openai-mini
124 wordsSTT 3.2sSTT $0.0036
Gemma 4 26B
Tone 5 · Clarity 5 · Efficiency 5
valid · fenced_json · 1 candidate(s)
outcome partial · unable to evaluate false

The agent provided clear information regarding the shipment's current location (leaving Poland) and the notification process. However, the customer's specific request for a Friday delivery could not be confirmed due to potential courier delays, resulting in a partial resolution.

Pipeline time & cost (STT + analysis) · 28.6s · $0.0124
PhaseLatencyFinancial cost
Speech-to-textOpenAI gpt-4o-mini-transcribe
3.2stime
$0.0036audio_minutes
AnalysisGemma 4 26B
25.5stime
$0.0088runtime_hours
Pipeline total28.6s$0.0124
GPT-5.5
Tone 5 · Clarity 4 · Efficiency 5
valid · direct_json · 1 candidate(s)
outcome partial · unable to evaluate false

The agent was polite and concise, provided the shipment status and an estimated timeframe, but could not confirm an exact delivery date.

  • Exact delivery date was not confirmed.
Pipeline time & cost (STT + analysis) · 7.8s · $0.0123
PhaseLatencyFinancial cost
Speech-to-textOpenAI gpt-4o-mini-transcribe
3.2stime
$0.0036audio_minutes
AnalysisGPT-5.5
4.6stime
$0.0087tokens
Pipeline total7.8s$0.0123

ElevenLabs Scribe v2

sample-06-elevenlabs
147 wordsSTT 8.9sSTT $0.0044
Gemma 4 26B
Tone 5 · Clarity 5 · Efficiency 5
valid · fenced_json · 1 candidate(s)
outcome resolved · unable to evaluate false

The agent was polite and provided a clear, detailed update regarding the shipment's current status, explaining that it has left Poland and is expected in Hungary within a few business days. The agent also correctly set expectations regarding potential courier delays and the notification process.

Pipeline time & cost (STT + analysis) · 32.8s · $0.0127
PhaseLatencyFinancial cost
Speech-to-textElevenLabs Scribe v2
8.9stime
$0.0044audio_minutes
AnalysisGemma 4 26B
23.9stime
$0.0083runtime_hours
Pipeline total32.8s$0.0127
GPT-5.5
Tone 5 · Clarity 5 · Efficiency 5
valid · direct_json · 1 candidate(s)
outcome resolved · unable to evaluate false

The agent was polite, quickly identified the order, provided the current shipping status, gave a realistic estimated timeframe, and explained that the customer would receive an email notification once the package reaches the local courier.

Pipeline time & cost (STT + analysis) · 14.9s · $0.0133
PhaseLatencyFinancial cost
Speech-to-textElevenLabs Scribe v2
8.9stime
$0.0044audio_minutes
AnalysisGPT-5.5
6.0stime
$0.0089tokens
Pipeline total14.9s$0.0133

Existing transcript

sample-06-received
151 wordsSTT 0.0sSTT $0.0000
Gemma 4 26B
Tone 5 · Clarity 5 · Efficiency 5
valid · fenced_json · 1 candidate(s)
outcome resolved · unable to evaluate false

The operator was polite and provided a clear, detailed update regarding the shipment's current location (having left Poland) and the expected timeline for arrival in Hungary.

Pipeline time & cost (STT + analysis) · 24.9s · $0.0086
PhaseLatencyFinancial cost
Speech-to-textExisting transcript
0.0stime
n/aunpriced
AnalysisGemma 4 26B
24.9stime
$0.0086runtime_hours
Pipeline total24.9s$0.0086
GPT-5.5
Tone 5 · Clarity 5 · Efficiency 5
valid · direct_json · 1 candidate(s)
outcome partial · unable to evaluate false

The operator was polite, clear, and efficient. They checked the order status, explained that the package had left Poland but had not yet arrived in Hungary, and gave an estimated timeframe while noting that Friday delivery could not be guaranteed.

  • No guaranteed delivery date was available.
Pipeline time & cost (STT + analysis) · 6.6s · $0.0102
PhaseLatencyFinancial cost
Speech-to-textExisting transcript
0.0stime
n/aunpriced
AnalysisGPT-5.5
6.6stime
$0.0102tokens
Pipeline total6.6s$0.0102

Sample 07

5 speech pipelines · 10 results

MLX Whisper large-v3

sample-07-mlx-known
1094 wordsSTT 42.1sSTT $0.0000
Gemma 4 26B
Tone 5 · Clarity 5 · Efficiency 5
valid · fenced_json · 1 candidate(s)
outcome resolved · unable to evaluate false

The agent was very professional and addressed all of the customer's concerns, including delivery timelines, return policies, dosage instructions, and the distinction between recurring and one-time orders. The agent also provided a clear solution for the customer's fear of automatic renewals by explaining how to contact them to pause shipments.

Pipeline time & cost (STT + analysis) · 64.9s · $0.0079
PhaseLatencyFinancial cost
Speech-to-textMLX Whisper large-v3
42.1stime
$0.0000runtime_hours
AnalysisGemma 4 26B
22.8stime
$0.0079runtime_hours
Pipeline total64.9s$0.0079
GPT-5.5
Tone 5 · Clarity 4 · Efficiency 4
valid · direct_json · 1 candidate(s)
outcome partial · unable to evaluate false

The agent was polite, patient, and answered the customer's questions about delivery, administration, returns, ingredients, payment, and recurring orders. The explanation of recurring versus non-recurring order options caused some confusion and could have been clearer. No order was placed during the call; the customer planned to order on the website.

  • Recurring order terms and cancellation process were not explained clearly enough at first.
  • The customer initially wanted to place an order, but the call ended without completing it.
  • Some product guidance was general and appropriately redirected to a doctor for treatment-related concerns.
Pipeline time & cost (STT + analysis) · 49.7s · $0.0201
PhaseLatencyFinancial cost
Speech-to-textMLX Whisper large-v3
42.1stime
$0.0000runtime_hours
AnalysisGPT-5.5
7.6stime
$0.0201tokens
Pipeline total49.7s$0.0201

OpenAI gpt-4o-transcribe

sample-07-openai-full
989 wordsSTT 25.3sSTT $0.0549
Gemma 4 26B
Tone 5 · Clarity 5 · Efficiency 5
valid · fenced_json · 1 candidate(s)
outcome resolved · unable to evaluate false

The agent was very professional, polite, and addressed all of the customer's concerns. Specifically, the agent provided a clear solution to the customer's fear of recurring/automatic orders by explaining how to pause shipments after the first delivery. The agent also clearly explained the differences in pricing between the two packages and provided necessary usage instructions.

Pipeline time & cost (STT + analysis) · 45.3s · $0.0619
PhaseLatencyFinancial cost
Speech-to-textOpenAI gpt-4o-transcribe
25.3stime
$0.0549audio_minutes
AnalysisGemma 4 26B
19.9stime
$0.0069runtime_hours
Pipeline total45.3s$0.0619
GPT-5.5
Tone 5 · Clarity 4 · Efficiency 4
valid · direct_json · 1 candidate(s)
outcome partial · unable to evaluate false

The agent was polite and helpful, answered questions about pricing, recurring orders, delivery time, returns, administration, ingredients, and payment. However, the customer remained somewhat confused about the recurring versus one-time order options and no order was placed during the call.

  • Recurring order terms and cancellation process could have been explained more clearly and proactively.
  • The agent did not fully confirm the customer's final intended order path before ending the call.
Pipeline time & cost (STT + analysis) · 32.2s · $0.0723
PhaseLatencyFinancial cost
Speech-to-textOpenAI gpt-4o-transcribe
25.3stime
$0.0549audio_minutes
AnalysisGPT-5.5
6.9stime
$0.0174tokens
Pipeline total32.2s$0.0723

OpenAI gpt-4o-mini-transcribe

sample-07-openai-mini
1103 wordsSTT 16.1sSTT $0.0275
Gemma 4 26B
Tone 5 · Clarity 5 · Efficiency 5
valid · fenced_json · 1 candidate(s)
outcome partial · unable to evaluate false

The agent was very professional, patient, and answered all of the customer's questions thoroughly, including details about dosage, ingredients, delivery timelines, and return policies. The agent also successfully addressed the customer's concern regarding recurring orders by explaining how to pause future shipments. The call is marked as partial because while all inquiries were resolved, the customer did not complete the order during the call, opting to place it via the website and call back later.

Pipeline time & cost (STT + analysis) · 41.0s · $0.0361
PhaseLatencyFinancial cost
Speech-to-textOpenAI gpt-4o-mini-transcribe
16.1stime
$0.0275audio_minutes
AnalysisGemma 4 26B
24.9stime
$0.0087runtime_hours
Pipeline total41.0s$0.0361
GPT-5.5
Tone 4 · Clarity 3 · Efficiency 3
valid · direct_json · 1 candidate(s)
outcome partial · unable to evaluate false

The agent was polite and generally answered the customer's questions about delivery, returns, dosage, ingredients, and recurring orders. However, the explanation of recurring versus one-time purchase options was somewhat confusing and repetitive, and the customer's original intent to place an order was not completed during the call.

  • No order was actually placed despite the customer initially wanting to order.
  • Recurring order terms and cancellation process were not explained as clearly as they could have been.
  • The agent repeatedly suggested contacting support after delivery to stop future shipments, which may create unnecessary friction for the customer.
  • Some product and medical guidance was given, but the agent appropriately advised consulting a doctor for existing treatments or chronic conditions.
Pipeline time & cost (STT + analysis) · 24.6s · $0.0498
PhaseLatencyFinancial cost
Speech-to-textOpenAI gpt-4o-mini-transcribe
16.1stime
$0.0275audio_minutes
AnalysisGPT-5.5
8.5stime
$0.0224tokens
Pipeline total24.6s$0.0498

ElevenLabs Scribe v2

sample-07-elevenlabs
1173 wordsSTT 47.0sSTT $0.0336
Gemma 4 26B
Tone 5 · Clarity 5 · Efficiency 5
valid · fenced_json · 1 candidate(s)
outcome resolved · unable to evaluate false

The agent was very professional, patient, and polite. She successfully addressed all of the customer's concerns, including delivery timelines, usage instructions, return policies, the difference between recurring and one-time orders, and potential side effects. The agent also provided clear guidance on how to manage the subscription to avoid unwanted future charges.

Pipeline time & cost (STT + analysis) · 70.4s · $0.0417
PhaseLatencyFinancial cost
Speech-to-textElevenLabs Scribe v2
47.0stime
$0.0336audio_minutes
AnalysisGemma 4 26B
23.5stime
$0.0082runtime_hours
Pipeline total70.4s$0.0417
GPT-5.5
Tone 5 · Clarity 4 · Efficiency 4
valid · direct_json · 1 candidate(s)
outcome partial · unable to evaluate false

The agent was polite, patient, and answered most of the customer's questions clearly. The customer did not place the order during the call and planned to order online, so the main sales outcome was only partially completed.

  • Some explanation around recurring versus one-time packages was initially confusing and required repeated clarification.
  • The agent gave product usage and suitability guidance that bordered on medical advice, including comments about long-term use and taking the capsule after it dissolves in the mouth.
  • The agent appropriately advised the customer to consult a doctor regarding other treatments or chronic conditions.
Pipeline time & cost (STT + analysis) · 56.5s · $0.0568
PhaseLatencyFinancial cost
Speech-to-textElevenLabs Scribe v2
47.0stime
$0.0336audio_minutes
AnalysisGPT-5.5
9.5stime
$0.0232tokens
Pipeline total56.5s$0.0568

Existing transcript

sample-07-received
1066 wordsSTT 0.0sSTT $0.0000
Gemma 4 26B
Tone 5 · Clarity 5 · Efficiency 5
valid · fenced_json · 1 candidate(s)
outcome resolved · unable to evaluate false

The operator provided clear and comprehensive answers to all the customer's questions, including delivery timelines, return policies, dosage instructions, and the distinction between the recurring and one-time purchase offers. The operator also proactively addressed the customer's concern regarding automatic renewals by suggesting a way to pause future shipments.

Pipeline time & cost (STT + analysis) · 25.2s · $0.0087
PhaseLatencyFinancial cost
Speech-to-textExisting transcript
0.0stime
n/aunpriced
AnalysisGemma 4 26B
25.2stime
$0.0087runtime_hours
Pipeline total25.2s$0.0087
GPT-5.5
Tone 5 · Clarity 4 · Efficiency 4
valid · direct_json · 1 candidate(s)
outcome partial · unable to evaluate false

The operator was polite, patient, and answered the customer's questions about pricing, delivery, returns, administration, and recurring orders. Clarity was generally good, though the recurring versus one-time order options required several explanations before the customer fully understood. No order was placed during the call, but the customer received enough information to order on the website.

  • The recurring order terms could have been explained more clearly and proactively at the start.
  • The customer initially intended to place an order, but the call ended without the operator completing it.
  • The operator gave general supplement guidance appropriately, but some health-related questions would be better redirected more firmly to a medical professional.
Pipeline time & cost (STT + analysis) · 8.4s · $0.0201
PhaseLatencyFinancial cost
Speech-to-textExisting transcript
0.0stime
n/aunpriced
AnalysisGPT-5.5
8.4stime
$0.0201tokens
Pipeline total8.4s$0.0201

Sample 08

5 speech pipelines · 10 results

MLX Whisper large-v3

sample-08-mlx-known
840 wordsSTT 31.3sSTT $0.0000
Gemma 4 26B
Tone 4 · Clarity 4 · Efficiency 4
valid · fenced_json · 1 candidate(s)
outcome resolved · unable to evaluate false

The agent handled a frustrated customer effectively. Despite the customer's heavy criticism regarding the website's usability and issues with a previous cancelled order, the agent successfully navigated the conversation to modify the subscription/order details and payment method to the customer's satisfaction. There was a brief mention of audio clarity issues, but the agent proactively repeated the total price when requested.

  • Customer reported significant difficulty navigating the website/UI, describing it as poorly designed.
  • Customer complained about a previous cancelled order causing friction when trying to place new orders.
  • Potential intermittent audio or connection issues during the call.
Pipeline time & cost (STT + analysis) · 54.7s · $0.0081
PhaseLatencyFinancial cost
Speech-to-textMLX Whisper large-v3
31.3stime
$0.0000runtime_hours
AnalysisGemma 4 26B
23.5stime
$0.0081runtime_hours
Pipeline total54.7s$0.0081
GPT-5.5
Tone 4 · Clarity 4 · Efficiency 4
valid · direct_json · 1 candidate(s)
outcome resolved · unable to evaluate false

The agent remained polite and professional, apologized for the inconvenience, explained the subscription options and pricing, confirmed the address and payment method, and placed the requested order. The issue was resolved, though the customer's complaints about the website experience were only briefly acknowledged and not fully explored.

  • Customer reported confusion caused by a previously canceled order appearing in the account.
  • Customer complained that the website is difficult to navigate and does not clearly show products.
  • Customer mentioned intermittent audio/hearing issues during the call.
  • Agent addressed the order request effectively but did not provide much support or guidance regarding the website usability concerns.
Pipeline time & cost (STT + analysis) · 39.0s · $0.0162
PhaseLatencyFinancial cost
Speech-to-textMLX Whisper large-v3
31.3stime
$0.0000runtime_hours
AnalysisGPT-5.5
7.7stime
$0.0162tokens
Pipeline total39.0s$0.0162

OpenAI gpt-4o-transcribe

sample-08-openai-full
1246 wordsSTT 24.3sSTT $0.0379
Gemma 4 26B
Tone 3 · Clarity 3 · Efficiency 2
valid · fenced_json · 1 candidate(s)
outcome unresolved · unable to evaluate false

The customer is highly frustrated with the website's user interface and a technical error where a previously cancelled order is preventing them from placing a new order. While the agent is polite and explains the subscription models clearly, they fail to address the primary technical issue (the blocked order) and instead focus on explaining subscription terms and upselling Vitamin C.

  • Website usability and navigation issues reported by the customer
  • Technical error: A cancelled order is preventing the customer from completing a new purchase
  • Agent failed to provide a solution for the technical blockage, focusing instead on subscription details and upselling
Pipeline time & cost (STT + analysis) · 49.7s · $0.0467
PhaseLatencyFinancial cost
Speech-to-textOpenAI gpt-4o-transcribe
24.3stime
$0.0379audio_minutes
AnalysisGemma 4 26B
25.4stime
$0.0088runtime_hours
Pipeline total49.7s$0.0467
GPT-5.5
Tone 3 · Clarity 2 · Efficiency 2
valid · direct_json · 1 candidate(s)
outcome unresolved · unable to evaluate false

The agent remained generally polite and apologized for the inconvenience, but did not clearly resolve the customer's main problem with the canceled order blocking a new purchase. The explanation about subscription options and pricing was somewhat clear, but the customer remained confused about whether it was an order or a subscription. The call was inefficient, with repeated information and no clear next step or confirmed resolution.

  • Agent did not adequately address the customer's issue with the canceled order appearing in the account.
  • Agent focused on subscription pricing rather than troubleshooting the website or account problem.
  • Customer confusion about order versus subscription was not fully resolved.
  • No clear resolution, escalation, or confirmed action was provided.
  • Call contained significant repetition and lacked efficient progression.
Pipeline time & cost (STT + analysis) · 32.4s · $0.0596
PhaseLatencyFinancial cost
Speech-to-textOpenAI gpt-4o-transcribe
24.3stime
$0.0379audio_minutes
AnalysisGPT-5.5
8.2stime
$0.0217tokens
Pipeline total32.4s$0.0596

OpenAI gpt-4o-mini-transcribe

sample-08-openai-mini
453 wordsSTT 8.9sSTT $0.0189
Gemma 4 26B
Tone 3 · Clarity 4 · Efficiency 4
valid · fenced_json · 1 candidate(s)
outcome resolved · unable to evaluate false

The customer was very frustrated with the website's user interface and a previous issue involving a canceled order and unexpected charges. The agent successfully bypassed the technical difficulties by manually processing a new subscription/order, which satisfied the customer's immediate need to purchase, though the underlying website complaints were not technically addressed.

  • Customer expressed significant dissatisfaction with the website's usability and design
  • Customer reported a previous error where a canceled order appeared to demand payment
  • Customer mentioned audio/connection issues during the call
  • The agent focused on completing the transaction rather than addressing the technical website issues
Pipeline time & cost (STT + analysis) · 30.9s · $0.0266
PhaseLatencyFinancial cost
Speech-to-textOpenAI gpt-4o-mini-transcribe
8.9stime
$0.0189audio_minutes
AnalysisGemma 4 26B
22.0stime
$0.0077runtime_hours
Pipeline total30.9s$0.0266
GPT-5.5
Tone 4 · Clarity 3 · Efficiency 3
valid · direct_json · 1 candidate(s)
outcome partial · unable to evaluate false

The agent remained polite and apologized for the inconvenience, and appears to have helped process or modify the customer's order. However, the explanation was somewhat unclear, with inconsistent pricing mentioned and the customer's website-related concerns not meaningfully addressed. Audio interruptions also affected understanding.

  • Pricing and product details were not clearly explained
  • Customer reported audio interruptions
  • Website usability complaint was acknowledged but not addressed
  • Resolution was mainly focused on placing/modifying the order, not on the broader account or site issue
Pipeline time & cost (STT + analysis) · 18.0s · $0.0363
PhaseLatencyFinancial cost
Speech-to-textOpenAI gpt-4o-mini-transcribe
8.9stime
$0.0189audio_minutes
AnalysisGPT-5.5
9.1stime
$0.0173tokens
Pipeline total18.0s$0.0363

ElevenLabs Scribe v2

sample-08-elevenlabs
922 wordsSTT 47.7sSTT $0.0232
Gemma 4 26B
Tone 4 · Clarity 4 · Efficiency 5
valid · fenced_json · 1 candidate(s)
outcome resolved · unable to evaluate false

The agent successfully resolved the customer's request by manually processing the order and modifying the subscription, effectively bypassing the website usability issues the customer was complaining about. Despite the customer's harsh criticism of the website and frustration regarding a previous canceled order, the agent remained professional and polite. There were some noted audio interruptions during the conversation.

  • Customer expressed significant dissatisfaction with the website's user interface and usability.
  • Customer reported a previous issue with a canceled order and a lost payment.
  • Intermittent audio quality issues were noted during the call.
Pipeline time & cost (STT + analysis) · 69.5s · $0.0307
PhaseLatencyFinancial cost
Speech-to-textElevenLabs Scribe v2
47.7stime
$0.0232audio_minutes
AnalysisGemma 4 26B
21.8stime
$0.0076runtime_hours
Pipeline total69.5s$0.0307
GPT-5.5
Tone 5 · Clarity 4 · Efficiency 4
valid · direct_json · 1 candidate(s)
outcome resolved · unable to evaluate false

The agent remained polite and patient, apologized for the inconvenience, clarified the subscription options and pricing, confirmed the delivery address and payment method, and successfully placed the order. Some customer concerns about the website and prior canceled order/payment confusion were not fully explored, but the main request was completed.

  • Customer's concerns about the confusing website were acknowledged but not meaningfully addressed.
  • The prior canceled order and lost payment concern was not fully investigated during the call.
Pipeline time & cost (STT + analysis) · 56.5s · $0.0400
PhaseLatencyFinancial cost
Speech-to-textElevenLabs Scribe v2
47.7stime
$0.0232audio_minutes
AnalysisGPT-5.5
8.8stime
$0.0168tokens
Pipeline total56.5s$0.0400

Existing transcript

sample-08-received
531 wordsSTT 0.0sSTT $0.0000
Gemma 4 26B
Tone 5 · Clarity 4 · Efficiency 5
valid · fenced_json · 1 candidate(s)
outcome resolved · unable to evaluate false

The operator was very professional, polite, and successfully resolved the customer's issue by manually updating the subscription and payment method, bypassing the website issues the customer was experiencing. The only drawback was a technical issue where the customer reported audio interruptions.

  • Audio/connection issues (customer reported hearing the operator interrupted/cutting out)
  • Customer reported difficulty navigating the website
Pipeline time & cost (STT + analysis) · 22.7s · $0.0079
PhaseLatencyFinancial cost
Speech-to-textExisting transcript
0.0stime
n/aunpriced
AnalysisGemma 4 26B
22.7stime
$0.0079runtime_hours
Pipeline total22.7s$0.0079
GPT-5.5
Tone 5 · Clarity 4 · Efficiency 4
valid · direct_json · 1 candidate(s)
outcome resolved · unable to evaluate false

The operator was polite, apologetic, and ultimately registered the customer's order with confirmed price, address, payment method, and delivery estimate. Clarity was generally good, though the customer reported hearing the operator intermittently and some subscription/pricing explanations were somewhat complex. Efficiency was good, but there was a brief irrelevant product recommendation before focusing on the customer's requested collagen subscription.

  • Customer reported interrupted audio and asked the operator to repeat information.
  • Operator introduced an unrelated vitamin C product, which did not directly address the customer's main request.
  • Subscription and pricing explanation may have been somewhat confusing for the customer.
Pipeline time & cost (STT + analysis) · 7.3s · $0.0133
PhaseLatencyFinancial cost
Speech-to-textExisting transcript
0.0stime
n/aunpriced
AnalysisGPT-5.5
7.3stime
$0.0133tokens
Pipeline total7.3s$0.0133

Sample 09

5 speech pipelines · 10 results

MLX Whisper large-v3

sample-09-mlx-known
491 wordsSTT 19.2sSTT $0.0000
Gemma 4 26B
Tone 5 · Clarity 5 · Efficiency 5
valid · fenced_json · 1 candidate(s)
outcome resolved · unable to evaluate false

The agent handled the complaint professionally and empathetically. They verified the delivery details, apologized for the inconvenience, and provided a clear solution by offering to resend the order and promising to escalate the issue with the courier service to the logistics department.

  • Customer reported a lack of response to previous email inquiries
  • Customer experienced issues with the tracking code provided by the courier
Pipeline time & cost (STT + analysis) · 39.4s · $0.0070
PhaseLatencyFinancial cost
Speech-to-textMLX Whisper large-v3
19.2stime
$0.0000runtime_hours
AnalysisGemma 4 26B
20.2stime
$0.0070runtime_hours
Pipeline total39.4s$0.0070
GPT-5.5
Tone 5 · Clarity 4 · Efficiency 4
valid · direct_json · 1 candidate(s)
outcome partial · unable to evaluate false

The agent was polite, apologetic, verified the order/address details, explained that the parcel was with the courier, and arranged for the order to be resent with an urgency note. The customer's immediate request was handled, but final delivery remains pending.

  • The original courier failure was not fully resolved during the call.
  • The agent did not provide a specific new tracking number or exact delivery date.
Pipeline time & cost (STT + analysis) · 24.5s · $0.0118
PhaseLatencyFinancial cost
Speech-to-textMLX Whisper large-v3
19.2stime
$0.0000runtime_hours
AnalysisGPT-5.5
5.4stime
$0.0118tokens
Pipeline total24.5s$0.0118

OpenAI gpt-4o-transcribe

sample-09-openai-full
534 wordsSTT 12.9sSTT $0.0235
Gemma 4 26B
Tone 5 · Clarity 5 · Efficiency 5
valid · fenced_json · 1 candidate(s)
outcome resolved · unable to evaluate false

The agent handled the customer's complaint regarding a failed delivery professionally. They apologized for the inconvenience, verified the details, and provided a clear resolution by offering to resend the order and promising to escalate the issue with the logistics team and the courier company. The agent also managed expectations regarding the new delivery timeline.

  • Courier ([courier redacted]) failed to contact the customer despite correct contact information being provided
  • The tracking/AVB code provided by the courier was reported as invalid by the customer
  • Customer experienced a lack of communication from the courier service
Pipeline time & cost (STT + analysis) · 33.3s · $0.0306
PhaseLatencyFinancial cost
Speech-to-textOpenAI gpt-4o-transcribe
12.9stime
$0.0235audio_minutes
AnalysisGemma 4 26B
20.4stime
$0.0071runtime_hours
Pipeline total33.3s$0.0306
GPT-5.5
Tone 5 · Clarity 4 · Efficiency 4
valid · direct_json · 1 candidate(s)
outcome partial · unable to evaluate false

The agent was polite, apologized for the inconvenience, confirmed the order status, explained that the parcel was with the courier, and offered to resend the order with an expedited note. The issue was not fully resolved during the call because delivery still depends on the new shipment and courier follow-up.

  • The customer reported no phone call or message from the courier despite the address and phone number being correct.
  • The courier tracking/AWB information appeared not to work for the customer.
  • The agent did not provide a new tracking number or a firm confirmation that the reshipment had already been processed during the call.
Pipeline time & cost (STT + analysis) · 19.3s · $0.0373
PhaseLatencyFinancial cost
Speech-to-textOpenAI gpt-4o-transcribe
12.9stime
$0.0235audio_minutes
AnalysisGPT-5.5
6.5stime
$0.0138tokens
Pipeline total19.3s$0.0373

OpenAI gpt-4o-mini-transcribe

sample-09-openai-mini
372 wordsSTT 5.3sSTT $0.0118
Gemma 4 26B
Tone 5 · Clarity 5 · Efficiency 5
valid · fenced_json · 1 candidate(s)
outcome resolved · unable to evaluate false

The agent handled a frustrated customer effectively. The agent acknowledged the delivery failure, identified the package status, and provided a clear, actionable solution by promising to resend the package and escalate the issue with the courier to the logistics department.

Pipeline time & cost (STT + analysis) · 54.4s · $0.0288
PhaseLatencyFinancial cost
Speech-to-textOpenAI gpt-4o-mini-transcribe
5.3stime
$0.0118audio_minutes
AnalysisGemma 4 26B
49.1stime
$0.0171runtime_hours
Pipeline total54.4s$0.0288
GPT-5.5
Tone 5 · Clarity 4 · Efficiency 4
valid · direct_json · 1 candidate(s)
outcome partial · unable to evaluate false

The agent was polite and empathetic, checked the order status, explained that the parcel was with the courier, and initiated a re-shipment with urgency and a request for logistics to contact the courier. The issue was not fully resolved during the call because the customer still had to wait for delivery.

  • No specific delivery timeframe was provided.
  • The agent did not clearly confirm the customer's phone number or address during the call.
  • Resolution depended on follow-up actions by logistics and the courier.
Pipeline time & cost (STT + analysis) · 10.7s · $0.0230
PhaseLatencyFinancial cost
Speech-to-textOpenAI gpt-4o-mini-transcribe
5.3stime
$0.0118audio_minutes
AnalysisGPT-5.5
5.4stime
$0.0112tokens
Pipeline total10.7s$0.0230

ElevenLabs Scribe v2

sample-09-elevenlabs
540 wordsSTT 24.0sSTT $0.0144
Gemma 4 26B
Tone 5 · Clarity 5 · Efficiency 5
valid · fenced_json · 1 candidate(s)
outcome resolved · unable to evaluate false

The agent handled the complaint professionally and empathetically. They successfully identified the issue with the courier service and provided a clear solution by offering to resend the order and promising to escalate the matter to the logistics team to ensure correct delivery.

  • Customer reported a lack of response to previous email inquiries regarding the delivery issue.
  • Customer experienced technical difficulties with the courier's tracking/AVB code system.
Pipeline time & cost (STT + analysis) · 44.7s · $0.0216
PhaseLatencyFinancial cost
Speech-to-textElevenLabs Scribe v2
24.0stime
$0.0144audio_minutes
AnalysisGemma 4 26B
20.8stime
$0.0072runtime_hours
Pipeline total44.7s$0.0216
GPT-5.5
Tone 5 · Clarity 5 · Efficiency 4
valid · direct_json · 1 candidate(s)
outcome partial · unable to evaluate false

The agent was polite, empathetic, apologized for the inconvenience, verified the address, explained that the parcel was with the courier, and offered to resend the order with expedited handling. The customer accepted the proposed solution, but final delivery was still pending.

  • Courier reportedly did not contact the customer or successfully locate the address.
  • Customer stated previous emails did not receive a useful response.
  • Customer reported that the courier tracking/reference code did not work.
  • Agent did not provide a confirmation number or exact follow-up details for the reshipment.
Pipeline time & cost (STT + analysis) · 31.4s · $0.0299
PhaseLatencyFinancial cost
Speech-to-textElevenLabs Scribe v2
24.0stime
$0.0144audio_minutes
AnalysisGPT-5.5
7.4stime
$0.0155tokens
Pipeline total31.4s$0.0299

Existing transcript

sample-09-received
493 wordsSTT 0.0sSTT $0.0000
Gemma 4 26B
Tone 5 · Clarity 5 · Efficiency 5
valid · fenced_json · 1 candidate(s)
outcome resolved · unable to evaluate false

The operator handled the customer's complaint regarding a failed delivery professionally. They apologized for the inconvenience, verified the delivery details, and provided a clear resolution by offering to resend the order with an urgency note and escalating the issue with the courier company.

  • The courier ([courier redacted]) failed to contact the customer or locate the address despite the information being correct in the system.
  • The customer reported that the AWB number provided by the courier was invalid/not found in the system.
Pipeline time & cost (STT + analysis) · 28.7s · $0.0100
PhaseLatencyFinancial cost
Speech-to-textExisting transcript
0.0stime
n/aunpriced
AnalysisGemma 4 26B
28.7stime
$0.0100runtime_hours
Pipeline total28.7s$0.0100
GPT-5.5
Tone 5 · Clarity 5 · Efficiency 4
valid · direct_json · 1 candidate(s)
outcome partial · unable to evaluate false

The operator was polite, apologetic, verified the order details, explained that the address appeared complete, and offered to resend the package with an urgency note. The issue was not fully resolved during the call because delivery still needs to be completed, but a clear next step was agreed upon.

  • Courier reportedly did not contact the customer and claimed delivery/address issues.
  • Customer's prior emails did not receive a reliable response.
  • Customer reported AWB/tracking information did not work.
  • Resolution depends on a new shipment and future delivery.
Pipeline time & cost (STT + analysis) · 6.4s · $0.0131
PhaseLatencyFinancial cost
Speech-to-textExisting transcript
0.0stime
n/aunpriced
AnalysisGPT-5.5
6.4stime
$0.0131tokens
Pipeline total6.4s$0.0131