CallsQA Experiment Dashboard

CallsQA six-call STT and evaluator matrix · run 20260710T103602640680Z
6unique calls
24speech pipelines
48end-to-end results
0execution errors
$0.8580batch known cost
1149.262ssummed step time
Group by

HU 1

4 speech pipelines · 8 results

MLX Whisper large-v3

hu-1-mlx
83 wordsSTT 8.1sSTT $0.0000
Gemma 4 26B
Tone 3 · Clarity 3 · Efficiency 2
valid · fenced_json · 1 candidate(s)
outcome unresolved · unable to evaluate false

The agent failed to acknowledge the customer's specific complaint regarding the incorrect product and the replacement request, immediately jumping to requesting the order number. The transcript ends abruptly without addressing the core issue.

  • Lack of empathy/acknowledgment of the customer's problem.
  • The conversation ends without a resolution to the customer's inquiry regarding the product replacement.
Phase time & cost · 42.3s · $0.0119
PhaseLatencyFinancial cost
Speech-to-textMLX Whisper large-v3
8.1stime
$0.0000runtime_hours
AnalysisGemma 4 26B
34.1stime
$0.0119runtime_hours
End-to-end42.3s$0.0119
GPT-5.5
Tone 0 · Clarity 0 · Efficiency 0
valid · direct_json · 1 candidate(s)
outcome unknown · unable to evaluate true

The transcript is incomplete and ends before the agent can investigate or provide a resolution. There is not enough information to reliably evaluate the full call quality or outcome.

  • Transcript appears truncated
  • No final resolution or next steps are captured
Phase time & cost · 15.8s · $0.0089
PhaseLatencyFinancial cost
Speech-to-textMLX Whisper large-v3
8.1stime
$0.0000runtime_hours
AnalysisGPT-5.5
7.7stime
$0.0089tokens
End-to-end15.8s$0.0089

OpenAI gpt-4o-transcribe

hu-1-openai-full
100 wordsSTT 4.1sSTT $0.0065
Gemma 4 26B
Tone 1 · Clarity 2 · Efficiency 2
valid · fenced_json · 1 candidate(s)
outcome unresolved · unable to evaluate false

The agent was unprofessional and dismissive at the end of the call, responding with 'No, it's not true, it's not true' when the customer realized they had the wrong information. Furthermore, the agent failed to address the customer's actual complaint regarding the product size discrepancy because the call derailed due to the incorrect order number.

  • Unprofessional and dismissive attitude
  • Failure to address the customer's primary concern
Phase time & cost · 32.4s · $0.0163
PhaseLatencyFinancial cost
Speech-to-textOpenAI gpt-4o-transcribe
4.1stime
$0.0065audio_minutes
AnalysisGemma 4 26B
28.4stime
$0.0098runtime_hours
End-to-end32.4s$0.0163
GPT-5.5
Tone 3 · Clarity 3 · Efficiency 2
valid · direct_json · 1 candidate(s)
outcome unresolved · unable to evaluate false

The agent greeted the customer politely and asked for an order number, but the interaction ended without addressing the customer's issue. The agent's explanation about the order number was brief and somewhat abrupt, and no next steps or alternative guidance were provided before the call ended.

  • Customer issue was not resolved
  • Agent did not provide clear next steps
  • Tone became somewhat abrupt when rejecting the provided number
  • No confirmation or follow-up path was offered
Phase time & cost · 11.0s · $0.0199
PhaseLatencyFinancial cost
Speech-to-textOpenAI gpt-4o-transcribe
4.1stime
$0.0065audio_minutes
AnalysisGPT-5.5
6.9stime
$0.0134tokens
End-to-end11.0s$0.0199

OpenAI gpt-4o-mini-transcribe

hu-1-openai-mini
98 wordsSTT 2.4sSTT $0.0032
Gemma 4 26B
Tone 5 · Clarity 4 · Efficiency 3
valid · fenced_json · 1 candidate(s)
outcome unresolved · unable to evaluate false

The customer called to address a discrepancy between the advertised product size (80ml) and the received size (50ml). However, the call ended without resolution because the customer realized they provided the wrong order number and likely called regarding a different issue.

  • Customer provided an incorrect order number
  • The original complaint regarding the product size was not addressed due to the customer's error
Phase time & cost · 25.3s · $0.0112
PhaseLatencyFinancial cost
Speech-to-textOpenAI gpt-4o-mini-transcribe
2.4stime
$0.0032audio_minutes
AnalysisGemma 4 26B
22.9stime
$0.0080runtime_hours
End-to-end25.3s$0.0112
GPT-5.5
Tone 4 · Clarity 4 · Efficiency 3
valid · direct_json · 1 candidate(s)
outcome unresolved · unable to evaluate false

The agent was polite and quickly identified that the number provided was not a valid order number for their system. However, the customer's issue was not resolved and the agent did not attempt alternative lookup options before the call ended.

  • Customer provided an invalid or incorrect order number.
  • No resolution or next steps were provided for the product size discrepancy.
  • Agent could have offered to search by another identifier such as email, name, or phone number.
Phase time & cost · 9.3s · $0.0156
PhaseLatencyFinancial cost
Speech-to-textOpenAI gpt-4o-mini-transcribe
2.4stime
$0.0032audio_minutes
AnalysisGPT-5.5
6.9stime
$0.0123tokens
End-to-end9.3s$0.0156

ElevenLabs Scribe v2

hu-1-scribe
104 wordsSTT 6.4sSTT $0.0040
Gemma 4 26B
Tone 5 · Clarity 5 · Efficiency 5
valid · fenced_json · 1 candidate(s)
outcome unresolved · unable to evaluate false

The customer called to address a discrepancy between an advertised product size (80ml) and the received size (50ml), but the call ended abruptly because the customer realized they were providing the wrong order number or had called the wrong number.

  • Customer error regarding the order number and/or contact information.
Phase time & cost · 34.8s · $0.0138
PhaseLatencyFinancial cost
Speech-to-textElevenLabs Scribe v2
6.4stime
$0.0040audio_minutes
AnalysisGemma 4 26B
28.4stime
$0.0099runtime_hours
End-to-end34.8s$0.0138
GPT-5.5
Tone 5 · Clarity 4 · Efficiency 5
valid · direct_json · 1 candidate(s)
outcome unknown · unable to evaluate false

The agent was polite and efficient, quickly identifying that the provided number was not a valid order number for their system. The customer then realized they may have called the wrong place or about a different product, so the original issue was not addressed.

  • The customer provided an invalid order number and appeared to have contacted the wrong company or about the wrong product.
Phase time & cost · 12.4s · $0.0153
PhaseLatencyFinancial cost
Speech-to-textElevenLabs Scribe v2
6.4stime
$0.0040audio_minutes
AnalysisGPT-5.5
6.0stime
$0.0113tokens
End-to-end12.4s$0.0153

HR 1

4 speech pipelines · 8 results

MLX Whisper large-v3

hr-1-mlx
175 wordsSTT 9.5sSTT $0.0000
Gemma 4 26B
Tone 5 · Clarity 4 · Efficiency 5
valid · fenced_json · 1 candidate(s)
outcome resolved · unable to evaluate false

The agent efficiently handled the customer's request to fix a duplicate order by canceling the existing order and explaining the refund process. The agent provided clear instructions on how the customer should proceed (reordering) and managed expectations regarding the refund timeline.

Phase time & cost · 36.0s · $0.0092
PhaseLatencyFinancial cost
Speech-to-textMLX Whisper large-v3
9.5stime
$0.0000runtime_hours
AnalysisGemma 4 26B
26.5stime
$0.0092runtime_hours
End-to-end36.0s$0.0092
GPT-5.5
Tone 5 · Clarity 4 · Efficiency 5
valid · direct_json · 1 candidate(s)
outcome resolved · unable to evaluate false

The agent was polite and helpful, clearly explained that the full order needed to be canceled and refunded before the customer could place a new order, then processed the cancellation and explained the refund timing. The issue was effectively resolved, though the refund timing explanation was somewhat broad.

  • Refund timing was not guaranteed and could have been explained more precisely.
Phase time & cost · 14.4s · $0.0094
PhaseLatencyFinancial cost
Speech-to-textMLX Whisper large-v3
9.5stime
$0.0000runtime_hours
AnalysisGPT-5.5
4.9stime
$0.0094tokens
End-to-end14.4s$0.0094

OpenAI gpt-4o-transcribe

hr-1-openai-full
180 wordsSTT 5.0sSTT $0.0086
Gemma 4 26B
Tone 5 · Clarity 5 · Efficiency 5
valid · fenced_json · 1 candidate(s)
outcome resolved · unable to evaluate false

The agent clearly explained the only available solution for the duplicate item (cancelling the entire order and re-ordering) and successfully processed the cancellation request for the customer.

Phase time & cost · 23.0s · $0.0149
PhaseLatencyFinancial cost
Speech-to-textOpenAI gpt-4o-transcribe
5.0stime
$0.0086audio_minutes
AnalysisGemma 4 26B
17.9stime
$0.0062runtime_hours
End-to-end23.0s$0.0149
GPT-5.5
Tone 5 · Clarity 5 · Efficiency 5
valid · direct_json · 1 candidate(s)
outcome resolved · unable to evaluate false

The agent was polite, clear, and efficient. They explained that the full order needed to be canceled, confirmed the customer's details and duplicate item, canceled the order, and explained the expected refund timing.

Phase time & cost · 9.3s · $0.0163
PhaseLatencyFinancial cost
Speech-to-textOpenAI gpt-4o-transcribe
5.0stime
$0.0086audio_minutes
AnalysisGPT-5.5
4.2stime
$0.0076tokens
End-to-end9.3s$0.0163

OpenAI gpt-4o-mini-transcribe

hr-1-openai-mini
178 wordsSTT 3.2sSTT $0.0043
Gemma 4 26B
Tone 5 · Clarity 5 · Efficiency 5
valid · fenced_json · 1 candidate(s)
outcome resolved · unable to evaluate false

The agent efficiently handled the customer's request to cancel an order containing a duplicate item. The agent clearly explained the necessary steps (canceling and reordering) and provided a realistic expectation regarding the refund timeline.

Phase time & cost · 27.1s · $0.0126
PhaseLatencyFinancial cost
Speech-to-textOpenAI gpt-4o-mini-transcribe
3.2stime
$0.0043audio_minutes
AnalysisGemma 4 26B
23.9stime
$0.0083runtime_hours
End-to-end27.1s$0.0126
GPT-5.5
Tone 5 · Clarity 4 · Efficiency 5
valid · direct_json · 1 candidate(s)
outcome resolved · unable to evaluate false

The agent was polite and helpful, explained that the order needed to be canceled and recreated, confirmed the duplicate item, canceled the order, and provided a refund timeframe. The issue appears resolved.

Phase time & cost · 7.8s · $0.0120
PhaseLatencyFinancial cost
Speech-to-textOpenAI gpt-4o-mini-transcribe
3.2stime
$0.0043audio_minutes
AnalysisGPT-5.5
4.6stime
$0.0077tokens
End-to-end7.8s$0.0120

ElevenLabs Scribe v2

hr-1-scribe
205 wordsSTT 14.9sSTT $0.0053
Gemma 4 26B
Tone 5 · Clarity 5 · Efficiency 5
valid · fenced_json · 1 candidate(s)
outcome resolved · unable to evaluate false

The agent efficiently handled the customer's request to cancel an order that contained a duplicate item. The agent clearly explained the necessary steps (canceling the current order and placing a new one) and provided realistic expectations regarding the refund timeline.

Phase time & cost · 41.6s · $0.0146
PhaseLatencyFinancial cost
Speech-to-textElevenLabs Scribe v2
14.9stime
$0.0053audio_minutes
AnalysisGemma 4 26B
26.7stime
$0.0093runtime_hours
End-to-end41.6s$0.0146
GPT-5.5
Tone 5 · Clarity 4 · Efficiency 5
valid · direct_json · 1 candidate(s)
outcome resolved · unable to evaluate false

The agent was polite, helpful, and quickly identified the duplicate item. They clearly explained that the only option was to cancel the full order and have the customer place a new one, then completed the cancellation and provided refund timing expectations.

Phase time & cost · 18.7s · $0.0129
PhaseLatencyFinancial cost
Speech-to-textElevenLabs Scribe v2
14.9stime
$0.0053audio_minutes
AnalysisGPT-5.5
3.8stime
$0.0076tokens
End-to-end18.7s$0.0129

RO 1

4 speech pipelines · 8 results

MLX Whisper large-v3

ro-1-mlx
139 wordsSTT 8.3sSTT $0.0000
Gemma 4 26B
Tone 4 · Clarity 3 · Efficiency 3
valid · fenced_json · 1 candidate(s)
outcome partial · unable to evaluate false

The agent provided an explanation for the delay by detailing the logistics route (from Oradea to the Czech Republic warehouse), which helped manage expectations, although the customer's request for a definitive update remains pending.

  • The conversation flow was somewhat fragmented and disjointed.
  • The customer expressed concern regarding the lack of an immediate response to their request.
Phase time & cost · 42.5s · $0.0119
PhaseLatencyFinancial cost
Speech-to-textMLX Whisper large-v3
8.3stime
$0.0000runtime_hours
AnalysisGemma 4 26B
34.2stime
$0.0119runtime_hours
End-to-end42.5s$0.0119
GPT-5.5
Tone 4 · Clarity 3 · Efficiency 3
valid · direct_json · 1 candidate(s)
outcome partial · unable to evaluate false

The agent was polite and provided a general explanation that the shipment/request had not yet arrived in the system and would likely arrive on Monday. However, the explanation was somewhat unclear and did not provide a firm resolution or proactive follow-up.

  • No immediate resolution was provided.
  • Timeline was approximate rather than confirmed.
  • The agent indicated the customer would need to call back instead of offering proactive follow-up.
  • Some explanations were unclear and fragmented.
Phase time & cost · 14.6s · $0.0114
PhaseLatencyFinancial cost
Speech-to-textMLX Whisper large-v3
8.3stime
$0.0000runtime_hours
AnalysisGPT-5.5
6.3stime
$0.0114tokens
End-to-end14.6s$0.0114

OpenAI gpt-4o-transcribe

ro-1-openai-full
214 wordsSTT 5.7sSTT $0.0093
Gemma 4 26B
Tone 4 · Clarity 4 · Efficiency 4
valid · fenced_json · 1 candidate(s)
outcome partial · unable to evaluate false

The agent clearly explained the return procedure (packaging, courier, and fees) and provided an update regarding a shipment currently in transit. The agent was polite and proactive in providing information.

Phase time & cost · 34.9s · $0.0194
PhaseLatencyFinancial cost
Speech-to-textOpenAI gpt-4o-transcribe
5.7stime
$0.0093audio_minutes
AnalysisGemma 4 26B
29.3stime
$0.0102runtime_hours
End-to-end34.9s$0.0194
GPT-5.5
Tone 4 · Clarity 3 · Efficiency 3
valid · direct_json · 1 candidate(s)
outcome partial · unable to evaluate false

The interaction was polite and respectful, and the agent provided general next steps about the return process and expected timing. However, there was some confusion about whether the product had already been sent and when it would arrive, and the issue remains pending until the returned item reaches the depot and is processed.

  • Initial instructions did not account for the customer having already sent the product.
  • Timeline and next steps were only approximate.
  • No immediate resolution was provided; the customer must follow up after the return arrives.
Phase time & cost · 11.8s · $0.0213
PhaseLatencyFinancial cost
Speech-to-textOpenAI gpt-4o-transcribe
5.7stime
$0.0093audio_minutes
AnalysisGPT-5.5
6.1stime
$0.0120tokens
End-to-end11.8s$0.0213

OpenAI gpt-4o-mini-transcribe

ro-1-openai-mini
212 wordsSTT 3.8sSTT $0.0046
Gemma 4 26B
Tone 5 · Clarity 5 · Efficiency 5
valid · fenced_json · 1 candidate(s)
outcome partial · unable to evaluate false

The agent proactively contacted the customer to provide an update on their complaint and the return process. The agent clearly explained the necessary steps for the return and provided information regarding the shipment's current status and the logistics involved (transit from Oradea to the Czech Republic warehouse). The issue is partially resolved as the customer is now informed, though the final resolution depends on the package arrival.

Phase time & cost · 36.0s · $0.0158
PhaseLatencyFinancial cost
Speech-to-textOpenAI gpt-4o-mini-transcribe
3.8stime
$0.0046audio_minutes
AnalysisGemma 4 26B
32.3stime
$0.0112runtime_hours
End-to-end36.0s$0.0158
GPT-5.5
Tone 4 · Clarity 3 · Efficiency 3
valid · direct_json · 1 candidate(s)
outcome partial · unable to evaluate false

The agent was polite and provided general information about the return process and expected next steps. However, there was some initial confusion because the customer had already sent the product back, and the agent could not confirm receipt yet. The issue remains pending until the package arrives and is processed.

  • Agent was not initially aware that the customer had already shipped the return.
  • No immediate resolution was possible because the return was not yet in the system.
  • Follow-up responsibility was not clearly owned by the agent, as the customer was asked to call again later.
Phase time & cost · 10.6s · $0.0166
PhaseLatencyFinancial cost
Speech-to-textOpenAI gpt-4o-mini-transcribe
3.8stime
$0.0046audio_minutes
AnalysisGPT-5.5
6.9stime
$0.0120tokens
End-to-end10.6s$0.0166

ElevenLabs Scribe v2

ro-1-scribe
229 wordsSTT 11.5sSTT $0.0057
Gemma 4 26B
Tone 4 · Clarity 4 · Efficiency 4
valid · fenced_json · 1 candidate(s)
outcome partial · unable to evaluate false

The agent proactively called the customer to discuss the return procedure for a complaint. The agent promised to send the instructions via email. The customer informed the agent that a package had already been dispatched via courier. The call concluded with an agreement to follow up once the package arrives at the warehouse.

  • The agent's explanation regarding the logistics and warehouse locations (Oradea to Czechia) was slightly disorganized.
Phase time & cost · 42.6s · $0.0165
PhaseLatencyFinancial cost
Speech-to-textElevenLabs Scribe v2
11.5stime
$0.0057audio_minutes
AnalysisGemma 4 26B
31.1stime
$0.0108runtime_hours
End-to-end42.6s$0.0165
GPT-5.5
Tone 4 · Clarity 3 · Efficiency 4
valid · direct_json · 1 candidate(s)
outcome partial · unable to evaluate false

The agent was polite and confirmed the return had not yet appeared in the system, but the issue was not resolved during the call because the returned product had not arrived yet and further follow-up is needed.

  • Agent was initially unaware that the customer had already sent the product back.
  • The next steps and timeline were only partially clarified.
  • The customer was asked to follow up later rather than being offered a proactive update.
Phase time & cost · 18.4s · $0.0179
PhaseLatencyFinancial cost
Speech-to-textElevenLabs Scribe v2
11.5stime
$0.0057audio_minutes
AnalysisGPT-5.5
6.8stime
$0.0123tokens
End-to-end18.4s$0.0179

BG 1

4 speech pipelines · 8 results

MLX Whisper large-v3

bg-1-mlx
324 wordsSTT 16.8sSTT $0.0000
Gemma 4 26B
Tone 4 · Clarity 4 · Efficiency 3
valid · fenced_json · 1 candidate(s)
outcome unresolved · unable to evaluate false

The agent was polite and followed the correct procedure for taking an order, but the call ended without a successful transaction because the requested item size was out of stock. The agent was somewhat repetitive with apologies.

  • Product unavailability (size out of stock)
  • Repetitive apologies
Phase time & cost · 35.3s · $0.0064
PhaseLatencyFinancial cost
Speech-to-textMLX Whisper large-v3
16.8stime
$0.0000runtime_hours
AnalysisGemma 4 26B
18.5stime
$0.0064runtime_hours
End-to-end35.3s$0.0064
GPT-5.5
Tone 4 · Clarity 3 · Efficiency 3
valid · direct_json · 1 candidate(s)
outcome unresolved · unable to evaluate false

The agent was generally polite and explained the delivery timeframe and lack of guarantee. However, the call ended without an order because the requested sizes were unavailable. Efficiency was reduced because customer details were collected before confirming stock availability.

  • Stock availability was checked only after collecting customer information.
  • The requested product sizes were unavailable, so the customer could not place the order.
  • Some explanations around delivery timing were not fully clear or definitive.
Phase time & cost · 22.7s · $0.0124
PhaseLatencyFinancial cost
Speech-to-textMLX Whisper large-v3
16.8stime
$0.0000runtime_hours
AnalysisGPT-5.5
5.9stime
$0.0124tokens
End-to-end22.7s$0.0124

OpenAI gpt-4o-transcribe

bg-1-openai-full
295 wordsSTT 8.2sSTT $0.0202
Gemma 4 26B
Tone 3 · Clarity 3 · Efficiency 2
valid · fenced_json · 1 candidate(s)
outcome unresolved · unable to evaluate false

The agent attempted to process an order for the customer, but the requested size (54) and the alternative size (52) were both out of stock. The call ended without a successful transaction.

  • Inventory discrepancy: The customer believed the items were in stock based on the website, but the agent confirmed they were unavailable.
  • The primary goal of the call (placing an order) was not achieved.
Phase time & cost · 32.1s · $0.0285
PhaseLatencyFinancial cost
Speech-to-textOpenAI gpt-4o-transcribe
8.2stime
$0.0202audio_minutes
AnalysisGemma 4 26B
23.9stime
$0.0083runtime_hours
End-to-end32.1s$0.0285
GPT-5.5
Tone 4 · Clarity 3 · Efficiency 2
valid · direct_json · 1 candidate(s)
outcome unresolved · unable to evaluate false

The agent was generally polite and explained that delivery could not be guaranteed, but the interaction became inefficient because customer details were collected before confirming product availability. The requested sizes were unavailable, so the order was not completed.

  • Agent checked stock only after collecting full customer and delivery information.
  • Some explanations were unclear or awkward, likely causing minor confusion.
  • Customer's desired product size was unavailable, leaving the issue unresolved.
Phase time & cost · 14.5s · $0.0320
PhaseLatencyFinancial cost
Speech-to-textOpenAI gpt-4o-transcribe
8.2stime
$0.0202audio_minutes
AnalysisGPT-5.5
6.2stime
$0.0118tokens
End-to-end14.5s$0.0320

OpenAI gpt-4o-mini-transcribe

bg-1-openai-mini
297 wordsSTT 6.1sSTT $0.0101
Gemma 4 26B
Tone 2 · Clarity 3 · Efficiency 1
valid · fenced_json · 1 candidate(s)
outcome unresolved · unable to evaluate false

The agent's performance was highly inefficient. They proceeded to collect the customer's full name, delivery address, and postal code, only to reveal at the very end of the call that the requested size was out of stock. This resulted in a wasted interaction for both parties. Additionally, the agent's phrasing regarding creating a 'dummy profile' was unprofessional.

  • Failed to check stock availability before proceeding with the order process
  • Inefficient use of time/process
  • Unprofessional language regarding account creation ('dummy profile')
Phase time & cost · 31.9s · $0.0191
PhaseLatencyFinancial cost
Speech-to-textOpenAI gpt-4o-mini-transcribe
6.1stime
$0.0101audio_minutes
AnalysisGemma 4 26B
25.8stime
$0.0090runtime_hours
End-to-end31.9s$0.0191
GPT-5.5
Tone 4 · Clarity 3 · Efficiency 2
valid · direct_json · 1 candidate(s)
outcome unresolved · unable to evaluate false

The agent was generally polite and answered the delivery question, but the order could not be completed because the requested size was unavailable. The interaction was somewhat inefficient because customer details were collected before confirming stock availability, and some explanations were not very clear.

  • Stock availability was checked only after collecting the customer's personal and delivery details.
  • Delivery timing was not guaranteed, though the agent gave an estimated timeframe.
  • Some wording and explanations were unclear, including the need to create a temporary profile.
Phase time & cost · 11.1s · $0.0211
PhaseLatencyFinancial cost
Speech-to-textOpenAI gpt-4o-mini-transcribe
6.1stime
$0.0101audio_minutes
AnalysisGPT-5.5
5.0stime
$0.0110tokens
End-to-end11.1s$0.0211

ElevenLabs Scribe v2

bg-1-scribe
384 wordsSTT 24.0sSTT $0.0124
Gemma 4 26B
Tone 3 · Clarity 3 · Efficiency 2
valid · fenced_json · 1 candidate(s)
outcome unresolved · unable to evaluate false

The customer intended to place an order, but the transaction could not be completed because the requested sizes were out of stock. The call was also characterized by initial connection instability and interruptions.

  • Initial connection issues and interruptions
  • Product unavailability (out of stock)
  • Unprofessional phrasing regarding the creation of a 'dummy profile'
Phase time & cost · 49.1s · $0.0211
PhaseLatencyFinancial cost
Speech-to-textElevenLabs Scribe v2
24.0stime
$0.0124audio_minutes
AnalysisGemma 4 26B
25.1stime
$0.0087runtime_hours
End-to-end49.1s$0.0211
GPT-5.5
Tone 4 · Clarity 3 · Efficiency 2
valid · direct_json · 1 candidate(s)
outcome unresolved · unable to evaluate false

The agent was generally polite and provided delivery information, but could not complete the order because the requested sizes were unavailable. The interaction was somewhat unclear and inefficient, as customer details were collected before confirming stock availability.

  • Agent did not verify product size availability before creating the order profile and collecting customer information.
  • Delivery timing was explained with uncertainty, which was appropriate, but the wording was somewhat repetitive and unclear.
  • The order was not completed because the requested sizes were out of stock.
Phase time & cost · 30.6s · $0.0260
PhaseLatencyFinancial cost
Speech-to-textElevenLabs Scribe v2
24.0stime
$0.0124audio_minutes
AnalysisGPT-5.5
6.6stime
$0.0136tokens
End-to-end30.6s$0.0260

CZ 1

4 speech pipelines · 8 results

MLX Whisper large-v3

cz-1-mlx
827 wordsSTT 37.0sSTT $0.0000
Gemma 4 26B
Tone 5 · Clarity 5 · Efficiency 5
valid · fenced_json · 1 candidate(s)
outcome resolved · unable to evaluate false

The agent provided excellent service, answering all the customer's questions regarding the pergola product. They were proactive in suggesting where to find customer photos and provided helpful advice regarding seasonal maintenance (removing the fabric in winter) and potential stock shortages. The agent also clearly managed expectations regarding the lack of a physical showroom.

Phase time & cost · 54.1s · $0.0060
PhaseLatencyFinancial cost
Speech-to-textMLX Whisper large-v3
37.0stime
$0.0000runtime_hours
AnalysisGemma 4 26B
17.1stime
$0.0060runtime_hours
End-to-end54.1s$0.0060
GPT-5.5
Tone 5 · Clarity 4 · Efficiency 3
valid · direct_json · 1 candidate(s)
outcome partial · unable to evaluate false

The agent was polite, patient, and helpful, and answered most product-related questions after checking the information. However, the response to payment and delivery timing was incomplete, and the call included some repetition and longer explanations than necessary.

  • Payment options were not clearly explained.
  • Estimated delivery time was not clearly provided.
  • Some answers were lengthy and repetitive.
Phase time & cost · 42.2s · $0.0159
PhaseLatencyFinancial cost
Speech-to-textMLX Whisper large-v3
37.0stime
$0.0000runtime_hours
AnalysisGPT-5.5
5.2stime
$0.0159tokens
End-to-end42.2s$0.0159

OpenAI gpt-4o-transcribe

cz-1-openai-full
864 wordsSTT 21.5sSTT $0.0425
Gemma 4 26B
Tone 5 · Clarity 5 · Efficiency 5
valid · fenced_json · 1 candidate(s)
outcome resolved · unable to evaluate false

The agent provided excellent service, answering all the customer's questions regarding product specifications, delivery, and maintenance. The agent was proactive, offering additional advice on seasonal care (taking the pergola down for winter) and suggesting where to find real-life customer photos on the website to assist with the purchasing decision.

Phase time & cost · 40.1s · $0.0490
PhaseLatencyFinancial cost
Speech-to-textOpenAI gpt-4o-transcribe
21.5stime
$0.0425audio_minutes
AnalysisGemma 4 26B
18.6stime
$0.0065runtime_hours
End-to-end40.1s$0.0490
GPT-5.5
Tone 5 · Clarity 4 · Efficiency 4
valid · direct_json · 1 candidate(s)
outcome resolved · unable to evaluate false

The agent was polite, patient, and helpful. They answered the customer's questions about whether the fabric cover is included, winter storage, delivery timing, payment/order process, viewing options, and returns. The explanation was generally clear, though somewhat repetitive and occasionally verbose.

  • Some answers were longer than necessary and repeated points about winter storage and product availability.
  • The agent initially needed time to verify product details, but handled it appropriately.
Phase time & cost · 25.3s · $0.0568
PhaseLatencyFinancial cost
Speech-to-textOpenAI gpt-4o-transcribe
21.5stime
$0.0425audio_minutes
AnalysisGPT-5.5
3.8stime
$0.0143tokens
End-to-end25.3s$0.0568

OpenAI gpt-4o-mini-transcribe

cz-1-openai-mini
577 wordsSTT 17.8sSTT $0.0212
Gemma 4 26B
Tone 5 · Clarity 5 · Efficiency 5
valid · fenced_json · 1 candidate(s)
outcome resolved · unable to evaluate false

The agent was very professional and helpful. They successfully provided the requested product information, answered questions about the canopy, and proactively offered advice on product maintenance (winter storage) and how to view real-life photos via customer reviews on the website. The agent also clearly explained the return policy.

Phase time & cost · 35.4s · $0.0273
PhaseLatencyFinancial cost
Speech-to-textOpenAI gpt-4o-mini-transcribe
17.8stime
$0.0212audio_minutes
AnalysisGemma 4 26B
17.6stime
$0.0061runtime_hours
End-to-end35.4s$0.0273
GPT-5.5
Tone 4 · Clarity 3 · Efficiency 3
valid · direct_json · 1 candidate(s)
outcome resolved · unable to evaluate false

The agent was polite and helpful, guided the customer to the product code, answered questions about the pergola, explained that there is no physical showroom, pointed to customer photos/reviews, and mentioned the legal return option. However, the transcript contains significant repeated segments and some unclear wording, making parts of the exchange difficult to follow. The answer about whether the fabric is included was somewhat unclear before the call moved on.

  • Transcript contains repeated sections, which reduces evaluability and apparent efficiency.
  • Some explanations were not fully clear, especially around whether the fabric is included.
  • Agent used informal phrasing and seemed to check information while speaking, which affected clarity.
Phase time & cost · 26.4s · $0.0393
PhaseLatencyFinancial cost
Speech-to-textOpenAI gpt-4o-mini-transcribe
17.8stime
$0.0212audio_minutes
AnalysisGPT-5.5
8.6stime
$0.0180tokens
End-to-end26.4s$0.0393

ElevenLabs Scribe v2

cz-1-scribe
1002 wordsSTT 40.8sSTT $0.0260
Gemma 4 26B
Tone 5 · Clarity 5 · Efficiency 5
valid · fenced_json · 1 candidate(s)
outcome resolved · unable to evaluate false

The agent provided excellent service. They answered all the customer's questions regarding the pergola, including whether the fabric is included, how to maintain it during winter, and delivery expectations. The agent was also proactive, suggesting the customer look at user reviews for real-life photos and mentioning the 14-day return policy.

Phase time & cost · 62.2s · $0.0334
PhaseLatencyFinancial cost
Speech-to-textElevenLabs Scribe v2
40.8stime
$0.0260audio_minutes
AnalysisGemma 4 26B
21.4stime
$0.0074runtime_hours
End-to-end62.2s$0.0334
GPT-5.5
Tone 4 · Clarity 4 · Efficiency 3
valid · direct_json · 1 candidate(s)
outcome resolved · unable to evaluate false

The agent was polite and helpful, verified the product details before answering, and addressed the customer's questions about included fabric, winter storage, delivery, payment/order process, viewing options, and returns. The call was somewhat inefficient due to filler language, repeated explanations, and extra commentary about product popularity.

  • Agent used many filler words and hesitations.
  • Some answers initially lacked confidence before verification.
  • The call included unnecessary extra commentary about seasonal demand and product popularity.
Phase time & cost · 46.8s · $0.0441
PhaseLatencyFinancial cost
Speech-to-textElevenLabs Scribe v2
40.8stime
$0.0260audio_minutes
AnalysisGPT-5.5
6.0stime
$0.0182tokens
End-to-end46.8s$0.0441

45998 hard stereo

4 speech pipelines · 8 results

MLX Whisper large-v3

45998-mlx
1495 wordsSTT 63.2sSTT $0.0000
Gemma 4 26B
Tone 4 · Clarity 4 · Efficiency 5
valid · fenced_json · 1 candidate(s)
outcome partial · unable to evaluate false

The agent handled a complex logistics inquiry professionally. They demonstrated empathy regarding the client's difficulty with identifying packages due to overwritten shipping labels and provided a clear path forward by promising to send a comprehensive price list and investigate labeling solutions. The call ended with clear next steps.

Phase time & cost · 88.0s · $0.0086
PhaseLatencyFinancial cost
Speech-to-textMLX Whisper large-v3
63.2stime
$0.0000runtime_hours
AnalysisGemma 4 26B
24.8stime
$0.0086runtime_hours
End-to-end88.0s$0.0086
GPT-5.5
Tone 4 · Clarity 3 · Efficiency 3
valid · direct_json · 1 candidate(s)
outcome partial · unable to evaluate false

The representative was polite, patient, and showed understanding of the customer's issue. The discussion clarified the service concept and ended with concrete next steps: sending prices for supported countries and confirming whether customer names can be marked on packages. However, explanations were sometimes lengthy and somewhat unclear, with some uncertainty about operational details that still need follow-up.

  • Explanations were occasionally repetitive and hard to follow.
  • Some key details, such as supported countries, pricing, and package labeling feasibility, were not fully confirmed during the call.
  • The issue was not fully resolved on the call and requires follow-up by email.
Phase time & cost · 70.4s · $0.0243
PhaseLatencyFinancial cost
Speech-to-textMLX Whisper large-v3
63.2stime
$0.0000runtime_hours
AnalysisGPT-5.5
7.3stime
$0.0243tokens
End-to-end70.4s$0.0243

OpenAI gpt-4o-transcribe

45998-openai-full
1083 wordsSTT 26.0sSTT $0.0592
Gemma 4 26B
Tone 4 · Clarity 4 · Efficiency 3
valid · fenced_json · 1 candidate(s)
outcome partial · unable to evaluate false

The agent is actively addressing the customer's concerns regarding the consolidation of shipments and the logistical difficulty of identifying packages when labels are covered by new shipping stickers. While the agent proposed looking into pricing for additional countries and suggested using order numbers for identification, the call ends before a final agreement or a complete pricing list is provided.

  • The customer highlighted a significant operational risk where original package information is lost due to overlapping shipping labels.
  • The resolution is only partial as the agent still needs to provide a comprehensive price list for the requested countries.
Phase time & cost · 48.7s · $0.0671
PhaseLatencyFinancial cost
Speech-to-textOpenAI gpt-4o-transcribe
26.0stime
$0.0592audio_minutes
AnalysisGemma 4 26B
22.7stime
$0.0079runtime_hours
End-to-end48.7s$0.0671
GPT-5.5
Tone 4 · Clarity 3 · Efficiency 3
valid · direct_json · 1 candidate(s)
outcome partial · unable to evaluate false

The agent was polite and tried to explain the logistics options, pricing approach, and possible package identification solutions. However, the explanation was lengthy and sometimes repetitive, and several key items such as pricing for additional countries and the exact order-number labeling process were left for follow-up. The transcript also appears to end before the call fully concludes.

  • Transcript appears to be cut off before the end of the call.
  • Agent was somewhat verbose and repetitive, which reduced clarity and efficiency.
  • Pricing for all requested countries was not confirmed during the call.
  • Package identification and order-number handling were discussed but not fully resolved.
Phase time & cost · 36.4s · $0.0860
PhaseLatencyFinancial cost
Speech-to-textOpenAI gpt-4o-transcribe
26.0stime
$0.0592audio_minutes
AnalysisGPT-5.5
10.4stime
$0.0268tokens
End-to-end36.4s$0.0860

OpenAI gpt-4o-mini-transcribe

45998-openai-mini
1076 wordsSTT 15.7sSTT $0.0296
Gemma 4 26B
Tone 4 · Clarity 4 · Efficiency 4
valid · fenced_json · 1 candidate(s)
outcome partial · unable to evaluate false

The agent effectively handled the inquiry by identifying the customer's specific geographic needs (Hungary, Spain) and addressing a critical operational pain point regarding obscured shipping labels. While a final agreement wasn't reached, a clear follow-up plan was established to provide a revised price list and investigate labeling solutions.

  • The agent needs to follow up with a comprehensive price list for the requested countries.
  • The company needs to address the operational issue of secondary labels obscuring original order identifiers.
Phase time & cost · 38.7s · $0.0376
PhaseLatencyFinancial cost
Speech-to-textOpenAI gpt-4o-mini-transcribe
15.7stime
$0.0296audio_minutes
AnalysisGemma 4 26B
22.9stime
$0.0080runtime_hours
End-to-end38.7s$0.0376
GPT-5.5
Tone 4 · Clarity 3 · Efficiency 3
valid · direct_json · 1 candidate(s)
outcome partial · unable to evaluate false

The agent was polite and professional, explained the consolidation and delivery options, and offered to verify pricing for relevant countries and labeling options. However, the explanation was quite repetitive and lengthy, some requested countries/prices were not immediately available, and the transcript ends before confirming a final solution.

  • Transcript appears to be cut off before the call conclusion.
  • Agent was verbose and repetitive, which reduced clarity and efficiency.
  • Pricing for all requested countries was not immediately available.
  • Customer's concern about package identification/order number labeling was not fully resolved during the available transcript.
Phase time & cost · 24.5s · $0.0536
PhaseLatencyFinancial cost
Speech-to-textOpenAI gpt-4o-mini-transcribe
15.7stime
$0.0296audio_minutes
AnalysisGPT-5.5
8.8stime
$0.0240tokens
End-to-end24.5s$0.0536

ElevenLabs Scribe v2

45998-scribe
1621 wordsSTT 39.8sSTT $0.0362
Gemma 4 26B
Tone 5 · Clarity 5 · Efficiency 5
valid · fenced_json · 1 candidate(s)
outcome resolved · unable to evaluate false

The agent demonstrated excellent empathy and professionalism, particularly when discussing the customer's difficulty with identifying packages due to overwritten labels. The agent provided a clear explanation of their logistics process and established a concrete follow-up plan: sending a comprehensive price list and investigating the possibility of adding identifiers to the packages.

Phase time & cost · 64.6s · $0.0448
PhaseLatencyFinancial cost
Speech-to-textElevenLabs Scribe v2
39.8stime
$0.0362audio_minutes
AnalysisGemma 4 26B
24.8stime
$0.0086runtime_hours
End-to-end64.6s$0.0448
GPT-5.5
Tone 5 · Clarity 4 · Efficiency 3
valid · direct_json · 1 candidate(s)
outcome partial · unable to evaluate false

The agent was polite, empathetic, and professional, and she acknowledged the customer's concerns about shipment identification. She explained the Zlin consolidation option and confirmed the intended next steps, but some explanations were lengthy and not all pricing or labeling answers were available during the call. The issue was partially addressed, with follow-up required to send prices and confirm whether client names can be added to packages.

  • Agent did not have complete pricing information available during the call.
  • Agent could not immediately confirm the exact package-labeling process and needed internal follow-up.
  • Explanations were somewhat long and repetitive, which reduced efficiency.
Phase time & cost · 45.6s · $0.0613
PhaseLatencyFinancial cost
Speech-to-textElevenLabs Scribe v2
39.8stime
$0.0362audio_minutes
AnalysisGPT-5.5
5.8stime
$0.0251tokens
End-to-end45.6s$0.0613