--- title: 'Evaluating ElevenLabs Voice AI' description: 'Step-by-step guide for testing ElevenLabs voice AI with Promptfoo - from TTS quality testing to conversational agent evaluation' --- # Evaluating ElevenLabs voice AI This guide walks you through testing ElevenLabs voice AI capabilities using Promptfoo, from basic text-to-speech quality testing to advanced conversational agent evaluation. ## Part 1: Text-to-Speech Quality Testing Let's start by comparing different voice models and measuring their quality. Turbo is retained here for comparison; [ElevenLabs recommends Flash over Turbo](https://elevenlabs.io/docs/overview/models#deprecated-models) for new configurations. ### Step 1: Setup Install Promptfoo and set your API key: ```sh npm install -g promptfoo export ELEVENLABS_API_KEY=your_api_key_here ``` ### Step 2: Create Your First Config Create `promptfooconfig.yaml`. The snippets use an example voice ID; replace it with a voice ID available in your [ElevenLabs voice list](https://elevenlabs.io/docs/api-reference/voices/search). Display names such as `rachel` are not resolved to IDs: ```yaml description: 'Compare ElevenLabs TTS models for customer service greetings' prompts: - "Thank you for calling TechSupport Inc. My name is Alex, and I'll be assisting you today. How can I help?" providers: - label: Flash Model (Fastest) id: elevenlabs:tts:21m00Tcm4TlvDq8ikWAM config: modelId: eleven_flash_v2_5 outputFormat: mp3_44100_128 - label: Turbo Model (Legacy Comparison) id: elevenlabs:tts:21m00Tcm4TlvDq8ikWAM config: modelId: eleven_turbo_v2_5 outputFormat: mp3_44100_128 tests: - description: Both models complete within 3 seconds assert: - type: latency threshold: 3000 - description: Cost is under $0.01 per greeting assert: - type: cost threshold: 0.01 ``` ### Step 3: Run Your First Eval ```sh promptfoo eval ``` You'll see results comparing both models: ```text ┌─────────────────────────┬──────────┬──────────┐ │ Prompt │ Flash │ Turbo │ ├─────────────────────────┼──────────┼──────────┤ │ Thank you for calling...│ ✓ Pass │ ✓ Pass │ │ Latency: <3s │ 847ms │ 1,234ms │ │ Cost: <$0.01 │ $0.003 │ $0.004 │ └─────────────────────────┴──────────┴──────────┘ ``` ### Step 4: View Results Open the web UI to listen to the audio: ```sh promptfoo view ``` ## Part 2: Voice Customization Now let's optimize voice settings for different use cases. ### Step 5: Add Voice Settings Update your config: ```yaml description: 'Test voice settings for different scenarios' prompts: - 'Welcome to our automated system.' # Formal announcement - 'Hey there! Thanks for reaching out.' # Casual greeting - 'I understand your frustration. Let me help.' # Empathetic response providers: - label: Professional Voice id: elevenlabs:tts:21m00Tcm4TlvDq8ikWAM config: modelId: eleven_flash_v2_5 voiceSettings: stability: 0.8 # Consistent tone similarity_boost: 0.85 speed: 0.95 - label: Friendly Voice id: elevenlabs:tts:21m00Tcm4TlvDq8ikWAM config: modelId: eleven_flash_v2_5 voiceSettings: stability: 0.4 # More variation similarity_boost: 0.75 speed: 1.1 # Slightly faster - label: Empathetic Voice id: elevenlabs:tts:21m00Tcm4TlvDq8ikWAM config: modelId: eleven_flash_v2_5 voiceSettings: stability: 0.5 similarity_boost: 0.7 style: 0.8 # More expressive speed: 0.9 # Slower, calmer tests: - vars: scenario: formal assert: - type: javascript value: | const audio = context.providerResponse.audio; return Boolean(audio?.data || audio?.blobRef); - vars: scenario: casual assert: - type: latency threshold: 2000 - vars: scenario: empathy assert: - type: cost threshold: 0.01 ``` Run the eval: ```sh promptfoo eval promptfoo view # Compare the different voice styles ``` ## Part 3: Speech-to-Text Accuracy Test transcription accuracy by creating a TTS → STT pipeline. ### Step 6: Create Transcription Pipeline Create `transcription-test.yaml`: ```yaml description: 'Test TTS → STT accuracy pipeline' prompts: - | The quarterly sales meeting is scheduled for Thursday, March 15th at 2:30 PM. Please bring your laptop, quarterly reports, and the Q4 projections spreadsheet. Conference room B has been reserved for this meeting. providers: # Step 1: Generate audio - label: tts-generator id: elevenlabs:tts:21m00Tcm4TlvDq8ikWAM config: modelId: eleven_flash_v2_5 saveAudio: true audioOutputPath: audio tests: - description: Generate audio and verify quality assert: - type: javascript value: | // Verify audio was generated const audio = context.providerResponse.audio; return Boolean(audio?.data || audio?.blobRef); ``` Run `promptfoo eval -c transcription-test.yaml --no-cache` to save `audio/tts-.mp3`, then copy the generated file to `audio/generated-speech.mp3`. Add STT to verify accuracy in a second config, `stt-accuracy.yaml`: ```yaml description: 'Test STT accuracy' prompts: - '{{audioFile}}' providers: - id: elevenlabs:stt config: modelId: scribe_v2 calculateWER: true referenceText: 'The quarterly sales meeting is scheduled for Thursday, March 15th at 2:30 PM. Please bring your laptop, quarterly reports, and the Q4 projections spreadsheet. Conference room B has been reserved for this meeting.' tests: - vars: audioFile: audio/generated-speech.mp3 # File path from the previous eval assert: - type: javascript value: | // Transcription text is output; WER is in response metadata. const wer = context.providerResponse.metadata?.wer?.wer; return typeof wer === 'number' && wer < 0.05; ``` Run the STT eval: ```sh promptfoo eval -c stt-accuracy.yaml ``` ## Part 4: Conversational Agent Testing Test a complete voice agent with evaluation criteria. ### Step 7: Create Agent Config Create `agent-test.yaml`: ```yaml description: 'Test customer support agent performance' prompts: - | User: Hi, I'm having trouble with my account User: I can't log in with my password User: My email is user@example.com User: I already tried resetting it twice providers: - id: elevenlabs:agents config: # Create an ephemeral agent for testing agentConfig: name: Support Agent prompt: | You are a helpful customer support agent for TechCorp. Your job is to: 1. Greet customers warmly 2. Understand their issue 3. Collect necessary information (email, account number) 4. Provide clear next steps 5. Maintain a professional, empathetic tone Never make promises you can't keep. Always set clear expectations. voiceId: 21m00Tcm4TlvDq8ikWAM # Rachel llmModel: gpt-5-mini # Define evaluation criteria evaluationCriteria: - id: greeting name: greeting description: Agent greets the user warmly weight: 0.8 passingThreshold: 0.8 - id: information_gathering name: information_gathering description: Agent asks for email or account details weight: 1.0 passingThreshold: 0.9 - id: empathy name: empathy description: Agent acknowledges user frustration weight: 0.9 passingThreshold: 0.7 - id: next_steps name: next_steps description: Agent provides clear next steps weight: 1.0 passingThreshold: 0.9 - id: professionalism name: professionalism description: Agent maintains professional tone weight: 0.8 passingThreshold: 0.8 # Limit newly simulated turns, excluding the supplied conversation history maxTurns: 8 timeout: 60000 tests: - description: Agent passes all critical evaluation criteria assert: - type: javascript value: | const results = context.providerResponse.metadata?.evaluationResults; const required = ['information_gathering', 'next_steps', 'professionalism']; return Array.isArray(results) && required.every(id => results.some(result => result.criterion === id && result.passed === true) ); - description: Agent returns a conversation history assert: - type: javascript value: | const history = context.providerResponse.metadata?.conversationHistory; return Array.isArray(history) && history.length > 0; - description: Agent responds within reasonable time assert: - type: latency threshold: 60000 ``` `maxTurns` limits newly simulated turns. The returned conversation history can also include the supplied turns. Run the agent eval: ```sh promptfoo eval -c agent-test.yaml ``` ### Step 8: Review Agent Performance View detailed results: ```sh promptfoo view ``` In the web UI, you'll see: - Full conversation transcript - Evaluation criteria scores - Pass/fail for each criterion - Conversation duration and cost - Audio playback for each turn ## Part 5: Tool Mocking ### Step 9: Add Tool Mocking Use an existing ElevenLabs agent configured with an `order_lookup` tool that accepts an `order_number` string. Set its ID in `agentId` below; this example mocks that tool's response. Create `agent-with-tools.yaml`: ```yaml description: 'Test agent with order lookup tool' prompts: - | User: What's the status of my order? User: Order number ORDER-12345 providers: - id: elevenlabs:agents config: agentId: your-agent-id-with-order-lookup # Mock tool responses for testing toolMockConfig: order_lookup: returnValue: order_number: 'ORDER-12345' status: 'Shipped' tracking_number: '1Z999AA10123456784' evaluationCriteria: - id: uses_tool name: uses_tool description: Agent calls order_lookup for ORDER-12345. weight: 1.0 passingThreshold: 0.9 - id: provides_tracking name: provides_tracking description: Agent tells the user that tracking number is 1Z999AA10123456784. weight: 1.0 passingThreshold: 0.9 tests: - description: Agent successfully looks up order assert: - type: javascript value: | const results = context.providerResponse.metadata?.evaluationResults; const required = ['uses_tool', 'provides_tracking']; return Array.isArray(results) && required.every(id => results.some(result => result.criterion === id && result.passed === true) ); ``` Run with tool mocking: ```sh promptfoo eval -c agent-with-tools.yaml ``` ## Next Steps You've learned to: - ✅ Compare TTS models and voices - ✅ Customize voice settings for different scenarios - ✅ Test STT accuracy with WER calculation - ✅ Evaluate conversational agents with criteria - ✅ Mock tools for agent testing ### Explore More - **Audio processing**: Use isolation for noise removal - **Regression testing**: Track agent performance over time - **Production monitoring**: Set up continuous testing ### Example Projects Check out complete examples: - [examples/provider-elevenlabs/tts-advanced](https://github.com/promptfoo/promptfoo/tree/main/examples/provider-elevenlabs/tts-advanced) - [examples/provider-elevenlabs/agents](https://github.com/promptfoo/promptfoo/tree/main/examples/provider-elevenlabs/agents) ### Resources - [ElevenLabs Provider Reference](/docs/providers/elevenlabs) - [Promptfoo Documentation](https://www.promptfoo.dev/docs/intro) - [ElevenLabs API Docs](https://elevenlabs.io/docs) ## Troubleshooting ### Common Issues **Agent conversations timeout:** - Increase `maxTurns` and `timeout` in config - Simplify evaluation criteria - Use faster LLM models **High costs during testing:** - Use `gpt-5-mini` instead of `gpt-5` - Enable caching for repeated tests - Implement LLM cascading - Test with shorter prompts first **Evaluation criteria always failing:** - Start with simple, objective criteria - Lower passing thresholds during development - Review agent transcript to understand behavior - Add more specific criteria descriptions **Audio quality issues:** - Try different `outputFormat` settings - Adjust voice settings (stability, similarity_boost) - Test with different models - Compare `eleven_multilingual_v2` with Flash for your speech-quality requirements ### Getting Help - [GitHub Issues](https://github.com/promptfoo/promptfoo/issues) - [Discord Community](https://discord.gg/promptfoo) - [ElevenLabs Support](https://elevenlabs.io/support)