docs: Update VCR testing documentation and fix PEP 8 import order
- Update docs/vcr-testing.md with new PII sanitization features - Document transport_helpers.inject_transport() for simpler test setup - Add sanitize_cassettes.py script documentation - Update file structure to include all new components - Fix PEP 8: Move copy import to top of openai_compatible.py - Enhance security notes about automatic sanitization 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -8,22 +8,19 @@ The HTTP Transport Recorder captures and replays HTTP interactions at the transp
|
|||||||
- Cost-efficient testing of expensive APIs (record once, replay forever)
|
- Cost-efficient testing of expensive APIs (record once, replay forever)
|
||||||
- Deterministic tests with real API responses
|
- Deterministic tests with real API responses
|
||||||
- Seamless integration with httpx and OpenAI SDK
|
- Seamless integration with httpx and OpenAI SDK
|
||||||
|
- Automatic PII sanitization for secure recordings
|
||||||
|
|
||||||
## Quick Start
|
## Quick Start
|
||||||
|
|
||||||
```python
|
```python
|
||||||
from tests.http_transport_recorder import TransportFactory
|
from tests.transport_helpers import inject_transport
|
||||||
from providers import ModelProviderRegistry
|
|
||||||
|
|
||||||
# Setup transport recorder
|
# Simple one-line setup with automatic transport injection
|
||||||
cassette_path = "tests/openai_cassettes/my_test.json"
|
def test_expensive_api_call(monkeypatch):
|
||||||
transport = TransportFactory.create_transport(cassette_path)
|
inject_transport(monkeypatch, "tests/openai_cassettes/my_test.json")
|
||||||
|
|
||||||
# Inject into provider
|
# Make API calls - automatically recorded/replayed with PII sanitization
|
||||||
provider = ModelProviderRegistry.get_provider_for_model("o3-pro")
|
result = await chat_tool.execute({"prompt": "2+2?", "model": "o3-pro"})
|
||||||
provider._test_transport = transport
|
|
||||||
|
|
||||||
# Make API calls - automatically recorded/replayed
|
|
||||||
```
|
```
|
||||||
|
|
||||||
## How It Works
|
## How It Works
|
||||||
@@ -34,18 +31,36 @@ provider._test_transport = transport
|
|||||||
|
|
||||||
## Usage in Tests
|
## Usage in Tests
|
||||||
|
|
||||||
See `test_o3_pro_output_text_fix.py` for a complete example:
|
The `transport_helpers.inject_transport()` function simplifies test setup:
|
||||||
|
|
||||||
```python
|
```python
|
||||||
async def test_with_recording():
|
from tests.transport_helpers import inject_transport
|
||||||
# Transport factory auto-detects record vs replay mode
|
|
||||||
transport = TransportFactory.create_transport("tests/openai_cassettes/my_test.json")
|
|
||||||
provider._test_transport = transport
|
|
||||||
|
|
||||||
# Use normally - recording happens transparently
|
async def test_with_recording(monkeypatch):
|
||||||
|
# One-line setup - handles all transport injection complexity
|
||||||
|
inject_transport(monkeypatch, "tests/openai_cassettes/my_test.json")
|
||||||
|
|
||||||
|
# Use API normally - recording/replay happens transparently
|
||||||
result = await chat_tool.execute({"prompt": "2+2?", "model": "o3-pro"})
|
result = await chat_tool.execute({"prompt": "2+2?", "model": "o3-pro"})
|
||||||
```
|
```
|
||||||
|
|
||||||
|
For manual setup, see `test_o3_pro_output_text_fix.py`.
|
||||||
|
|
||||||
|
## Automatic PII Sanitization
|
||||||
|
|
||||||
|
All recordings are automatically sanitized to remove sensitive data:
|
||||||
|
|
||||||
|
- **API Keys & Tokens**: Bearer tokens, API keys, and auth headers
|
||||||
|
- **Personal Data**: Email addresses, IP addresses, phone numbers
|
||||||
|
- **URLs**: Sensitive query parameters and paths
|
||||||
|
- **Custom Patterns**: Add your own sanitization rules
|
||||||
|
|
||||||
|
Sanitization is enabled by default in `RecordingTransport`. To disable:
|
||||||
|
|
||||||
|
```python
|
||||||
|
transport = TransportFactory.create_transport(cassette_path, sanitize=False)
|
||||||
|
```
|
||||||
|
|
||||||
## File Structure
|
## File Structure
|
||||||
|
|
||||||
```
|
```
|
||||||
@@ -53,9 +68,32 @@ tests/
|
|||||||
├── openai_cassettes/ # Recorded API interactions
|
├── openai_cassettes/ # Recorded API interactions
|
||||||
│ └── *.json # Cassette files
|
│ └── *.json # Cassette files
|
||||||
├── http_transport_recorder.py # Transport implementation
|
├── http_transport_recorder.py # Transport implementation
|
||||||
|
├── pii_sanitizer.py # Automatic PII sanitization
|
||||||
|
├── transport_helpers.py # Simplified transport injection
|
||||||
|
├── sanitize_cassettes.py # Batch sanitization script
|
||||||
└── test_o3_pro_output_text_fix.py # Example usage
|
└── test_o3_pro_output_text_fix.py # Example usage
|
||||||
```
|
```
|
||||||
|
|
||||||
|
## Sanitizing Existing Cassettes
|
||||||
|
|
||||||
|
Use the `sanitize_cassettes.py` script to clean existing recordings:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Sanitize all cassettes (creates backups)
|
||||||
|
python tests/sanitize_cassettes.py
|
||||||
|
|
||||||
|
# Sanitize specific cassette
|
||||||
|
python tests/sanitize_cassettes.py tests/openai_cassettes/my_test.json
|
||||||
|
|
||||||
|
# Skip backup creation
|
||||||
|
python tests/sanitize_cassettes.py --no-backup
|
||||||
|
```
|
||||||
|
|
||||||
|
The script will:
|
||||||
|
- Create timestamped backups of original files
|
||||||
|
- Apply comprehensive PII sanitization
|
||||||
|
- Preserve JSON structure and functionality
|
||||||
|
|
||||||
## Cost Management
|
## Cost Management
|
||||||
|
|
||||||
- **One-time cost**: Initial recording only
|
- **One-time cost**: Initial recording only
|
||||||
@@ -76,12 +114,15 @@ python -m pytest tests/test_o3_pro_output_text_fix.py
|
|||||||
|
|
||||||
## Implementation Details
|
## Implementation Details
|
||||||
|
|
||||||
- **RecordingTransport**: Captures real HTTP calls
|
- **RecordingTransport**: Captures real HTTP calls with automatic PII sanitization
|
||||||
- **ReplayTransport**: Serves saved responses
|
- **ReplayTransport**: Serves saved responses from cassettes
|
||||||
- **TransportFactory**: Auto-selects mode based on cassette existence
|
- **TransportFactory**: Auto-selects mode based on cassette existence
|
||||||
- **PII Sanitization**: Automatically removes API keys from recordings
|
- **PIISanitizer**: Comprehensive sanitization of sensitive data (integrated by default)
|
||||||
|
|
||||||
**Security Note**: Always review new cassette files before committing to ensure no sensitive data is included.
|
**Security Note**: While recordings are automatically sanitized, always review new cassette files before committing. The sanitizer removes known patterns of sensitive data, but domain-specific secrets may need custom rules.
|
||||||
|
|
||||||
For implementation details, see `tests/http_transport_recorder.py`.
|
For implementation details, see:
|
||||||
|
- `tests/http_transport_recorder.py` - Core transport implementation
|
||||||
|
- `tests/pii_sanitizer.py` - Sanitization patterns and logic
|
||||||
|
- `tests/transport_helpers.py` - Simplified test integration
|
||||||
|
|
||||||
|
|||||||
@@ -1,6 +1,7 @@
|
|||||||
"""Base class for OpenAI-compatible API providers."""
|
"""Base class for OpenAI-compatible API providers."""
|
||||||
|
|
||||||
import base64
|
import base64
|
||||||
|
import copy
|
||||||
import ipaddress
|
import ipaddress
|
||||||
import logging
|
import logging
|
||||||
import os
|
import os
|
||||||
@@ -283,8 +284,6 @@ class OpenAICompatibleProvider(ModelProvider):
|
|||||||
Returns:
|
Returns:
|
||||||
dict: Sanitized copy of parameters safe for logging
|
dict: Sanitized copy of parameters safe for logging
|
||||||
"""
|
"""
|
||||||
import copy
|
|
||||||
|
|
||||||
sanitized = copy.deepcopy(params)
|
sanitized = copy.deepcopy(params)
|
||||||
|
|
||||||
# Sanitize messages content
|
# Sanitize messages content
|
||||||
|
|||||||
Reference in New Issue
Block a user