163 lines
3.9 KiB
Markdown
163 lines
3.9 KiB
Markdown
# Next Steps for Virtual Banker
|
|
|
|
## ✅ Completed Integration Steps
|
|
|
|
1. **Service Integration**: All services are now connected:
|
|
- Orchestrator → LLM Gateway
|
|
- Orchestrator → RAG Service
|
|
- Orchestrator → Tool Executor
|
|
- Banking tools → Banking API client
|
|
|
|
2. **WebSocket Support**: Realtime gateway integrated into API routes
|
|
|
|
3. **Startup Scripts**: Created scripts for database setup and backend startup
|
|
|
|
4. **Banking Integration**: Tools now connect to backend banking services with fallback
|
|
|
|
## 🔄 Remaining Integration Tasks
|
|
|
|
### 1. Replace Mock Services with Real APIs
|
|
|
|
**ASR Service** (Speech-to-Text):
|
|
- [ ] Integrate Deepgram API
|
|
- Get API key from Deepgram
|
|
- Update `backend/asr/service.go` to use Deepgram streaming API
|
|
- Test with real audio streams
|
|
|
|
- [ ] Or integrate Google Speech-to-Text
|
|
- Set up Google Cloud credentials
|
|
- Implement streaming transcription
|
|
|
|
**TTS Service** (Text-to-Speech):
|
|
- [ ] Integrate ElevenLabs API
|
|
- Get API key from ElevenLabs
|
|
- Update `backend/tts/service.go` to use ElevenLabs API
|
|
- Configure voice selection per tenant
|
|
|
|
- [ ] Or integrate Azure TTS
|
|
- Set up Azure credentials
|
|
- Implement SSML support
|
|
|
|
**LLM Gateway**:
|
|
- [ ] Integrate OpenAI API
|
|
- Get API key
|
|
- Update `backend/llm/gateway.go` to use OpenAI
|
|
- Implement function calling
|
|
- Add streaming support
|
|
|
|
- [ ] Or integrate Anthropic Claude
|
|
- Get API key
|
|
- Implement tool use
|
|
|
|
### 2. Complete WebRTC Implementation
|
|
|
|
- [ ] Implement full WebRTC signaling
|
|
- SDP offer/answer exchange
|
|
- ICE candidate handling
|
|
- TURN server configuration
|
|
|
|
- [ ] Add media streaming
|
|
- Audio stream from client → ASR
|
|
- Audio stream from TTS → client
|
|
- Video stream from avatar → client
|
|
|
|
### 3. Connect to Existing Banking Services
|
|
|
|
Update banking tool integrations to match actual API endpoints:
|
|
|
|
```go
|
|
// Check actual endpoints in backend/banking/
|
|
// Update integration.go with correct paths
|
|
```
|
|
|
|
### 4. Unreal Engine Avatar Setup
|
|
|
|
- [ ] Install Unreal Engine 5.3+
|
|
- [ ] Create new project
|
|
- [ ] Enable PixelStreaming plugin
|
|
- [ ] Import digital human character
|
|
- [ ] Set up blendshapes for visemes
|
|
- [ ] Configure animation blueprints
|
|
- [ ] Package for Linux deployment
|
|
|
|
### 5. Testing
|
|
|
|
- [ ] Unit tests for all services
|
|
- [ ] Integration tests for API endpoints
|
|
- [ ] E2E tests for widget
|
|
- [ ] Load testing for concurrent sessions
|
|
|
|
### 6. Production Deployment
|
|
|
|
- [ ] Set up secrets management
|
|
- [ ] Configure monitoring (Prometheus/Grafana)
|
|
- [ ] Set up logging aggregation
|
|
- [ ] Configure auto-scaling
|
|
- [ ] Security audit
|
|
- [ ] Performance optimization
|
|
|
|
## Quick Start Commands
|
|
|
|
```bash
|
|
# Setup database
|
|
cd virtual-banker
|
|
./scripts/setup-database.sh
|
|
|
|
# Start backend
|
|
./scripts/start-backend.sh
|
|
|
|
# Build widget
|
|
cd widget
|
|
npm install
|
|
npm run build
|
|
```
|
|
|
|
## Environment Variables
|
|
|
|
Create `.env` file:
|
|
|
|
```bash
|
|
DATABASE_URL=postgres://explorer:changeme@localhost:5432/explorer?sslmode=disable
|
|
REDIS_URL=redis://localhost:6379
|
|
PORT=8081
|
|
|
|
# For real services (when ready):
|
|
DEEPGRAM_API_KEY=your_key_here
|
|
ELEVENLABS_API_KEY=your_key_here
|
|
OPENAI_API_KEY=your_key_here
|
|
BANKING_API_URL=http://localhost:8080
|
|
```
|
|
|
|
## Testing the Integration
|
|
|
|
1. **Start services**:
|
|
```bash
|
|
docker-compose up -d postgres redis
|
|
./scripts/start-backend.sh
|
|
```
|
|
|
|
2. **Create a session**:
|
|
```bash
|
|
curl -X POST http://localhost:8081/v1/sessions \
|
|
-H "Content-Type: application/json" \
|
|
-d '{
|
|
"tenant_id": "default",
|
|
"user_id": "test-user",
|
|
"auth_assertion": "test-token"
|
|
}'
|
|
```
|
|
|
|
3. **Test WebSocket**:
|
|
```bash
|
|
# Use wscat or similar tool
|
|
wscat -c ws://localhost:8081/v1/realtime/{session_id}
|
|
```
|
|
|
|
## Notes
|
|
|
|
- All mock services are functional and can be tested independently
|
|
- Banking tools have fallback to mock data if service unavailable
|
|
- WebRTC gateway is ready but needs full signaling implementation
|
|
- Widget is fully functional for text chat (voice requires WebRTC completion)
|
|
|