virtual-banker/NEXT_STEPS.md

Next Steps for Virtual Banker

✅ Completed Integration Steps

1. Service Integration: All services are now connected:

   • Orchestrator → LLM Gateway
   • Orchestrator → RAG Service
   • Orchestrator → Tool Executor
   • Banking tools → Banking API client

2. WebSocket Support: Realtime gateway integrated into API routes

3. Startup Scripts: Created scripts for database setup and backend startup

4. Banking Integration: Tools now connect to backend banking services with fallback

🔄 Remaining Integration Tasks

1. Replace Mock Services with Real APIs

ASR Service (Speech-to-Text):

• Integrate Deepgram API

  • Get API key from Deepgram
  • Update backend/asr/service.go to use Deepgram streaming API
  • Test with real audio streams

• Or integrate Google Speech-to-Text

  • Set up Google Cloud credentials
  • Implement streaming transcription

TTS Service (Text-to-Speech):

• Integrate ElevenLabs API

  • Get API key from ElevenLabs
  • Update backend/tts/service.go to use ElevenLabs API
  • Configure voice selection per tenant

• Or integrate Azure TTS

  • Set up Azure credentials
  • Implement SSML support

LLM Gateway:

• Integrate OpenAI API

  • Get API key
  • Update backend/llm/gateway.go to use OpenAI
  • Implement function calling
  • Add streaming support

• Or integrate Anthropic Claude

  • Get API key
  • Implement tool use

2. Complete WebRTC Implementation

• Implement full WebRTC signaling

  • SDP offer/answer exchange
  • ICE candidate handling
  • TURN server configuration

• Add media streaming

  • Audio stream from client → ASR
  • Audio stream from TTS → client
  • Video stream from avatar → client

3. Connect to Existing Banking Services

Update banking tool integrations to match actual API endpoints:

// Check actual endpoints in backend/banking/
// Update integration.go with correct paths

4. Unreal Engine Avatar Setup

• Install Unreal Engine 5.3+
• Create new project
• Enable PixelStreaming plugin
• Import digital human character
• Set up blendshapes for visemes
• Configure animation blueprints
• Package for Linux deployment

5. Testing

• Unit tests for all services
• Integration tests for API endpoints
• E2E tests for widget
• Load testing for concurrent sessions

6. Production Deployment

• Set up secrets management
• Configure monitoring (Prometheus/Grafana)
• Set up logging aggregation
• Configure auto-scaling
• Security audit
• Performance optimization

Quick Start Commands

# Setup database
cd virtual-banker
./scripts/setup-database.sh

# Start backend
./scripts/start-backend.sh

# Build widget
cd widget
npm install
npm run build

Environment Variables

Create .env file:

DATABASE_URL=postgres://explorer:changeme@localhost:5432/explorer?sslmode=disable
REDIS_URL=redis://localhost:6379
PORT=8081

# For real services (when ready):
DEEPGRAM_API_KEY=your_key_here
ELEVENLABS_API_KEY=your_key_here
OPENAI_API_KEY=your_key_here
BANKING_API_URL=http://localhost:8080

Testing the Integration

1. Start services:

   docker-compose up -d postgres redis
   ./scripts/start-backend.sh
   

2. Create a session:

   curl -X POST http://localhost:8081/v1/sessions \
     -H "Content-Type: application/json" \
     -d '{
       "tenant_id": "default",
       "user_id": "test-user",
       "auth_assertion": "test-token"
     }'
   

3. Test WebSocket:

   # Use wscat or similar tool
   wscat -c ws://localhost:8081/v1/realtime/{session_id}
   

Notes

• All mock services are functional and can be tested independently
• Banking tools have fallback to mock data if service unavailable
• WebRTC gateway is ready but needs full signaling implementation
• Widget is fully functional for text chat (voice requires WebRTC completion)