# Next Steps for Virtual Banker

## ✅ Completed Integration Steps

1. **Service Integration**: All services are now connected:
   - Orchestrator → LLM Gateway
   - Orchestrator → RAG Service
   - Orchestrator → Tool Executor
   - Banking tools → Banking API client

2. **WebSocket Support**: Realtime gateway integrated into API routes

3. **Startup Scripts**: Created scripts for database setup and backend startup

4. **Banking Integration**: Tools now connect to backend banking services with fallback

## 🔄 Remaining Integration Tasks

### 1. Replace Mock Services with Real APIs

**ASR Service** (Speech-to-Text):
- [ ] Integrate Deepgram API
  - Get API key from Deepgram
  - Update `backend/asr/service.go` to use Deepgram streaming API
  - Test with real audio streams

- [ ] Or integrate Google Speech-to-Text
  - Set up Google Cloud credentials
  - Implement streaming transcription

**TTS Service** (Text-to-Speech):
- [ ] Integrate ElevenLabs API
  - Get API key from ElevenLabs
  - Update `backend/tts/service.go` to use ElevenLabs API
  - Configure voice selection per tenant

- [ ] Or integrate Azure TTS
  - Set up Azure credentials
  - Implement SSML support

**LLM Gateway**:
- [ ] Integrate OpenAI API
  - Get API key
  - Update `backend/llm/gateway.go` to use OpenAI
  - Implement function calling
  - Add streaming support

- [ ] Or integrate Anthropic Claude
  - Get API key
  - Implement tool use

### 2. Complete WebRTC Implementation

- [ ] Implement full WebRTC signaling
  - SDP offer/answer exchange
  - ICE candidate handling
  - TURN server configuration

- [ ] Add media streaming
  - Audio stream from client → ASR
  - Audio stream from TTS → client
  - Video stream from avatar → client

### 3. Connect to Existing Banking Services

Update banking tool integrations to match actual API endpoints:

```go
// Check actual endpoints in backend/banking/
// Update integration.go with correct paths
```

### 4. Unreal Engine Avatar Setup

- [ ] Install Unreal Engine 5.3+
- [ ] Create new project
- [ ] Enable PixelStreaming plugin
- [ ] Import digital human character
- [ ] Set up blendshapes for visemes
- [ ] Configure animation blueprints
- [ ] Package for Linux deployment

### 5. Testing

- [ ] Unit tests for all services
- [ ] Integration tests for API endpoints
- [ ] E2E tests for widget
- [ ] Load testing for concurrent sessions

### 6. Production Deployment

- [ ] Set up secrets management
- [ ] Configure monitoring (Prometheus/Grafana)
- [ ] Set up logging aggregation
- [ ] Configure auto-scaling
- [ ] Security audit
- [ ] Performance optimization

## Quick Start Commands

```bash
# Setup database
cd virtual-banker
./scripts/setup-database.sh

# Start backend
./scripts/start-backend.sh

# Build widget
cd widget
npm install
npm run build
```

## Environment Variables

Create `.env` file:

```bash
DATABASE_URL=postgres://explorer:changeme@localhost:5432/explorer?sslmode=disable
REDIS_URL=redis://localhost:6379
PORT=8081

# For real services (when ready):
DEEPGRAM_API_KEY=your_key_here
ELEVENLABS_API_KEY=your_key_here
OPENAI_API_KEY=your_key_here
BANKING_API_URL=http://localhost:8080
```

## Testing the Integration

1. **Start services**:
   ```bash
   docker-compose up -d postgres redis
   ./scripts/start-backend.sh
   ```

2. **Create a session**:
   ```bash
   curl -X POST http://localhost:8081/v1/sessions \
     -H "Content-Type: application/json" \
     -d '{
       "tenant_id": "default",
       "user_id": "test-user",
       "auth_assertion": "test-token"
     }'
   ```

3. **Test WebSocket**:
   ```bash
   # Use wscat or similar tool
   wscat -c ws://localhost:8081/v1/realtime/{session_id}
   ```

## Notes

- All mock services are functional and can be tested independently
- Banking tools have fallback to mock data if service unavailable
- WebRTC gateway is ready but needs full signaling implementation
- Widget is fully functional for text chat (voice requires WebRTC completion)