# Authentication Error Analysis - "invalid PVE ticket"

**Date**: 2025-12-13  
**Status**: 🔍 **INVESTIGATING**

---

## Current Situation

### Error
- **Message**: "401 permission denied - invalid PVE ticket"
- **Frequency**: Continuous
- **Impact**: **BLOCKS ALL VM CREATION**
- **Affected**: Both ML110-01 and R630-01 nodes

### Status
- ✅ Token tested manually - **WORKS**
- ✅ Code fix applied - Cookie header using `Header.Set()`
- ✅ Provider rebuilt and restarted
- ❌ Errors **STILL OCCURRING**

---

## Investigation Results

### 1. Token Validity
- ✅ **Token is valid** - Manual curl test succeeds
- ✅ Token format: `root@pam!sankofa-instance-1-api-token=73c7e1a2-c969-409c-ae5b-68e83f012ee9`
- ✅ Token length: 36 characters (UUID format)

### 2. Code Implementation
- ✅ Token authentication code is correct
- ✅ Cookie header set using `req.Header.Set("Cookie", fmt.Sprintf("PVEAuthCookie=%s", c.token))`
- ✅ Debug logging confirms token is being sent
- ✅ Token format matches Proxmox expectations

### 3. Manual Testing
- ✅ `curl -H "Cookie: PVEAuthCookie=$TOKEN" https://192.168.11.11:8006/api2/json/version` **WORKS**
- ✅ Token authenticates successfully with curl

### 4. Provider Behavior
- ❌ Provider still getting "invalid PVE ticket" errors
- ❌ Node health checks failing
- ❌ VM creation blocked

---

## Root Cause Hypothesis

### Possible Causes

1. **HTTP Client Differences**
   - Go's `http.Client` might handle cookies differently than curl
   - Request headers might be formatted differently
   - TLS configuration might affect authentication

2. **Token Permissions**
   - Token might not have permissions for `/nodes/status` endpoint
   - Token might be restricted to certain operations
   - Token might need additional privileges

3. **Time Synchronization**
   - Proxmox nodes might have time sync issues
   - Token expiration might be affected by time drift
   - NTP synchronization might be required

4. **Request Format**
   - HTTP request format might differ between curl and Go
   - Headers might be processed differently
   - Cookie parsing might be different

---

## Next Steps

1. **Verify Token Permissions**
   - Check token privileges in Proxmox
   - Verify token has access to node status endpoints
   - Test token with different endpoints

2. **Check Time Synchronization**
   - Verify NTP is working on Proxmox nodes
   - Check time sync between Kubernetes and Proxmox
   - Ensure clocks are synchronized

3. **Compare HTTP Requests**
   - Capture actual HTTP request from provider
   - Compare with working curl request
   - Identify differences in request format

4. **Test Alternative Authentication**
   - Try username/password authentication
   - Verify if issue is token-specific
   - Check if ticket authentication works

---

## Debug Information

### Token Format
```
TokenID: root@pam!sankofa-instance-1-api-token
Token: 73c7e1a2-c969-409c-ae5b-68e83f012ee9
Full Token: root@pam!sankofa-instance-1-api-token=73c7e1a2-c969-409c-ae5b-68e83f012ee9
```

### Code Location
- **File**: `crossplane-provider-proxmox/pkg/proxmox/http_client.go`
- **Line**: 146
- **Implementation**: `req.Header.Set("Cookie", fmt.Sprintf("PVEAuthCookie=%s", c.token))`

### Error Location
- **File**: `crossplane-provider-proxmox/pkg/proxmox/client.go`
- **Line**: 1119
- **Function**: `CheckNodeHealth`

---

## Status

🔍 **INVESTIGATION ONGOING**

The token is valid and works with curl, but the provider is still failing. This suggests the issue is in how the HTTP request is constructed or processed by Go's HTTP client.

---

**Last Updated**: 2025-12-13  
**Status**: 🔍 **INVESTIGATING**

