Sankofa/docs/proxmox/AUTHENTICATION_ERROR_ANALYSIS.md

Authentication Error Analysis - "invalid PVE ticket"

Date: 2025-12-13
Status: 🔍 INVESTIGATING

---

Current Situation

Error

• Message: "401 permission denied - invalid PVE ticket"
• Frequency: Continuous
• Impact: BLOCKS ALL VM CREATION
• Affected: Both ML110-01 and R630-01 nodes

Status

• ✅ Token tested manually - WORKS
• ✅ Code fix applied - Cookie header using Header.Set()
• ✅ Provider rebuilt and restarted
• ❌ Errors STILL OCCURRING

---

Investigation Results

1. Token Validity

• ✅ Token is valid - Manual curl test succeeds
• ✅ Token format: root@pam!sankofa-instance-1-api-token=73c7e1a2-c969-409c-ae5b-68e83f012ee9
• ✅ Token length: 36 characters (UUID format)

2. Code Implementation

• ✅ Token authentication code is correct
• ✅ Cookie header set using req.Header.Set("Cookie", fmt.Sprintf("PVEAuthCookie=%s", c.token))
• ✅ Debug logging confirms token is being sent
• ✅ Token format matches Proxmox expectations

3. Manual Testing

• ✅ curl -H "Cookie: PVEAuthCookie=$TOKEN" https://192.168.11.11:8006/api2/json/version WORKS
• ✅ Token authenticates successfully with curl

4. Provider Behavior

• ❌ Provider still getting "invalid PVE ticket" errors
• ❌ Node health checks failing
• ❌ VM creation blocked

---

Root Cause Hypothesis

Possible Causes

1. HTTP Client Differences

   • Go's http.Client might handle cookies differently than curl
   • Request headers might be formatted differently
   • TLS configuration might affect authentication

2. Token Permissions

   • Token might not have permissions for /nodes/status endpoint
   • Token might be restricted to certain operations
   • Token might need additional privileges

3. Time Synchronization

   • Proxmox nodes might have time sync issues
   • Token expiration might be affected by time drift
   • NTP synchronization might be required

4. Request Format

   • HTTP request format might differ between curl and Go
   • Headers might be processed differently
   • Cookie parsing might be different

---

Next Steps

1. Verify Token Permissions

   • Check token privileges in Proxmox
   • Verify token has access to node status endpoints
   • Test token with different endpoints

2. Check Time Synchronization

   • Verify NTP is working on Proxmox nodes
   • Check time sync between Kubernetes and Proxmox
   • Ensure clocks are synchronized

3. Compare HTTP Requests

   • Capture actual HTTP request from provider
   • Compare with working curl request
   • Identify differences in request format

4. Test Alternative Authentication

   • Try username/password authentication
   • Verify if issue is token-specific
   • Check if ticket authentication works

---

Debug Information

Token Format

TokenID: root@pam!sankofa-instance-1-api-token
Token: 73c7e1a2-c969-409c-ae5b-68e83f012ee9
Full Token: root@pam!sankofa-instance-1-api-token=73c7e1a2-c969-409c-ae5b-68e83f012ee9

Code Location

• File: crossplane-provider-proxmox/pkg/proxmox/http_client.go
• Line: 146
• Implementation: req.Header.Set("Cookie", fmt.Sprintf("PVEAuthCookie=%s", c.token))

Error Location

• File: crossplane-provider-proxmox/pkg/proxmox/client.go
• Line: 1119
• Function: CheckNodeHealth

---

Status

🔍 INVESTIGATION ONGOING

The token is valid and works with curl, but the provider is still failing. This suggests the issue is in how the HTTP request is constructed or processed by Go's HTTP client.

---

Last Updated: 2025-12-13
Status: 🔍 INVESTIGATING