chore: consolidate local WIP (repo cleanup 20260707)

Co-authored-by: Cursor <cursoragent@cursor.com>

docs/AUDIT_FINDINGS_DETAILED.md

@@ -0,0 +1,302 @@
+# Detailed Audit Findings
+
+**Date**: 2025-12-13  
+**Audit Type**: Comprehensive Production System Audit  
+**Status**: 🔴 **CRITICAL ISSUES IDENTIFIED**
+
+---
+
+## Executive Summary
+
+### Mode: **PRODUCTION** ✅
+- System is in **PRODUCTION MODE** (not dry-run)
+- VMs deployed in `examples/production/` directory
+- All VMs marked with `environment: "production"`
+
+### Critical Status: **BLOCKED** 🔴
+- **0 VMs created** on Proxmox (all 30 VMs pending)
+- **Authentication errors** blocking VM creation
+- **Resource over-allocation** detected
+
+---
+
+## 🔴 CRITICAL ISSUES
+
+### 1. Authentication Errors (BLOCKING)
+**Severity**: 🔴 **CRITICAL**  
+**Status**: **ACTIVE**  
+**Impact**: **BLOCKS ALL VM CREATION**
+
+**Details**:
+- Recent logs show "invalid PVE ticket" errors
+- Node health checks failing for both ML110-01 and R630-01
+- Error: `401 permission denied - invalid PVE ticket`
+- Token authentication being used but still failing
+
+**Evidence**:
+```
+2025-12-13T17:57:01Z ERROR node health check failed
+GET /nodes/r630-01/status failed: 401 permission denied - invalid PVE ticket
+```
+
+**Root Cause**:
+- Cookie header fix may not be working correctly
+- Token may be invalid or expired
+- Provider may not be using latest code
+
+**Action Required**: **URGENT**
+1. Verify token is valid and not expired
+2. Test token authentication manually
+3. Verify provider is using latest code
+4. Check token permissions on Proxmox
+
+---
+
+### 2. No VMs Created (BLOCKED)
+**Severity**: 🔴 **CRITICAL**  
+**Status**: **BLOCKED**  
+**Impact**: **ZERO PROGRESS**
+
+**Details**:
+- Total VMs: 30
+- VMs with VMID: 0
+- All VMs stuck in pending state
+- Blocked by authentication errors
+
+**Action Required**: **URGENT**
+- Fix authentication issue (Issue #1)
+- Once fixed, VMs should start creating
+
+---
+
+### 3. Resource Over-Allocation
+**Severity**: 🟡 **HIGH**  
+**Status**: **DETECTED**  
+**Impact**: **POTENTIAL DEPLOYMENT FAILURES**
+
+**Details**:
+
+#### ML110-01
+- **Allocated**: 24 CPU cores
+- **Available**: 6 CPU cores
+- **Over-allocation**: **4x** (400% over capacity)
+- **Memory Allocated**: 48 GiB
+- **Memory Available**: 256 GiB
+- **Memory Status**: ✅ Within limits
+
+#### R630-01
+- **Allocated**: 62 CPU cores
+- **Available**: 52 CPU cores
+- **Over-allocation**: **1.2x** (20% over capacity)
+- **Memory Allocated**: 220 GiB
+- **Memory Available**: 768 GiB
+- **Memory Status**: ✅ Within limits
+
+**Action Required**: **HIGH PRIORITY**
+1. Review VM CPU allocations
+2. Reduce CPU allocations to fit within node capacity
+3. Consider moving more VMs to R630-01
+4. Verify deployment plan matches actual allocations
+
+---
+
+### 4. VM Without Node Assignment
+**Severity**: 🟡 **MEDIUM**  
+**Status**: **DETECTED**  
+**Impact**: **VM CANNOT BE CREATED**
+
+**Details**:
+- 1 VM found without node assignment
+- VM cannot be created without node specification
+
+**Action Required**: **MEDIUM PRIORITY**
+1. Identify the VM without node
+2. Assign appropriate node
+3. Verify all VMs have node assignments
+
+---
+
+## 🟡 MEDIUM PRIORITY ISSUES
+
+### 5. ProviderConfig Name Mismatch
+**Severity**: 🟡 **MEDIUM**  
+**Status**: **DETECTED**  
+**Impact**: **CONFIGURATION CONFUSION**
+
+**Details**:
+- ProviderConfig name: `proxmox-provider-config`
+- Scripts may reference `default`
+- No functional impact but causes confusion
+
+**Action Required**: **LOW PRIORITY**
+- Update documentation to use correct name
+- Verify all references use correct name
+
+---
+
+### 6. Network Connectivity Test Failed
+**Severity**: 🟡 **MEDIUM**  
+**Status**: **INCONCLUSIVE**  
+**Impact**: **UNCERTAIN**
+
+**Details**:
+- Connectivity test command had parsing issues
+- Actual connectivity status unknown
+- Provider logs show connection attempts
+
+**Action Required**: **MEDIUM PRIORITY**
+- Test connectivity manually
+- Verify endpoints are reachable
+- Check network configuration
+
+---
+
+## ✅ POSITIVE FINDINGS
+
+### 1. Production Mode Confirmed ✅
+- No dry-run mode detected
+- All VMs in production directory
+- Environment tags set to "production"
+
+### 2. Provider Running ✅
+- Provider pod: Running (1/1 Ready)
+- Age: 32+ minutes
+- No crashes or restarts
+
+### 3. Configuration Present ✅
+- ProviderConfig exists
+- Credentials secret exists
+- All 30 VMs deployed
+
+### 4. Storage Configuration ✅
+- Storage types properly configured
+- 21 VMs using ceph-fs
+- 9 VMs using local-lvm
+
+### 5. VM Specifications Complete ✅
+- All VMs have required fields (except 1 without node)
+- CPU, memory, storage specified
+- Network configuration present
+
+---
+
+## 📊 Resource Allocation Summary
+
+### ML110-01
+- **CPU**: 24 cores allocated / 6 available (**4x OVER**)
+- **Memory**: 48 GiB allocated / 256 GiB available ✅
+- **VMs**: 8 VMs
+
+### R630-01
+- **CPU**: 62 cores allocated / 52 available (**1.2x OVER**)
+- **Memory**: 220 GiB allocated / 768 GiB available ✅
+- **VMs**: 22 VMs
+
+---
+
+## 🎯 Immediate Action Items
+
+### Priority 1: CRITICAL (Blocking)
+1. 🔴 **Fix authentication errors**
+   - Verify token validity
+   - Test token manually
+   - Rebuild provider if needed
+   - Restart provider pod
+
+2. 🔴 **Verify VM creation can proceed**
+   - Once auth fixed, monitor VM creation
+   - Verify VMs get VMIDs assigned
+
+### Priority 2: HIGH (Before Production)
+3. 🟡 **Fix resource over-allocation**
+   - Reduce ML110-01 CPU allocation (24 → 6 cores max)
+   - Reduce R630-01 CPU allocation (62 → 52 cores max)
+   - Update VM specifications
+   - Re-apply VM configurations
+
+4. 🟡 **Fix VM without node**
+   - Identify and fix VM missing node assignment
+
+### Priority 3: MEDIUM (Operational)
+5. 🟡 **Test network connectivity**
+   - Verify Proxmox endpoints accessible
+   - Test from provider pod
+   - Verify network configuration
+
+6. 🟡 **Update documentation**
+   - Fix ProviderConfig name references
+   - Update deployment procedures
+
+---
+
+## 📋 Recommendations
+
+### Immediate (Today)
+1. Fix authentication issue - **BLOCKING**
+2. Verify VM creation starts - **BLOCKING**
+3. Fix resource over-allocation - **HIGH PRIORITY**
+
+### Short-term (This Week)
+1. Complete network connectivity testing
+2. Fix VM without node assignment
+3. Update documentation
+4. Implement monitoring alerts
+
+### Long-term (Ongoing)
+1. Set up automated health checks
+2. Implement resource monitoring
+3. Create deployment runbooks
+4. Set up alerting for critical issues
+
+---
+
+## 🔍 Detailed Error Analysis
+
+### Authentication Errors
+- **Frequency**: Continuous
+- **Pattern**: All node health checks failing
+- **Error**: `401 permission denied - invalid PVE ticket`
+- **Impact**: Blocks all VM creation
+
+### Node Health Check Failures
+- **ML110-01**: Failing
+- **R630-01**: Failing
+- **Root Cause**: Authentication errors
+- **Impact**: Cannot verify node health before VM creation
+
+---
+
+## 📈 Progress Tracking
+
+### Current Status
+- **VMs Deployed**: 30/30 (100%)
+- **VMs Created**: 0/30 (0%)
+- **Progress**: **0%** (BLOCKED)
+
+### Expected Progress
+- **Per VM**: 2-5 minutes
+- **Total Time**: 30-60 minutes
+- **Current**: **BLOCKED** by authentication
+
+---
+
+## ✅ Verification Checklist
+
+- [ ] Authentication working
+- [ ] Provider running
+- [ ] ProviderConfig configured
+- [ ] Credentials secret present
+- [ ] VMs deployed
+- [ ] Resource allocation within limits
+- [ ] Network connectivity verified
+- [ ] Storage configured
+- [ ] VM specifications complete
+- [ ] VMs being created
+
+**Current Status**: **3/10 Complete** (30%)
+
+---
+
+**Last Updated**: 2025-12-13  
+**Status**: 🔴 **CRITICAL ISSUES - ACTION REQUIRED**
+