9daf1fd378
- Add comprehensive database migrations (001-024) for schema evolution - Enhance API schema with expanded type definitions and resolvers - Add new middleware: audit logging, rate limiting, MFA enforcement, security, tenant auth - Implement new services: AI optimization, billing, blockchain, compliance, marketplace - Add adapter layer for cloud integrations (Cloudflare, Kubernetes, Proxmox, storage) - Update Crossplane provider with enhanced VM management capabilities - Add comprehensive test suite for API endpoints and services - Update frontend components with improved GraphQL subscriptions and real-time updates - Enhance security configurations and headers (CSP, CORS, etc.) - Update documentation and configuration files - Add new CI/CD workflows and validation scripts - Implement design system improvements and UI enhancements
135 lines
2.5 KiB
Markdown
135 lines
2.5 KiB
Markdown
# Force Unlock VM Instructions
|
|
|
|
**Date**: 2025-12-09
|
|
**Issue**: `qm unlock 100` is timing out
|
|
|
|
---
|
|
|
|
## Problem
|
|
|
|
The `qm unlock` command is timing out, which indicates:
|
|
- A stuck process is holding the lock
|
|
- The lock file is corrupted or in an invalid state
|
|
- Another operation is blocking the unlock
|
|
|
|
---
|
|
|
|
## Solution: Force Unlock
|
|
|
|
### Option 1: Use the Script (Recommended)
|
|
|
|
**On Proxmox Node (root@ml110-01)**:
|
|
|
|
```bash
|
|
# Copy the script to the Proxmox node
|
|
# Or run commands manually (see Option 2)
|
|
|
|
# Run the script
|
|
bash force-unlock-vm-proxmox.sh 100
|
|
```
|
|
|
|
### Option 2: Manual Commands
|
|
|
|
**On Proxmox Node (root@ml110-01)**:
|
|
|
|
```bash
|
|
# 1. Check for stuck processes
|
|
ps aux | grep -E 'qm|qemu' | grep 100
|
|
|
|
# 2. Check lock file
|
|
ls -la /var/lock/qemu-server/lock-100.conf
|
|
cat /var/lock/qemu-server/lock-100.conf 2>/dev/null
|
|
|
|
# 3. Kill stuck processes (if found)
|
|
pkill -9 -f 'qm.*100'
|
|
pkill -9 -f 'qemu.*100'
|
|
|
|
# 4. Wait a moment
|
|
sleep 2
|
|
|
|
# 5. Force remove lock file
|
|
rm -f /var/lock/qemu-server/lock-100.conf
|
|
|
|
# 6. Verify lock is gone
|
|
ls -la /var/lock/qemu-server/lock-100.conf
|
|
# Should show: No such file or directory
|
|
|
|
# 7. Check VM status
|
|
qm status 100
|
|
|
|
# 8. Try unlock again (should work now)
|
|
qm unlock 100
|
|
```
|
|
|
|
---
|
|
|
|
## If Lock Persists
|
|
|
|
### Check for Other Issues
|
|
|
|
```bash
|
|
# Check if VM is in a transitional state
|
|
qm status 100
|
|
|
|
# Check VM configuration
|
|
qm config 100
|
|
|
|
# Check for other locks
|
|
ls -la /var/lock/qemu-server/lock-*.conf
|
|
|
|
# Check system resources
|
|
df -h
|
|
free -h
|
|
```
|
|
|
|
### Nuclear Option: Restart Proxmox Services
|
|
|
|
**⚠️ WARNING: This will affect all VMs on the node**
|
|
|
|
```bash
|
|
# Only if absolutely necessary
|
|
systemctl restart pve-cluster
|
|
systemctl restart pvedaemon
|
|
```
|
|
|
|
---
|
|
|
|
## After Successful Unlock
|
|
|
|
1. **Monitor VM Status**:
|
|
```bash
|
|
qm status 100
|
|
```
|
|
|
|
2. **Check Provider Logs** (from Kubernetes):
|
|
```bash
|
|
kubectl logs -n crossplane-system -l app=crossplane-provider-proxmox --tail=50 -f
|
|
```
|
|
|
|
3. **Watch VM Resource**:
|
|
```bash
|
|
kubectl get proxmoxvm basic-vm-001 -w
|
|
```
|
|
|
|
4. **Expected Outcome**:
|
|
- Provider will retry within 1 minute
|
|
- VM configuration will complete
|
|
- VM will boot successfully
|
|
|
|
---
|
|
|
|
## Prevention
|
|
|
|
To prevent this issue in the future:
|
|
|
|
1. **Ensure proper VM shutdown** before operations
|
|
2. **Wait for operations to complete** before starting new ones
|
|
3. **Monitor for stuck processes** regularly
|
|
4. **Implement lock timeout handling** in provider code (already added)
|
|
|
|
---
|
|
|
|
**Last Updated**: 2025-12-09
|
|
**Status**: ⚠️ **MANUAL FORCE UNLOCK REQUIRED**
|
|
|