135 lines
2.5 KiB
Markdown
135 lines
2.5 KiB
Markdown
|
# Force Unlock VM Instructions
|
||
|
|
|
||
|
|
**Date**: 2025-12-09
|
||
|
|
**Issue**: `qm unlock 100` is timing out
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Problem
|
||
|
|
|
||
|
|
The `qm unlock` command is timing out, which indicates:
|
||
|
|
- A stuck process is holding the lock
|
||
|
|
- The lock file is corrupted or in an invalid state
|
||
|
|
- Another operation is blocking the unlock
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Solution: Force Unlock
|
||
|
|
|
||
|
|
### Option 1: Use the Script (Recommended)
|
||
|
|
|
||
|
|
**On Proxmox Node (root@ml110-01)**:
|
||
|
|
|
||
|
|
```bash
|
||
|
|
# Copy the script to the Proxmox node
|
||
|
|
# Or run commands manually (see Option 2)
|
||
|
|
|
||
|
|
# Run the script
|
||
|
|
bash force-unlock-vm-proxmox.sh 100
|
||
|
|
```
|
||
|
|
|
||
|
|
### Option 2: Manual Commands
|
||
|
|
|
||
|
|
**On Proxmox Node (root@ml110-01)**:
|
||
|
|
|
||
|
|
```bash
|
||
|
|
# 1. Check for stuck processes
|
||
|
|
ps aux | grep -E 'qm|qemu' | grep 100
|
||
|
|
|
||
|
|
# 2. Check lock file
|
||
|
|
ls -la /var/lock/qemu-server/lock-100.conf
|
||
|
|
cat /var/lock/qemu-server/lock-100.conf 2>/dev/null
|
||
|
|
|
||
|
|
# 3. Kill stuck processes (if found)
|
||
|
|
pkill -9 -f 'qm.*100'
|
||
|
|
pkill -9 -f 'qemu.*100'
|
||
|
|
|
||
|
|
# 4. Wait a moment
|
||
|
|
sleep 2
|
||
|
|
|
||
|
|
# 5. Force remove lock file
|
||
|
|
rm -f /var/lock/qemu-server/lock-100.conf
|
||
|
|
|
||
|
|
# 6. Verify lock is gone
|
||
|
|
ls -la /var/lock/qemu-server/lock-100.conf
|
||
|
|
# Should show: No such file or directory
|
||
|
|
|
||
|
|
# 7. Check VM status
|
||
|
|
qm status 100
|
||
|
|
|
||
|
|
# 8. Try unlock again (should work now)
|
||
|
|
qm unlock 100
|
||
|
|
```
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## If Lock Persists
|
||
|
|
|
||
|
|
### Check for Other Issues
|
||
|
|
|
||
|
|
```bash
|
||
|
|
# Check if VM is in a transitional state
|
||
|
|
qm status 100
|
||
|
|
|
||
|
|
# Check VM configuration
|
||
|
|
qm config 100
|
||
|
|
|
||
|
|
# Check for other locks
|
||
|
|
ls -la /var/lock/qemu-server/lock-*.conf
|
||
|
|
|
||
|
|
# Check system resources
|
||
|
|
df -h
|
||
|
|
free -h
|
||
|
|
```
|
||
|
|
|
||
|
|
### Nuclear Option: Restart Proxmox Services
|
||
|
|
|
||
|
|
**⚠️ WARNING: This will affect all VMs on the node**
|
||
|
|
|
||
|
|
```bash
|
||
|
|
# Only if absolutely necessary
|
||
|
|
systemctl restart pve-cluster
|
||
|
|
systemctl restart pvedaemon
|
||
|
|
```
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## After Successful Unlock
|
||
|
|
|
||
|
|
1. **Monitor VM Status**:
|
||
|
|
```bash
|
||
|
|
qm status 100
|
||
|
|
```
|
||
|
|
|
||
|
|
2. **Check Provider Logs** (from Kubernetes):
|
||
|
|
```bash
|
||
|
|
kubectl logs -n crossplane-system -l app=crossplane-provider-proxmox --tail=50 -f
|
||
|
|
```
|
||
|
|
|
||
|
|
3. **Watch VM Resource**:
|
||
|
|
```bash
|
||
|
|
kubectl get proxmoxvm basic-vm-001 -w
|
||
|
|
```
|
||
|
|
|
||
|
|
4. **Expected Outcome**:
|
||
|
|
- Provider will retry within 1 minute
|
||
|
|
- VM configuration will complete
|
||
|
|
- VM will boot successfully
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Prevention
|
||
|
|
|
||
|
|
To prevent this issue in the future:
|
||
|
|
|
||
|
|
1. **Ensure proper VM shutdown** before operations
|
||
|
|
2. **Wait for operations to complete** before starting new ones
|
||
|
|
3. **Monitor for stuck processes** regularly
|
||
|
|
4. **Implement lock timeout handling** in provider code (already added)
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
**Last Updated**: 2025-12-09
|
||
|
|
**Status**: ⚠️ **MANUAL FORCE UNLOCK REQUIRED**
|
||
|
|
|