loc_az_hci/docs/temporary/TROUBLESHOOTING_VM_9000.md

Troubleshooting VM 9000 Creation - I/O Errors

Error Summary

The VM creation failed with multiple I/O errors when reading from the source image:

• qemu-img: error while reading at byte 130023424: Input/output error
• Transfer stopped at ~23% (138.0 MiB of 597.2 MiB)

Root Causes

1. Corrupted source image file - The uploaded image may be damaged
2. Disk I/O issues on Proxmox host - Storage problems on the Proxmox node
3. File location mismatch - File may be in wrong location or format
4. Incomplete upload - File transfer may have been interrupted

Diagnostic Steps

1. Check File Integrity on Proxmox Host

SSH into your Proxmox host and run:

# Check if file exists and its size
ls -lh /var/lib/vz/import/ubuntu-24.04-server-cloudimg-amd64.img.raw
ls -lh /var/lib/vz/template/iso/ubuntu-24.04-server-cloudimg-amd64.img

# Check file integrity (if file is readable)
file /var/lib/vz/import/ubuntu-24.04-server-cloudimg-amd64.img.raw
file /var/lib/vz/template/iso/ubuntu-24.04-server-cloudimg-amd64.img

# Try to read file metadata
qemu-img info /var/lib/vz/import/ubuntu-24.04-server-cloudimg-amd64.img.raw 2>&1
qemu-img info /var/lib/vz/template/iso/ubuntu-24.04-server-cloudimg-amd64.img 2>&1

2. Check Disk Health

# Check disk space
df -h /var/lib/vz

# Check for disk errors
dmesg | grep -i error | tail -20
dmesg | grep -i "i/o error" | tail -20

# Check storage pool health
pvesm status
lvdisplay | grep -A 10 "pve"

3. Verify File Checksum (if original available)

If you have the original file, compare checksums:

# On your local machine (if you have the original)
sha256sum ubuntu-24.04-server-cloudimg-amd64.img

# On Proxmox host
sha256sum /var/lib/vz/import/ubuntu-24.04-server-cloudimg-amd64.img.raw
sha256sum /var/lib/vz/template/iso/ubuntu-24.04-server-cloudimg-amd64.img

Quick Fix Script

Automated fix (recommended):

./scripts/fix-corrupted-image.sh

This script will:

1. Verify your local image (or download if missing)
2. Remove corrupted files on Proxmox host
3. Upload a fresh copy via SCP
4. Verify the uploaded image

Solutions

Solution 1: Re-upload the Image (Recommended)

1. Delete the corrupted file (on Proxmox host):

rm -f /var/lib/vz/import/ubuntu-24.04-server-cloudimg-amd64.img.raw
rm -f /var/lib/vz/template/iso/ubuntu-24.04-server-cloudimg-amd64.img

2. Re-download the image (on your local machine):

cd /home/intlc/projects/loc_az_hci
./scripts/download-ubuntu-cloud-image.sh 24.04

3. Upload via Proxmox Web UI:

   • Go to: Datacenter → local → Content → Upload
   • Select: downloads/ubuntu-24.04-server-cloudimg-amd64.img
   • Wait for upload to complete
   • Verify file appears in storage

4. Verify upload (on Proxmox host):

qemu-img info /var/lib/vz/template/iso/ubuntu-24.04-server-cloudimg-amd64.img

5. Retry VM creation using the steps in CREATE_VM_9000_STEPS.md

Solution 2: Use API/CLI to Upload (Alternative)

If Web UI upload fails, use command line:

# On Proxmox host, copy file to correct location
scp ubuntu-24.04-server-cloudimg-amd64.img root@<proxmox-ip>:/var/lib/vz/template/iso/

# Or use Proxmox API (from local machine with API access)
# See scripts/create-template-via-api.sh

Solution 3: Download Directly on Proxmox Host

# SSH into Proxmox host
cd /var/lib/vz/template/iso

# Download directly
wget https://cloud-images.ubuntu.com/releases/24.04/release/ubuntu-24.04-server-cloudimg-amd64.img

# Verify
qemu-img info ubuntu-24.04-server-cloudimg-amd64.img

Solution 4: Use Different Storage Location

If local storage has issues, try a different storage:

1. Check available storage:

pvesm status

2. Upload to different storage (e.g., local-lvm or NFS)

3. Create VM using different storage in the Disks tab

Solution 5: Check and Fix Storage Issues

If disk I/O errors persist:

# Check LVM status
vgdisplay
lvdisplay

# Check for filesystem errors
fsck -n /dev/pve/root  # Dry run, don't fix yet

# If errors found, schedule filesystem check on next reboot
touch /forcefsck
reboot

Prevention

1. Always verify uploads: Check file size and integrity after upload
2. Use checksums: Compare SHA256 checksums before and after upload
3. Monitor disk health: Regularly check dmesg for I/O errors
4. Use reliable storage: Prefer local-lvm or NFS over local if available

Alternative: Create VM from ISO Instead

If cloud image continues to fail, use ISO installation method:

1. Download Ubuntu Server ISO
2. Upload ISO to Proxmox
3. Create VM with ISO attached
4. Install Ubuntu manually
5. Configure Cloud-Init
6. Convert to template

See scripts/create-vms-from-iso.sh for automation.

Next Steps After Fix

Once the image is successfully uploaded and verified:

1. Follow CREATE_VM_9000_STEPS.md to create VM 9000
2. Configure Cloud-Init settings
3. Convert to template
4. Verify template works by cloning a test VM

Verification Scripts

After fixing the issue, verify everything is working:

# Verify image integrity on Proxmox host
./scripts/verify-proxmox-image.sh

# Or manually check (SSH into Proxmox)
qemu-img info /var/lib/vz/template/iso/ubuntu-24.04-server-cloudimg-amd64.img

Related Files

• CREATE_VM_9000_STEPS.md - Main creation steps
• scripts/fix-corrupted-image.sh - Automated fix script (use this first!)
• scripts/verify-proxmox-image.sh - Image verification script
• scripts/download-ubuntu-cloud-image.sh - Download script
• scripts/create-proxmox-template.sh - Template creation script
• docs/runbooks/proxmox-operations.md - General Proxmox operations