Complete markdown files cleanup and organization
- Organized 252 files across project - Root directory: 187 → 2 files (98.9% reduction) - Moved configuration guides to docs/04-configuration/ - Moved troubleshooting guides to docs/09-troubleshooting/ - Moved quick start guides to docs/01-getting-started/ - Moved reports to reports/ directory - Archived temporary files - Generated comprehensive reports and documentation - Created maintenance scripts and guides All files organized according to established standards.
This commit is contained in:
defiQUG
377
docs/OUTSTANDING_ISSUES_RESOLUTION_GUIDE.md
Normal file
377
docs/OUTSTANDING_ISSUES_RESOLUTION_GUIDE.md
Normal file
@@ -0,0 +1,377 @@
|
||||
# Outstanding Issues Resolution Guide
|
||||
|
||||
**Date:** 2026-01-03
|
||||
**Status:** Requires Physical/Console Access
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
Two servers require physical or console access to complete setup:
|
||||
|
||||
1. **r630-03** (192.168.11.13) - Network connectivity issue
|
||||
2. **r630-04** (192.168.11.14) - SSH password authentication failure
|
||||
|
||||
---
|
||||
|
||||
## Issue 1: r630-03 Network Connectivity
|
||||
|
||||
### Symptoms
|
||||
- Host does not respond to ping
|
||||
- "Destination Host Unreachable" error
|
||||
- Cannot access via SSH or web interface
|
||||
|
||||
### Diagnosis
|
||||
```bash
|
||||
# From your machine
|
||||
ping -c 3 192.168.11.13
|
||||
# Result: 100% packet loss, Destination Host Unreachable
|
||||
```
|
||||
|
||||
### Possible Causes
|
||||
1. **Server powered off**
|
||||
- Check physical power status
|
||||
- Check power LED indicators
|
||||
- Verify power cable connections
|
||||
|
||||
2. **Network cable disconnected**
|
||||
- Check physical network cable
|
||||
- Verify cable is connected to correct switch port
|
||||
- Test with known-good cable
|
||||
|
||||
3. **Network switch port issue**
|
||||
- Check switch port status
|
||||
- Verify port is enabled
|
||||
- Check for port errors on switch
|
||||
|
||||
4. **IP configuration issue**
|
||||
- Server may have different IP
|
||||
- Network configuration may be incorrect
|
||||
- DHCP may have assigned different IP
|
||||
|
||||
5. **Network routing issue**
|
||||
- Check router/switch configuration
|
||||
- Verify VLAN configuration
|
||||
- Check firewall rules
|
||||
|
||||
### Resolution Steps
|
||||
|
||||
#### Step 1: Physical Inspection
|
||||
1. **Check Power Status**
|
||||
- Verify server is powered on
|
||||
- Check power LED indicators
|
||||
- Listen for fan noise
|
||||
|
||||
2. **Check Network Cable**
|
||||
- Verify cable is connected
|
||||
- Check for physical damage
|
||||
- Try different cable if available
|
||||
|
||||
3. **Check Network Switch**
|
||||
- Verify switch port is active
|
||||
- Check for port errors
|
||||
- Verify VLAN configuration
|
||||
|
||||
#### Step 2: Access via iDRAC (Dell R630)
|
||||
If iDRAC is configured, access via:
|
||||
- **iDRAC IP:** Check network configuration or use default
|
||||
- **Default credentials:** root/calvin (if not changed)
|
||||
- **Access method:** Web browser to iDRAC IP
|
||||
|
||||
Once in iDRAC:
|
||||
1. Check server power status
|
||||
2. Access console (KVM)
|
||||
3. Check network configuration
|
||||
4. Verify IP address assignment
|
||||
|
||||
#### Step 3: Console Access
|
||||
If physical access is available:
|
||||
1. Connect keyboard/monitor
|
||||
2. Boot server if needed
|
||||
3. Check network configuration:
|
||||
```bash
|
||||
ip addr show
|
||||
cat /etc/network/interfaces
|
||||
```
|
||||
4. Verify IP address: `192.168.11.13`
|
||||
5. Test connectivity: `ping 192.168.11.1`
|
||||
|
||||
#### Step 4: Network Configuration Fix
|
||||
If IP is incorrect or network is misconfigured:
|
||||
```bash
|
||||
# Edit network configuration
|
||||
nano /etc/network/interfaces
|
||||
|
||||
# Ensure configuration matches:
|
||||
# auto vmbr0
|
||||
# iface vmbr0 inet static
|
||||
# address 192.168.11.13/24
|
||||
# gateway 192.168.11.1
|
||||
# bridge-ports <physical-interface>
|
||||
# bridge-vlan-aware yes
|
||||
|
||||
# Restart networking
|
||||
systemctl restart networking
|
||||
```
|
||||
|
||||
#### Step 5: Verification
|
||||
Once accessible:
|
||||
```bash
|
||||
# Run verification script
|
||||
./scripts/verify-r630-03-cluster-storage.sh
|
||||
|
||||
# Check cluster membership
|
||||
ssh root@192.168.11.13 "pvecm status"
|
||||
|
||||
# Verify storage
|
||||
ssh root@192.168.11.13 "pvesm status"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Issue 2: r630-04 SSH Password Authentication
|
||||
|
||||
### Symptoms
|
||||
- Host is reachable (ping works)
|
||||
- SSH connection attempts fail with "Permission denied"
|
||||
- All known passwords fail
|
||||
- Web interface not accessible (port 8006)
|
||||
|
||||
### Diagnosis
|
||||
```bash
|
||||
# Host is reachable
|
||||
ping -c 2 192.168.11.14
|
||||
# Result: Success
|
||||
|
||||
# SSH fails
|
||||
ssh root@192.168.11.14
|
||||
# Result: Permission denied (password authentication)
|
||||
```
|
||||
|
||||
### Tried Passwords
|
||||
- ❌ L@kers2010
|
||||
- ❌ password
|
||||
- ❌ L@kers2010!
|
||||
- ❌ L@kers2010@
|
||||
- ❌ L@kers2010#
|
||||
- ❌ All variations tested
|
||||
|
||||
### Resolution Steps
|
||||
|
||||
#### Step 1: Console Access (Required)
|
||||
Physical or iDRAC console access is required to reset password.
|
||||
|
||||
**Option A: Physical Console**
|
||||
1. Connect keyboard/monitor to r630-04
|
||||
2. Boot server if needed
|
||||
3. Login with current password (if known)
|
||||
4. Or boot into single-user mode
|
||||
|
||||
**Option B: iDRAC Console**
|
||||
1. Access iDRAC web interface
|
||||
2. Use Remote Console (KVM)
|
||||
3. Access server console remotely
|
||||
|
||||
#### Step 2: Reset Root Password
|
||||
|
||||
**Method 1: If you can login**
|
||||
```bash
|
||||
# Login to console
|
||||
# Then run:
|
||||
passwd root
|
||||
# Enter new password: L@kers2010
|
||||
# Confirm password
|
||||
```
|
||||
|
||||
**Method 2: Single-User Mode (if password unknown)**
|
||||
1. Boot server
|
||||
2. At GRUB menu, press 'e' to edit
|
||||
3. Find line starting with "linux"
|
||||
4. Add `init=/bin/bash` or `single` to end of line
|
||||
5. Press Ctrl+X to boot
|
||||
6. Mount filesystem: `mount -o remount,rw /`
|
||||
7. Reset password: `passwd root`
|
||||
8. Reboot: `reboot -f`
|
||||
|
||||
#### Step 3: Fix Proxmox Services
|
||||
|
||||
Once password is reset and SSH works:
|
||||
```bash
|
||||
# Run the complete fix script
|
||||
./scripts/fix-r630-04-complete.sh L@kers2010
|
||||
|
||||
# Or manually:
|
||||
ssh root@192.168.11.14
|
||||
|
||||
# Fix /etc/hosts
|
||||
echo "192.168.11.14 r630-04 r630-04.sankofa.nexus" >> /etc/hosts
|
||||
|
||||
# Restart Proxmox services
|
||||
systemctl restart pve-cluster
|
||||
systemctl restart pvestatd pvedaemon pveproxy
|
||||
|
||||
# Verify
|
||||
systemctl status pveproxy
|
||||
ss -tlnp | grep 8006
|
||||
curl -k https://localhost:8006
|
||||
```
|
||||
|
||||
#### Step 4: Join Cluster (if not already)
|
||||
|
||||
If r630-04 is not in cluster:
|
||||
```bash
|
||||
# From ml110, get join information
|
||||
ssh root@192.168.11.10 "pvecm create join-info"
|
||||
|
||||
# On r630-04, join cluster
|
||||
ssh root@192.168.11.14
|
||||
pvecm add <join-info-from-ml110>
|
||||
```
|
||||
|
||||
#### Step 5: Verify Storage
|
||||
|
||||
```bash
|
||||
# Check storage status
|
||||
ssh root@192.168.11.14 "pvesm status"
|
||||
|
||||
# Activate storage if needed
|
||||
# (Similar to r630-01 and r630-02)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Alternative Access Methods
|
||||
|
||||
### iDRAC Access (Dell R630)
|
||||
|
||||
Both r630-03 and r630-04 are Dell R630 servers with iDRAC.
|
||||
|
||||
**Default iDRAC Configuration:**
|
||||
- **IP Range:** Usually 192.168.0.x or DHCP assigned
|
||||
- **Default User:** root
|
||||
- **Default Password:** calvin (if not changed)
|
||||
|
||||
**Finding iDRAC IP:**
|
||||
1. Check network documentation
|
||||
2. Check DHCP leases on router
|
||||
3. Check Omada controller for connected devices
|
||||
4. Use default IP range: 192.168.0.100-200
|
||||
|
||||
**Accessing iDRAC:**
|
||||
1. Open web browser
|
||||
2. Navigate to iDRAC IP
|
||||
3. Login with credentials
|
||||
4. Use Remote Console (KVM) for full access
|
||||
|
||||
### Network-Based Solutions
|
||||
|
||||
**Option 1: Check from Working Nodes**
|
||||
```bash
|
||||
# From ml110, r630-01, or r630-02
|
||||
ssh root@192.168.11.10
|
||||
ping 192.168.11.13 # r630-03
|
||||
ping 192.168.11.14 # r630-04
|
||||
|
||||
# If reachable from cluster network, try SSH
|
||||
ssh root@192.168.11.14
|
||||
```
|
||||
|
||||
**Option 2: Check Network Switch**
|
||||
- Access switch management interface
|
||||
- Check port status for servers
|
||||
- Verify VLAN configuration
|
||||
- Check for port errors
|
||||
|
||||
**Option 3: Check Router/Firewall**
|
||||
- Verify firewall rules
|
||||
- Check routing tables
|
||||
- Verify NAT configuration
|
||||
|
||||
---
|
||||
|
||||
## Verification Checklist
|
||||
|
||||
### r630-03 Checklist
|
||||
- [ ] Server is powered on
|
||||
- [ ] Network cable connected
|
||||
- [ ] Network switch port active
|
||||
- [ ] IP address correct (192.168.11.13)
|
||||
- [ ] Ping works from cluster network
|
||||
- [ ] SSH access works
|
||||
- [ ] Proxmox services running
|
||||
- [ ] Web interface accessible (port 8006)
|
||||
- [ ] Cluster membership verified
|
||||
- [ ] Storage configuration verified
|
||||
|
||||
### r630-04 Checklist
|
||||
- [ ] Console access obtained
|
||||
- [ ] Root password reset
|
||||
- [ ] SSH access works
|
||||
- [ ] /etc/hosts configured correctly
|
||||
- [ ] Proxmox services running
|
||||
- [ ] Web interface accessible (port 8006)
|
||||
- [ ] Cluster membership verified
|
||||
- [ ] Storage configuration verified
|
||||
|
||||
---
|
||||
|
||||
## Scripts Available
|
||||
|
||||
All scripts are in `/home/intlc/projects/proxmox/scripts/`:
|
||||
|
||||
1. **`verify-r630-03-cluster-storage.sh`**
|
||||
- Run once r630-03 is accessible
|
||||
- Verifies cluster membership and storage
|
||||
|
||||
2. **`fix-r630-04-complete.sh`**
|
||||
- Run once r630-04 password is reset
|
||||
- Complete fix for all r630-04 issues
|
||||
|
||||
3. **`verify-all-nodes-complete.sh`**
|
||||
- Run after both issues are resolved
|
||||
- Comprehensive verification of all nodes
|
||||
|
||||
---
|
||||
|
||||
## Next Steps
|
||||
|
||||
### Immediate Actions
|
||||
1. **Physical Access Required**
|
||||
- Access r630-03 and r630-04 via console/iDRAC
|
||||
- Resolve network/password issues
|
||||
- Run verification scripts
|
||||
|
||||
2. **Documentation Update**
|
||||
- Update passwords in documentation
|
||||
- Update status reports
|
||||
- Document any configuration changes
|
||||
|
||||
### After Resolution
|
||||
1. **Run Verification**
|
||||
```bash
|
||||
./scripts/verify-all-nodes-complete.sh
|
||||
```
|
||||
|
||||
2. **Update Status Report**
|
||||
- Mark issues as resolved
|
||||
- Update cluster membership
|
||||
- Update storage status
|
||||
|
||||
3. **Cluster Health Check**
|
||||
- Verify all 5 nodes in cluster
|
||||
- Check quorum status
|
||||
- Verify storage on all nodes
|
||||
|
||||
---
|
||||
|
||||
## Support Resources
|
||||
|
||||
- **Console Access Guide:** `R630-04-CONSOLE-ACCESS-GUIDE.md`
|
||||
- **Troubleshooting Guide:** `R630-04-PROXMOX-TROUBLESHOOTING.md`
|
||||
- **Status Report:** `docs/PROXMOX_CLUSTER_STORAGE_STATUS_REPORT.md`
|
||||
- **Hardware Inventory:** `config/physical-hardware-inventory.md`
|
||||
|
||||
---
|
||||
|
||||
**Last Updated:** 2026-01-03
|
||||
**Status:** Awaiting Physical/Console Access
|
||||
Reference in New Issue
Block a user