Sankofa/docs/infrastructure/CEPH_INSTALLATION_ISSUE.md

Ceph Installation Issue - Package Version Conflict

Date: 2025-12-15
Status: ⚠️ BLOCKED - Package Version Conflict

Problem Summary

Attempting to install Ceph cluster services (ceph-mon, ceph-osd) is blocked by package version conflicts between Proxmox and Debian repositories.

Current State

Installed Packages (Proxmox Version)

• ceph-common: 19.2.3-pve2 (from Proxmox repository)
• ceph-fuse: 19.2.3-pve2
• Related libraries: libcephfs2, librados2, librbd1, python3-* packages all at 19.2.3-pve2

Available Packages (Debian Version)

• ceph-mon: 18.2.7+ds-1 (from Debian repository)
• ceph-osd: 18.2.7+ds-1
• Requires: ceph-common version 18.2.7+ds-1

Conflict

• Debian's ceph-mon and ceph-osd require ceph-common version 18.2.7+ds-1
• System has ceph-common version 19.2.3-pve2 installed
• Cannot downgrade due to dependency chain and Proxmox protection

Repository Issues

Enterprise Repository

• Status: Configured but requires subscription (401 Unauthorized)
• Location: https://enterprise.proxmox.com/debian/

No-Subscription Repository

• Status: SSL certificate verification failing
• Error: SSL connection failed: error:0A000086:SSL routines::certificate verify failed
• Location: https://download.proxmox.com/debian/

Debian Repository

• Status: ✅ Working
• Issue: Only has older Ceph version (18.2.7) that conflicts with installed Proxmox packages (19.2.3)

Attempted Solutions

1. Switch to No-Subscription Repository

• ✅ Updated repository configuration files
• ❌ SSL certificate verification failing
• Next Step: Fix SSL certificates or system clock

2. Downgrade to Debian Packages

• ❌ Blocked by dependency chain
• ❌ Proxmox protection prevents removing packages that would affect proxmox-ve meta-package
• Error: "You are attempting to remove the meta-package 'proxmox-ve'!"

3. Install Matching Proxmox Packages

• ❌ Cannot access Proxmox repositories (SSL/authentication issues)
• Requires: Fix repository access first

Root Cause

1. Mixed Package Sources: System has Proxmox Ceph packages installed but cannot access Proxmox repositories to get matching ceph-mon and ceph-osd packages
2. SSL Certificate Issue: Cannot verify Proxmox repository certificates
3. Version Mismatch: Debian repositories only have older Ceph version that conflicts with installed Proxmox version

Recommended Solutions

Option 1: Fix SSL Certificate Issue (Recommended)

# Check system time
date

# Update CA certificates
apt update && apt install -y ca-certificates

# Try repository access again
apt update
pveceph install

Option 2: Fix Repository Configuration

# Verify repository files are correct
cat /etc/apt/sources.list.d/pve-no-subscription.sources
cat /etc/apt/sources.list.d/ceph-no-subscription.sources

# Ensure correct components:
# - pve-no-subscription (not pve-enterprise)
# - no-subscription (not enterprise)

Option 3: Use Proxmox Web UI

• Access Proxmox web interface
• Use GUI to install Ceph (may handle repository issues automatically)
• Navigate to: Datacenter > Ceph > Install

Option 4: Manual Package Installation (Advanced)

If Proxmox repositories remain inaccessible:

1. Download Proxmox Ceph packages manually
2. Install with dpkg -i (may have dependency issues)
3. Or use apt with --allow-downgrades and --force-depends (risky)

Option 5: Fresh Ceph Installation

If cluster can be reinitialized:

1. Remove all Ceph packages (with Proxmox protection override)
2. Install Debian Ceph packages fresh
3. Initialize new cluster

Current Repository Configuration

ML110-01 and R630-01

/etc/apt/sources.list.d/pve-no-subscription.sources:
  Types: deb
  URIs: https://download.proxmox.com/debian/pve
  Suites: trixie
  Components: pve-no-subscription

/etc/apt/sources.list.d/ceph-no-subscription.sources:
  Types: deb
  URIs: https://download.proxmox.com/debian/ceph-squid
  Suites: trixie
  Components: no-subscription

Additional Findings

• System Date: Correct (Mon Dec 15 12:51:36 PM PST 2025)
• CA Certificates: Up to date
• SSL Error Persists: Even after certificate update
• 401 Unauthorized: Suggests authentication/authorization issue, not just SSL
• Possible Issue: "trixie" (Debian 13 testing) may not have full Proxmox repository support

Next Steps

1. Verify Proxmox Version and Repository Suite (Priority 1)

   • Check pveversion to determine correct repository suite
   • Verify if "trixie" is correct or should use "bookworm" or other
   • Update repository configuration if needed

2. Alternative: Use Proxmox Web UI (Priority 2)

   • Access Proxmox web interface
   • Install Ceph through GUI (may handle repository issues)
   • Navigate to: Datacenter > Ceph > Install

3. Manual Package Resolution (Priority 3)

   • If repositories remain inaccessible, consider:
     • Using Debian packages and accepting version downgrade
     • Or waiting for Proxmox repository access to be resolved

4. Alternative Installation Method (Priority 3)

   • Use Proxmox Web UI if CLI continues to fail
   • Or manually download and install packages

5. Document Resolution (Priority 4)

   • Update this document with solution
   • Update NEXT_STEPS.md with resolved path

Related Files

• docs/infrastructure/NEXT_STEPS.md - Overall next steps
• scripts/fresh-ceph-install-r630.sh - Fresh installation script
• /etc/apt/sources.list.d/ - Repository configuration

Notes

• Proxmox VE installation includes Ceph client tools by default
• Full Ceph cluster services require additional packages
• pveceph command should handle installation automatically if repositories are accessible
• SSL certificate issues may indicate system time is incorrect or certificates need updating