Sankofa/docs/infrastructure/CEPH_INSTALLATION_ISSUE.md

# Ceph Installation Issue - Package Version Conflict

**Date**: 2025-12-15  
**Status**: ⚠️ **BLOCKED - Package Version Conflict**

## Problem Summary

Attempting to install Ceph cluster services (`ceph-mon`, `ceph-osd`) is blocked by package version conflicts between Proxmox and Debian repositories.

## Current State

### Installed Packages (Proxmox Version)
- `ceph-common`: 19.2.3-pve2 (from Proxmox repository)
- `ceph-fuse`: 19.2.3-pve2
- Related libraries: `libcephfs2`, `librados2`, `librbd1`, `python3-*` packages all at 19.2.3-pve2

### Available Packages (Debian Version)
- `ceph-mon`: 18.2.7+ds-1 (from Debian repository)
- `ceph-osd`: 18.2.7+ds-1
- **Requires**: `ceph-common` version 18.2.7+ds-1

### Conflict
- Debian's `ceph-mon` and `ceph-osd` require `ceph-common` version 18.2.7+ds-1
- System has `ceph-common` version 19.2.3-pve2 installed
- Cannot downgrade due to dependency chain and Proxmox protection

## Repository Issues

### Enterprise Repository
- **Status**: Configured but requires subscription (401 Unauthorized)
- **Location**: `https://enterprise.proxmox.com/debian/`

### No-Subscription Repository
- **Status**: SSL certificate verification failing
- **Error**: `SSL connection failed: error:0A000086:SSL routines::certificate verify failed`
- **Location**: `https://download.proxmox.com/debian/`

### Debian Repository
- **Status**: ✅ Working
- **Issue**: Only has older Ceph version (18.2.7) that conflicts with installed Proxmox packages (19.2.3)

## Attempted Solutions

### 1. Switch to No-Subscription Repository
- ✅ Updated repository configuration files
- ❌ SSL certificate verification failing
- **Next Step**: Fix SSL certificates or system clock

### 2. Downgrade to Debian Packages
- ❌ Blocked by dependency chain
- ❌ Proxmox protection prevents removing packages that would affect `proxmox-ve` meta-package
- **Error**: "You are attempting to remove the meta-package 'proxmox-ve'!"

### 3. Install Matching Proxmox Packages
- ❌ Cannot access Proxmox repositories (SSL/authentication issues)
- **Requires**: Fix repository access first

## Root Cause

1. **Mixed Package Sources**: System has Proxmox Ceph packages installed but cannot access Proxmox repositories to get matching `ceph-mon` and `ceph-osd` packages
2. **SSL Certificate Issue**: Cannot verify Proxmox repository certificates
3. **Version Mismatch**: Debian repositories only have older Ceph version that conflicts with installed Proxmox version

## Recommended Solutions

### Option 1: Fix SSL Certificate Issue (Recommended)
```bash
# Check system time
date

# Update CA certificates
apt update && apt install -y ca-certificates

# Try repository access again
apt update
pveceph install
```

### Option 2: Fix Repository Configuration
```bash
# Verify repository files are correct
cat /etc/apt/sources.list.d/pve-no-subscription.sources
cat /etc/apt/sources.list.d/ceph-no-subscription.sources

# Ensure correct components:
# - pve-no-subscription (not pve-enterprise)
# - no-subscription (not enterprise)
```

### Option 3: Use Proxmox Web UI
- Access Proxmox web interface
- Use GUI to install Ceph (may handle repository issues automatically)
- Navigate to: Datacenter > Ceph > Install

### Option 4: Manual Package Installation (Advanced)
If Proxmox repositories remain inaccessible:
1. Download Proxmox Ceph packages manually
2. Install with `dpkg -i` (may have dependency issues)
3. Or use `apt` with `--allow-downgrades` and `--force-depends` (risky)

### Option 5: Fresh Ceph Installation
If cluster can be reinitialized:
1. Remove all Ceph packages (with Proxmox protection override)
2. Install Debian Ceph packages fresh
3. Initialize new cluster

## Current Repository Configuration

### ML110-01 and R630-01
```
/etc/apt/sources.list.d/pve-no-subscription.sources:
  Types: deb
  URIs: https://download.proxmox.com/debian/pve
  Suites: trixie
  Components: pve-no-subscription

/etc/apt/sources.list.d/ceph-no-subscription.sources:
  Types: deb
  URIs: https://download.proxmox.com/debian/ceph-squid
  Suites: trixie
  Components: no-subscription
```

## Additional Findings

- **System Date**: Correct (Mon Dec 15 12:51:36 PM PST 2025)
- **CA Certificates**: Up to date
- **SSL Error Persists**: Even after certificate update
- **401 Unauthorized**: Suggests authentication/authorization issue, not just SSL
- **Possible Issue**: "trixie" (Debian 13 testing) may not have full Proxmox repository support

## Next Steps

1. **Verify Proxmox Version and Repository Suite** (Priority 1)
   - Check `pveversion` to determine correct repository suite
   - Verify if "trixie" is correct or should use "bookworm" or other
   - Update repository configuration if needed

2. **Alternative: Use Proxmox Web UI** (Priority 2)
   - Access Proxmox web interface
   - Install Ceph through GUI (may handle repository issues)
   - Navigate to: Datacenter > Ceph > Install

3. **Manual Package Resolution** (Priority 3)
   - If repositories remain inaccessible, consider:
     - Using Debian packages and accepting version downgrade
     - Or waiting for Proxmox repository access to be resolved

3. **Alternative Installation Method** (Priority 3)
   - Use Proxmox Web UI if CLI continues to fail
   - Or manually download and install packages

4. **Document Resolution** (Priority 4)
   - Update this document with solution
   - Update NEXT_STEPS.md with resolved path

## Related Files

- `docs/infrastructure/NEXT_STEPS.md` - Overall next steps
- `scripts/fresh-ceph-install-r630.sh` - Fresh installation script
- `/etc/apt/sources.list.d/` - Repository configuration

## Notes

- Proxmox VE installation includes Ceph client tools by default
- Full Ceph cluster services require additional packages
- `pveceph` command should handle installation automatically if repositories are accessible
- SSL certificate issues may indicate system time is incorrect or certificates need updating