chore: consolidate local WIP (repo cleanup 20260707)
Some checks failed
API CI / API Lint (push) Successful in 47s
API CI / API Type Check (push) Failing after 47s
API CI / API Test (push) Successful in 1m0s
API CI / API Build (push) Failing after 50s
API CI / Build Docker Image (push) Has been skipped
Build Crossplane Provider / build (push) Failing after 5m51s
CD Pipeline / Deploy to Staging (push) Failing after 29s
CI Pipeline / Lint and Type Check (push) Failing after 36s
CI Pipeline / Build (push) Has been skipped
CI Pipeline / Test Backend (push) Failing after 1m33s
CI Pipeline / Test Frontend (push) Failing after 30s
CI Pipeline / Security Scan (push) Failing after 1m16s
Crossplane Provider CI / Go Test (push) Failing after 3m23s
Crossplane Provider CI / Go Lint (push) Failing after 7m27s
Crossplane Provider CI / Go Build (push) Failing after 3m27s
Deploy to Staging / Deploy to Staging (push) Failing after 30s
Portal CI / Portal Lint (push) Failing after 21s
Portal CI / Portal Type Check (push) Failing after 21s
Portal CI / Portal Test (push) Failing after 21s
Portal CI / Portal Build (push) Failing after 22s
Test Suite / frontend-tests (push) Failing after 30s
Test Suite / api-tests (push) Failing after 49s
Test Suite / blockchain-tests (push) Failing after 30s
Type Check / type-check (map[directory:. name:root]) (push) Failing after 23s
Type Check / type-check (map[directory:api name:api]) (push) Failing after 21s
Type Check / type-check (map[directory:portal name:portal]) (push) Failing after 19s
Validate Configuration Files / validate (push) Failing after 1m52s
CD Pipeline / Deploy to Production (push) Has been skipped
Some checks failed
API CI / API Lint (push) Successful in 47s
API CI / API Type Check (push) Failing after 47s
API CI / API Test (push) Successful in 1m0s
API CI / API Build (push) Failing after 50s
API CI / Build Docker Image (push) Has been skipped
Build Crossplane Provider / build (push) Failing after 5m51s
CD Pipeline / Deploy to Staging (push) Failing after 29s
CI Pipeline / Lint and Type Check (push) Failing after 36s
CI Pipeline / Build (push) Has been skipped
CI Pipeline / Test Backend (push) Failing after 1m33s
CI Pipeline / Test Frontend (push) Failing after 30s
CI Pipeline / Security Scan (push) Failing after 1m16s
Crossplane Provider CI / Go Test (push) Failing after 3m23s
Crossplane Provider CI / Go Lint (push) Failing after 7m27s
Crossplane Provider CI / Go Build (push) Failing after 3m27s
Deploy to Staging / Deploy to Staging (push) Failing after 30s
Portal CI / Portal Lint (push) Failing after 21s
Portal CI / Portal Type Check (push) Failing after 21s
Portal CI / Portal Test (push) Failing after 21s
Portal CI / Portal Build (push) Failing after 22s
Test Suite / frontend-tests (push) Failing after 30s
Test Suite / api-tests (push) Failing after 49s
Test Suite / blockchain-tests (push) Failing after 30s
Type Check / type-check (map[directory:. name:root]) (push) Failing after 23s
Type Check / type-check (map[directory:api name:api]) (push) Failing after 21s
Type Check / type-check (map[directory:portal name:portal]) (push) Failing after 19s
Validate Configuration Files / validate (push) Failing after 1m52s
CD Pipeline / Deploy to Production (push) Has been skipped
Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
defiQUG
113
docs/infrastructure/CEPH_CLUSTER_SETUP.md
Normal file
113
docs/infrastructure/CEPH_CLUSTER_SETUP.md
Normal file
@@ -0,0 +1,113 @@
|
||||
# Ceph Cluster Setup - sfvalley-01
|
||||
|
||||
**Date**: 2025-12-15
|
||||
**Cluster Name**: sfvalley-01 (to be configured)
|
||||
**Status**: ⚠️ **IN PROGRESS - Monitor Configuration Needed**
|
||||
|
||||
## Current State
|
||||
|
||||
### Installation Status
|
||||
- ✅ **Ceph Packages**: Installed on both servers (version 19.2.3-pve2)
|
||||
- ml110-01 (192.168.11.10): ✅ Installed
|
||||
- r630-01 (192.168.11.11): ✅ Installed
|
||||
|
||||
### Cluster Configuration
|
||||
- **FSID**: f601f0e2-cd09-402f-9e15-4b1c9a7a7b25
|
||||
- **Network**: 192.168.11.0/24 (public and cluster)
|
||||
- **Monitors**:
|
||||
- ml110-01: Created (may need reconfiguration)
|
||||
- r630-01: Created separately (needs to join main cluster)
|
||||
|
||||
### Current Issues
|
||||
1. **Cluster Name**: Currently using default "ceph" name, needs to be "sfvalley-01"
|
||||
2. **Monitor Configuration**:
|
||||
- ml110-01: ✅ Running (cluster f601f0e2-cd09-402f-9e15-4b1c9a7a7b25)
|
||||
- r630-01: ❌ Failed - Monitor filesystem not created
|
||||
3. **Cluster Access**: Timeout issues when querying cluster status (may be normal during setup)
|
||||
|
||||
### Required Actions
|
||||
1. **Add r630-01 monitor to existing cluster**:
|
||||
```bash
|
||||
# On ml110-01: Get monmap and keyring
|
||||
ceph mon getmap -o /tmp/monmap
|
||||
scp /tmp/monmap root@192.168.11.11:/tmp/
|
||||
scp /etc/pve/priv/ceph.mon.keyring root@192.168.11.11:/tmp/mon-keyring
|
||||
|
||||
# On r630-01: Create monitor filesystem
|
||||
mkdir -p /var/lib/ceph/mon/ceph-r630-01
|
||||
chown ceph:ceph /var/lib/ceph/mon/ceph-r630-01
|
||||
ceph-mon --mkfs -i r630-01 --monmap /tmp/monmap --keyring /tmp/mon-keyring
|
||||
chown -R ceph:ceph /var/lib/ceph/mon/ceph-r630-01
|
||||
systemctl start ceph-mon@r630-01.service
|
||||
```
|
||||
|
||||
2. **Update cluster name to "sfvalley-01"** (if needed):
|
||||
- This may require reinitializing if cluster name change is required
|
||||
- Or use "sfvalley-01" as display name in Proxmox UI
|
||||
|
||||
## Configuration Files
|
||||
|
||||
### `/etc/pve/ceph.conf` (Current)
|
||||
```
|
||||
[global]
|
||||
auth_client_required = cephx
|
||||
auth_cluster_required = cephx
|
||||
auth_service_required = cephx
|
||||
cluster_network = 192.168.11.0/24
|
||||
fsid = f601f0e2-cd09-402f-9e15-4b1c9a7a7b25
|
||||
mon_allow_pool_delete = true
|
||||
mon_host = 192.168.11.10
|
||||
ms_bind_ipv4 = true
|
||||
ms_bind_ipv6 = false
|
||||
osd_pool_default_min_size = 2
|
||||
osd_pool_default_size = 3
|
||||
public_network = 192.168.11.0/24
|
||||
|
||||
[client]
|
||||
keyring = /etc/pve/priv/$cluster.$name.keyring
|
||||
|
||||
[client.crash]
|
||||
keyring = /etc/pve/ceph/$cluster.$name.keyring
|
||||
|
||||
[mon.ml110-01]
|
||||
public_addr = 192.168.11.10
|
||||
```
|
||||
|
||||
## Next Steps to Configure sfvalley-01 Cluster
|
||||
|
||||
### Option 1: Reinitialize with Cluster Name (Recommended if cluster is new/empty)
|
||||
Since the cluster appears to be newly created and may not have data, we can reinitialize with the proper cluster name:
|
||||
|
||||
1. **Stop all Ceph services**
|
||||
2. **Remove existing cluster data**
|
||||
3. **Reinitialize with cluster name "sfvalley-01"**
|
||||
4. **Create monitors on both nodes**
|
||||
|
||||
### Option 2: Update Existing Cluster (If cluster has data)
|
||||
If the cluster already has important data:
|
||||
1. Update configuration files to reference "sfvalley-01"
|
||||
2. Update keyring paths
|
||||
3. Restart services
|
||||
|
||||
## Cluster Name Configuration
|
||||
|
||||
In Ceph, the cluster name affects:
|
||||
- Configuration file location: `/etc/ceph/sfvalley-01.conf`
|
||||
- Data directories: `/var/lib/ceph/mon/sfvalley-01-*`
|
||||
- Keyring paths: `/etc/pve/priv/sfvalley-01.*.keyring`
|
||||
- Command usage: `ceph -n sfvalley-01 -s`
|
||||
|
||||
## Network Configuration
|
||||
|
||||
- **Public Network**: 192.168.11.0/24
|
||||
- **Cluster Network**: 192.168.11.0/24
|
||||
- **Monitor Hosts**:
|
||||
- ml110-01: 192.168.11.10
|
||||
- r630-01: 192.168.11.11
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- `docs/infrastructure/NEXT_STEPS.md` - Overall infrastructure plan
|
||||
- `docs/infrastructure/CEPH_INSTALLATION_ISSUE.md` - Installation troubleshooting
|
||||
- `scripts/fresh-ceph-install-r630.sh` - Fresh installation script
|
||||
|
||||
49
docs/infrastructure/CEPH_CLUSTER_STATUS.md
Normal file
49
docs/infrastructure/CEPH_CLUSTER_STATUS.md
Normal file
@@ -0,0 +1,49 @@
|
||||
# Ceph Cluster Status - sfvalley-01
|
||||
|
||||
**Date**: 2025-12-15
|
||||
**Last Updated**: 2025-12-15
|
||||
|
||||
## Summary
|
||||
|
||||
✅ **Ceph Installation**: Complete on both servers
|
||||
⚠️ **Cluster Configuration**: Monitor on r630-01 needs to be added to existing cluster
|
||||
📝 **Cluster Name**: sfvalley-01 (to be configured)
|
||||
|
||||
## Installation Status
|
||||
|
||||
### ML110-01 (192.168.11.10)
|
||||
- ✅ Ceph packages installed (19.2.3-pve2)
|
||||
- ✅ Monitor service: **ACTIVE** (ml110-01)
|
||||
- ✅ Manager service: **ACTIVE** (ml110-01)
|
||||
- ✅ Cluster FSID: f601f0e2-cd09-402f-9e15-4b1c9a7a7b25
|
||||
|
||||
### R630-01 (192.168.11.11)
|
||||
- ✅ Ceph packages installed (19.2.3-pve2)
|
||||
- ❌ Monitor service: **FAILED** (needs filesystem creation)
|
||||
- ⚠️ Manager service: Not yet configured
|
||||
- ⚠️ Needs to join existing cluster
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. **Add r630-01 monitor to cluster** (see CEPH_CLUSTER_SETUP.md for commands)
|
||||
2. **Verify quorum** once both monitors are running
|
||||
3. **Configure cluster name** as "sfvalley-01" if required
|
||||
4. **Add OSDs** using available disks (6x 250GB SSDs on r630-01)
|
||||
|
||||
## Commands Reference
|
||||
|
||||
```bash
|
||||
# Check cluster status
|
||||
ceph -s
|
||||
|
||||
# Check monitor status
|
||||
systemctl status ceph-mon@ml110-01.service
|
||||
systemctl status ceph-mon@r630-01.service
|
||||
|
||||
# Check monitor quorum
|
||||
ceph quorum_status
|
||||
|
||||
# List monitors
|
||||
ceph mon dump
|
||||
```
|
||||
|
||||
123
docs/infrastructure/CEPH_CURRENT_STATE.md
Normal file
123
docs/infrastructure/CEPH_CURRENT_STATE.md
Normal file
@@ -0,0 +1,123 @@
|
||||
# Ceph Cluster Current State - sfvalley-01
|
||||
|
||||
**Date**: 2025-12-15 15:18 PST
|
||||
**Status**: ⚠️ **CLUSTER NOT FULLY OPERATIONAL**
|
||||
|
||||
## Summary
|
||||
|
||||
The Ceph cluster on ml110-01 exists but is not responding to commands. The monitor is running but in "probing" state and cannot form quorum, which prevents:
|
||||
- Adding r630-01 monitor
|
||||
- Creating OSDs
|
||||
- Querying cluster status
|
||||
|
||||
## Current Configuration
|
||||
|
||||
### Both Nodes (Synced via Proxmox cluster)
|
||||
```
|
||||
[global]
|
||||
auth_client_required = cephx
|
||||
auth_cluster_required = cephx
|
||||
auth_service_required = cephx
|
||||
cluster_network = 192.168.11.0/24
|
||||
fsid = 021d20d3-446f-42cb-a219-5e01213b7b2d
|
||||
mon_allow_pool_delete = true
|
||||
mon_host = 192.168.11.10 192.168.11.11
|
||||
ms_bind_ipv4 = true
|
||||
ms_bind_ipv6 = false
|
||||
osd_pool_default_min_size = 2
|
||||
osd_pool_default_size = 3
|
||||
public_network = 192.168.11.0/24
|
||||
|
||||
[client]
|
||||
keyring = /etc/pve/priv/$cluster.$name.keyring
|
||||
|
||||
[client.crash]
|
||||
keyring = /etc/pve/ceph/$cluster.$name.keyring
|
||||
|
||||
[mon.ml110-01]
|
||||
public_addr = 192.168.11.10
|
||||
|
||||
[mon.r630-01]
|
||||
public_addr = 192.168.11.11
|
||||
```
|
||||
|
||||
## Service Status
|
||||
|
||||
### ML110-01
|
||||
- ✅ ceph-mon@ml110-01.service: **ACTIVE (running)**
|
||||
- ⚠️ Monitor state: "probing" (cannot form quorum)
|
||||
- ❌ Ceph commands: Timeout (cluster not accessible)
|
||||
|
||||
### R630-01
|
||||
- ❌ ceph-mon@r630-01.service: **FAILED** (filesystem not created)
|
||||
- ⚠️ Cannot create monitor without monmap from ml110-01
|
||||
- ⚠️ Cannot create OSDs without cluster access
|
||||
|
||||
## Available Disks for OSDs (R630-01)
|
||||
|
||||
- `/dev/sdc`: 232.9G (CT250MX500SSD1)
|
||||
- `/dev/sdd`: 232.9G (CT250MX500SSD1)
|
||||
- `/dev/sde`: 232.9G (CT250MX500SSD1)
|
||||
- `/dev/sdf`: 232.9G (CT250MX500SSD1)
|
||||
- `/dev/sdg`: 232.9G (CT250MX500SSD1)
|
||||
- `/dev/sdh`: 232.9G (CT250MX500SSD1)
|
||||
|
||||
**Total Available**: ~1.4TB across 6 SSDs
|
||||
|
||||
## Blocking Issues
|
||||
|
||||
1. **Monitor Quorum Not Established**
|
||||
- ML110-01 monitor in "probing" state
|
||||
- Cannot form quorum with single monitor
|
||||
- R630-01 monitor cannot be added without quorum
|
||||
|
||||
2. **Cluster Commands Timeout**
|
||||
- `ceph -s`: Timeout
|
||||
- `ceph mon getmap`: Timeout
|
||||
- `ceph quorum_status`: Timeout
|
||||
- Cannot query or manage cluster
|
||||
|
||||
3. **Monitor Filesystem Not Created**
|
||||
- R630-01 monitor directory exists but is empty
|
||||
- Needs monmap and keyring from ml110-01
|
||||
- Cannot create without cluster access
|
||||
|
||||
## Recommended Solutions
|
||||
|
||||
### Option 1: Use Proxmox Web UI (Recommended)
|
||||
The Proxmox web interface may handle the monitor and OSD creation more reliably:
|
||||
1. Access Proxmox web UI
|
||||
2. Navigate to: Datacenter > Ceph > Monitors
|
||||
3. Add r630-01 monitor through UI
|
||||
4. Navigate to: Datacenter > Ceph > OSDs
|
||||
5. Create OSDs on r630-01 disks through UI
|
||||
|
||||
### Option 2: Reinitialize Ceph Cluster
|
||||
If the cluster is not critical or can be reinitialized:
|
||||
1. Stop Ceph on ml110-01
|
||||
2. Remove existing Ceph data
|
||||
3. Reinitialize cluster with proper configuration
|
||||
4. Add r630-01 monitor
|
||||
5. Add OSDs
|
||||
|
||||
### Option 3: Fix Monitor Quorum
|
||||
If there's existing data that must be preserved:
|
||||
1. Investigate why ml110-01 monitor is in "probing" state
|
||||
2. Check monitor logs for errors
|
||||
3. Fix underlying issue preventing quorum
|
||||
4. Then add r630-01 monitor
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. **Investigate Monitor Issue**: Check why ml110-01 monitor cannot form quorum
|
||||
2. **Use Web UI**: Try adding monitor and OSDs through Proxmox web interface
|
||||
3. **Or Reinitialize**: If acceptable, reinitialize cluster with proper setup
|
||||
|
||||
## Notes
|
||||
|
||||
- Configuration is correct and synced across cluster
|
||||
- Network connectivity is working (ping successful)
|
||||
- Monitor ports are open (6789, 3300)
|
||||
- Issue appears to be with monitor quorum formation
|
||||
- May require cluster reinitialization or web UI approach
|
||||
|
||||
169
docs/infrastructure/CEPH_INSTALLATION_ISSUE.md
Normal file
169
docs/infrastructure/CEPH_INSTALLATION_ISSUE.md
Normal file
@@ -0,0 +1,169 @@
|
||||
# Ceph Installation Issue - Package Version Conflict
|
||||
|
||||
**Date**: 2025-12-15
|
||||
**Status**: ⚠️ **BLOCKED - Package Version Conflict**
|
||||
|
||||
## Problem Summary
|
||||
|
||||
Attempting to install Ceph cluster services (`ceph-mon`, `ceph-osd`) is blocked by package version conflicts between Proxmox and Debian repositories.
|
||||
|
||||
## Current State
|
||||
|
||||
### Installed Packages (Proxmox Version)
|
||||
- `ceph-common`: 19.2.3-pve2 (from Proxmox repository)
|
||||
- `ceph-fuse`: 19.2.3-pve2
|
||||
- Related libraries: `libcephfs2`, `librados2`, `librbd1`, `python3-*` packages all at 19.2.3-pve2
|
||||
|
||||
### Available Packages (Debian Version)
|
||||
- `ceph-mon`: 18.2.7+ds-1 (from Debian repository)
|
||||
- `ceph-osd`: 18.2.7+ds-1
|
||||
- **Requires**: `ceph-common` version 18.2.7+ds-1
|
||||
|
||||
### Conflict
|
||||
- Debian's `ceph-mon` and `ceph-osd` require `ceph-common` version 18.2.7+ds-1
|
||||
- System has `ceph-common` version 19.2.3-pve2 installed
|
||||
- Cannot downgrade due to dependency chain and Proxmox protection
|
||||
|
||||
## Repository Issues
|
||||
|
||||
### Enterprise Repository
|
||||
- **Status**: Configured but requires subscription (401 Unauthorized)
|
||||
- **Location**: `https://enterprise.proxmox.com/debian/`
|
||||
|
||||
### No-Subscription Repository
|
||||
- **Status**: SSL certificate verification failing
|
||||
- **Error**: `SSL connection failed: error:0A000086:SSL routines::certificate verify failed`
|
||||
- **Location**: `https://download.proxmox.com/debian/`
|
||||
|
||||
### Debian Repository
|
||||
- **Status**: ✅ Working
|
||||
- **Issue**: Only has older Ceph version (18.2.7) that conflicts with installed Proxmox packages (19.2.3)
|
||||
|
||||
## Attempted Solutions
|
||||
|
||||
### 1. Switch to No-Subscription Repository
|
||||
- ✅ Updated repository configuration files
|
||||
- ❌ SSL certificate verification failing
|
||||
- **Next Step**: Fix SSL certificates or system clock
|
||||
|
||||
### 2. Downgrade to Debian Packages
|
||||
- ❌ Blocked by dependency chain
|
||||
- ❌ Proxmox protection prevents removing packages that would affect `proxmox-ve` meta-package
|
||||
- **Error**: "You are attempting to remove the meta-package 'proxmox-ve'!"
|
||||
|
||||
### 3. Install Matching Proxmox Packages
|
||||
- ❌ Cannot access Proxmox repositories (SSL/authentication issues)
|
||||
- **Requires**: Fix repository access first
|
||||
|
||||
## Root Cause
|
||||
|
||||
1. **Mixed Package Sources**: System has Proxmox Ceph packages installed but cannot access Proxmox repositories to get matching `ceph-mon` and `ceph-osd` packages
|
||||
2. **SSL Certificate Issue**: Cannot verify Proxmox repository certificates
|
||||
3. **Version Mismatch**: Debian repositories only have older Ceph version that conflicts with installed Proxmox version
|
||||
|
||||
## Recommended Solutions
|
||||
|
||||
### Option 1: Fix SSL Certificate Issue (Recommended)
|
||||
```bash
|
||||
# Check system time
|
||||
date
|
||||
|
||||
# Update CA certificates
|
||||
apt update && apt install -y ca-certificates
|
||||
|
||||
# Try repository access again
|
||||
apt update
|
||||
pveceph install
|
||||
```
|
||||
|
||||
### Option 2: Fix Repository Configuration
|
||||
```bash
|
||||
# Verify repository files are correct
|
||||
cat /etc/apt/sources.list.d/pve-no-subscription.sources
|
||||
cat /etc/apt/sources.list.d/ceph-no-subscription.sources
|
||||
|
||||
# Ensure correct components:
|
||||
# - pve-no-subscription (not pve-enterprise)
|
||||
# - no-subscription (not enterprise)
|
||||
```
|
||||
|
||||
### Option 3: Use Proxmox Web UI
|
||||
- Access Proxmox web interface
|
||||
- Use GUI to install Ceph (may handle repository issues automatically)
|
||||
- Navigate to: Datacenter > Ceph > Install
|
||||
|
||||
### Option 4: Manual Package Installation (Advanced)
|
||||
If Proxmox repositories remain inaccessible:
|
||||
1. Download Proxmox Ceph packages manually
|
||||
2. Install with `dpkg -i` (may have dependency issues)
|
||||
3. Or use `apt` with `--allow-downgrades` and `--force-depends` (risky)
|
||||
|
||||
### Option 5: Fresh Ceph Installation
|
||||
If cluster can be reinitialized:
|
||||
1. Remove all Ceph packages (with Proxmox protection override)
|
||||
2. Install Debian Ceph packages fresh
|
||||
3. Initialize new cluster
|
||||
|
||||
## Current Repository Configuration
|
||||
|
||||
### ML110-01 and R630-01
|
||||
```
|
||||
/etc/apt/sources.list.d/pve-no-subscription.sources:
|
||||
Types: deb
|
||||
URIs: https://download.proxmox.com/debian/pve
|
||||
Suites: trixie
|
||||
Components: pve-no-subscription
|
||||
|
||||
/etc/apt/sources.list.d/ceph-no-subscription.sources:
|
||||
Types: deb
|
||||
URIs: https://download.proxmox.com/debian/ceph-squid
|
||||
Suites: trixie
|
||||
Components: no-subscription
|
||||
```
|
||||
|
||||
## Additional Findings
|
||||
|
||||
- **System Date**: Correct (Mon Dec 15 12:51:36 PM PST 2025)
|
||||
- **CA Certificates**: Up to date
|
||||
- **SSL Error Persists**: Even after certificate update
|
||||
- **401 Unauthorized**: Suggests authentication/authorization issue, not just SSL
|
||||
- **Possible Issue**: "trixie" (Debian 13 testing) may not have full Proxmox repository support
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. **Verify Proxmox Version and Repository Suite** (Priority 1)
|
||||
- Check `pveversion` to determine correct repository suite
|
||||
- Verify if "trixie" is correct or should use "bookworm" or other
|
||||
- Update repository configuration if needed
|
||||
|
||||
2. **Alternative: Use Proxmox Web UI** (Priority 2)
|
||||
- Access Proxmox web interface
|
||||
- Install Ceph through GUI (may handle repository issues)
|
||||
- Navigate to: Datacenter > Ceph > Install
|
||||
|
||||
3. **Manual Package Resolution** (Priority 3)
|
||||
- If repositories remain inaccessible, consider:
|
||||
- Using Debian packages and accepting version downgrade
|
||||
- Or waiting for Proxmox repository access to be resolved
|
||||
|
||||
3. **Alternative Installation Method** (Priority 3)
|
||||
- Use Proxmox Web UI if CLI continues to fail
|
||||
- Or manually download and install packages
|
||||
|
||||
4. **Document Resolution** (Priority 4)
|
||||
- Update this document with solution
|
||||
- Update NEXT_STEPS.md with resolved path
|
||||
|
||||
## Related Files
|
||||
|
||||
- `docs/infrastructure/NEXT_STEPS.md` - Overall next steps
|
||||
- `scripts/fresh-ceph-install-r630.sh` - Fresh installation script
|
||||
- `/etc/apt/sources.list.d/` - Repository configuration
|
||||
|
||||
## Notes
|
||||
|
||||
- Proxmox VE installation includes Ceph client tools by default
|
||||
- Full Ceph cluster services require additional packages
|
||||
- `pveceph` command should handle installation automatically if repositories are accessible
|
||||
- SSL certificate issues may indicate system time is incorrect or certificates need updating
|
||||
|
||||
115
docs/infrastructure/CEPH_PROGRESS_SUMMARY.md
Normal file
115
docs/infrastructure/CEPH_PROGRESS_SUMMARY.md
Normal file
@@ -0,0 +1,115 @@
|
||||
# Ceph Setup Progress Summary - sfvalley-01
|
||||
|
||||
**Date**: 2025-12-15 15:25 PST
|
||||
**Status**: ⚠️ **PARTIAL SUCCESS - Monitor Filesystem Created, Service Issues Remain**
|
||||
|
||||
## Completed Steps
|
||||
|
||||
### ✅ Infrastructure Setup
|
||||
1. **SSH Key Authentication**: Configured for both servers
|
||||
2. **Disk Layout Documentation**: fdisk output documented for both servers
|
||||
3. **Ceph Packages**: Installed on both ml110-01 and r630-01 (19.2.3-pve2)
|
||||
4. **Proxmox Cluster**: r630-01 successfully joined cluster "sfvalley-01"
|
||||
|
||||
### ✅ Configuration
|
||||
1. **Ceph Configuration**: Updated and synced across cluster
|
||||
- Network: 192.168.11.0/24 (corrected from 192.168.11.11/24)
|
||||
- FSID: 021d20d3-446f-42cb-a219-5e01213b7b2d
|
||||
- Monitor hosts: 192.168.11.10, 192.168.11.11
|
||||
- Both monitor sections configured
|
||||
|
||||
### ✅ Monitor Setup Progress
|
||||
1. **Monmap Extracted**: Successfully extracted from ml110-01 monitor
|
||||
2. **Monitor Keyring**: Retrieved from ml110-01
|
||||
3. **Monitor Filesystem Created**: `/var/lib/ceph/mon/ceph-r630-01/` filesystem created with:
|
||||
- keyring
|
||||
- kv_backend
|
||||
- store.db directory
|
||||
|
||||
## Current Issues
|
||||
|
||||
### ⚠️ Monitor Service
|
||||
- **Status**: Monitor filesystem exists but service fails to start
|
||||
- **Error**: Service exits with code 1
|
||||
- **Location**: `/var/lib/ceph/mon/ceph-r630-01/` exists with proper structure
|
||||
- **Next Step**: Investigate why service fails despite filesystem being created
|
||||
|
||||
### ⚠️ Cluster Accessibility
|
||||
- **Status**: Ceph commands timeout
|
||||
- **Impact**: Cannot verify quorum, cannot create OSDs
|
||||
- **Root Cause**: ML110-01 monitor in "probing" state, cannot form quorum
|
||||
- **Next Step**: Resolve monitor quorum issue on ml110-01
|
||||
|
||||
### ⚠️ OSD Creation
|
||||
- **Status**: Cannot create OSDs (cluster not accessible)
|
||||
- **Available Disks**: 6x 232.9G SSDs on r630-01 (/dev/sdc through /dev/sdh)
|
||||
- **Next Step**: Create OSDs once cluster is operational
|
||||
|
||||
## Technical Details
|
||||
|
||||
### Monitor Filesystem Structure (r630-01)
|
||||
```
|
||||
/var/lib/ceph/mon/ceph-r630-01/
|
||||
├── keyring (77 bytes)
|
||||
├── kv_backend (8 bytes)
|
||||
└── store.db/ (database directory)
|
||||
```
|
||||
|
||||
### Configuration Files
|
||||
- **Location**: `/etc/pve/ceph.conf` (synced via Proxmox cluster)
|
||||
- **Status**: ✅ Correct and synced
|
||||
- **Monitors Configured**: ml110-01, r630-01
|
||||
|
||||
## Next Steps
|
||||
|
||||
### Priority 1: Fix Monitor Service
|
||||
1. Check detailed error logs: `journalctl -xeu ceph-mon@r630-01.service`
|
||||
2. Verify filesystem permissions and ownership
|
||||
3. Check if monitor can connect to ml110-01
|
||||
4. Resolve service startup issue
|
||||
|
||||
### Priority 2: Establish Monitor Quorum
|
||||
1. Investigate why ml110-01 monitor is in "probing" state
|
||||
2. Check monitor logs for errors
|
||||
3. Verify network connectivity between monitors
|
||||
4. Ensure both monitors can form quorum
|
||||
|
||||
### Priority 3: Create OSDs
|
||||
1. Once cluster is accessible, create OSDs on r630-01:
|
||||
```bash
|
||||
pveceph osd create /dev/sdc
|
||||
pveceph osd create /dev/sdd
|
||||
pveceph osd create /dev/sde
|
||||
pveceph osd create /dev/sdf
|
||||
pveceph osd create /dev/sdg
|
||||
pveceph osd create /dev/sdh
|
||||
```
|
||||
|
||||
## Commands Reference
|
||||
|
||||
### Check Monitor Status
|
||||
```bash
|
||||
systemctl status ceph-mon@r630-01.service
|
||||
journalctl -u ceph-mon@r630-01.service -n 20
|
||||
```
|
||||
|
||||
### Check Cluster Status
|
||||
```bash
|
||||
ceph quorum_status
|
||||
ceph -s
|
||||
ceph mon dump
|
||||
```
|
||||
|
||||
### Create OSDs (once cluster is operational)
|
||||
```bash
|
||||
pveceph osd create /dev/sdc
|
||||
# Repeat for sdd, sde, sdf, sdg, sdh
|
||||
```
|
||||
|
||||
## Notes
|
||||
|
||||
- Monitor filesystem was successfully created using monmap and keyring from ml110-01
|
||||
- Configuration is correct and synced across Proxmox cluster
|
||||
- The main blocker is the monitor service not starting and cluster not being accessible
|
||||
- May need to investigate deeper into monitor logs or use Proxmox web UI for final setup
|
||||
|
||||
115
docs/infrastructure/CEPH_REMOVAL_COMPLETE.md
Normal file
115
docs/infrastructure/CEPH_REMOVAL_COMPLETE.md
Normal file
@@ -0,0 +1,115 @@
|
||||
# Ceph Removal Complete - sfvalley-01
|
||||
|
||||
**Date**: 2025-12-15
|
||||
**Status**: ✅ **CEPH REMOVED FROM BOTH SERVERS**
|
||||
|
||||
## Removal Actions Performed
|
||||
|
||||
### ML110-01 (192.168.11.10)
|
||||
1. ✅ Stopped all Ceph services (mon, mgr, osd, mds)
|
||||
2. ✅ Disabled Ceph services
|
||||
3. ✅ Ran `pveceph purge` (removed service symlinks)
|
||||
4. ✅ Removed Ceph data directories:
|
||||
- `/var/lib/ceph/` - Removed
|
||||
- `/etc/ceph/*` - Cleaned (directory remains empty)
|
||||
- `/etc/pve/ceph.conf` - Removed
|
||||
- `/var/log/ceph/` - Removed
|
||||
- `/run/ceph/` - Removed
|
||||
5. ⚠️ Ceph packages: Some packages may remain due to Proxmox dependencies
|
||||
|
||||
### R630-01 (192.168.11.11)
|
||||
1. ✅ Stopped all Ceph services (mon, mgr, osd, mds)
|
||||
2. ✅ Disabled Ceph services
|
||||
3. ✅ Ran `pveceph purge` (removed service symlinks)
|
||||
4. ✅ Removed Ceph data directories:
|
||||
- `/var/lib/ceph/` - Removed
|
||||
- `/etc/ceph/*` - Cleaned (directory remains empty)
|
||||
- `/etc/pve/ceph.conf` - Removed
|
||||
- `/var/log/ceph/` - Removed
|
||||
- `/run/ceph/` - Removed
|
||||
5. ⚠️ Ceph packages: Some packages may remain due to Proxmox dependencies
|
||||
|
||||
## Current State
|
||||
|
||||
### Data Removal
|
||||
- ✅ All Ceph data directories removed
|
||||
- ✅ Configuration files removed:
|
||||
- `/etc/pve/ceph.conf` - Removed
|
||||
- `/etc/pve/priv/*ceph*` keyrings - Removed
|
||||
- `/etc/pve/ceph/` directory - Removed
|
||||
- `/etc/ceph/` directory - Cleaned
|
||||
- `/etc/apt/sources.list.d/ceph-enterprise.sources` - Removed
|
||||
- `/etc/apt/sources.list.d/ceph.sources` - Removed
|
||||
- `/etc/default/ceph` - Removed
|
||||
- `/etc/logrotate.d/ceph-common` - Removed
|
||||
- `/etc/sudoers.d/ceph-smartctl` - Removed
|
||||
- `/etc/sysctl.d/30-ceph-osd.conf` - Removed
|
||||
- `/etc/bash_completion.d/ceph` - Removed
|
||||
- `/etc/lvm/archive/ceph-*` archives - Removed
|
||||
- ✅ Service symlinks removed
|
||||
- ✅ Log directories removed
|
||||
|
||||
### Packages
|
||||
- ✅ **Core Ceph services REMOVED**: ceph, ceph-mon, ceph-osd, ceph-mgr, ceph-mds, ceph-base, ceph-volume
|
||||
- ⚠️ **Client tools remain**: ceph-common, ceph-fuse (may be required by Proxmox)
|
||||
- ⚠️ **Libraries remain**: libcephfs2, python3-ceph-* (dependencies)
|
||||
- **Status**: Core cluster services removed, client tools remain for Proxmox compatibility
|
||||
|
||||
### Services
|
||||
- ✅ All Ceph daemon services stopped and disabled
|
||||
- ✅ No active Ceph services running
|
||||
- ⚠️ Device mappers may remain from previous OSDs (harmless)
|
||||
|
||||
### Services
|
||||
- ✅ All Ceph daemon services stopped and disabled
|
||||
- ✅ Only systemd targets/slices remain (harmless, no active services)
|
||||
|
||||
## Final Verification
|
||||
|
||||
### ML110-01
|
||||
- ✅ Core Ceph binaries: Removed (ceph-mon, ceph-osd, ceph-mgr not found)
|
||||
- ✅ Ceph data: Removed
|
||||
- ✅ Configuration: Removed
|
||||
- ✅ LVM volumes: Removed (ceph-ba672668-8341-42e0-a8fd-277cee5491e4)
|
||||
- ⚠️ Client tools: Remain (ceph-common, ceph-fuse)
|
||||
|
||||
### R630-01
|
||||
- ✅ Core Ceph binaries: Removed (ceph-mon, ceph-osd, ceph-mgr not found)
|
||||
- ✅ Ceph data: Removed
|
||||
- ✅ Configuration: Removed
|
||||
- ⚠️ Client tools: Remain (ceph-common, ceph-fuse)
|
||||
|
||||
### Verification Commands
|
||||
```bash
|
||||
# Check remaining Ceph packages
|
||||
dpkg -l | grep ceph
|
||||
|
||||
# Check Ceph services
|
||||
systemctl list-units --all | grep ceph
|
||||
|
||||
# Check data directories
|
||||
ls -la /var/lib/ceph /etc/ceph /etc/pve/ceph.conf
|
||||
|
||||
# Check for Ceph binaries
|
||||
which ceph-mon ceph-osd ceph-mgr
|
||||
```
|
||||
|
||||
## Notes
|
||||
|
||||
- **Proxmox Protection**: Some Ceph packages may remain if Proxmox VE requires them as dependencies
|
||||
- **Client Tools**: `ceph-common` and `ceph-fuse` may remain if needed by Proxmox storage features
|
||||
- **No Reinstallation**: As requested, Ceph has NOT been reinstalled
|
||||
- **Clean State**: All Ceph cluster data and configuration has been removed
|
||||
- **Proxmox Cluster**: Still intact and operational (sfvalley-01)
|
||||
|
||||
## Next Steps (If Needed)
|
||||
|
||||
If you need to completely remove all Ceph packages including client tools:
|
||||
```bash
|
||||
# This may require touching the Proxmox protection file
|
||||
touch /please-remove-proxmox-ve
|
||||
apt remove --purge ceph-common ceph-fuse
|
||||
```
|
||||
|
||||
However, this is generally not recommended as Proxmox may use these for storage features.
|
||||
|
||||
50
docs/infrastructure/CEPH_SETUP_ISSUE.md
Normal file
50
docs/infrastructure/CEPH_SETUP_ISSUE.md
Normal file
@@ -0,0 +1,50 @@
|
||||
# Ceph Setup Issue - Configuration Mismatch
|
||||
|
||||
**Date**: 2025-12-15
|
||||
**Issue**: Cannot add r630-01 monitor due to configuration mismatch
|
||||
|
||||
## Problem
|
||||
|
||||
When attempting to add r630-01 as a Ceph monitor, we discovered:
|
||||
|
||||
1. **Different FSIDs**:
|
||||
- ML110-01 cluster: `f601f0e2-cd09-402f-9e15-4b1c9a7a7b25` (from earlier status)
|
||||
- R630-01 config: `021d20d3-446f-42cb-a219-5e01213b7b2d` (from current /etc/pve/ceph.conf)
|
||||
|
||||
2. **Network Configuration Mismatch**:
|
||||
- ML110-01: `192.168.11.0/24`
|
||||
- R630-01: `192.168.11.11/24` (incorrect - should be 192.168.11.0/24)
|
||||
|
||||
3. **Monitor Connection Issues**:
|
||||
- ML110-01 monitor is running but commands timeout
|
||||
- Cannot get monmap from ml110-01
|
||||
- R630-01 cannot connect to cluster
|
||||
|
||||
## Root Cause
|
||||
|
||||
The `/etc/pve/ceph.conf` on r630-01 appears to have a different/incorrect configuration, possibly from a previous cluster setup or misconfiguration. The Proxmox cluster filesystem should sync this, but there may be a conflict.
|
||||
|
||||
## Solution Options
|
||||
|
||||
### Option 1: Sync Configuration from ML110-01
|
||||
Since Proxmox cluster filesystem should sync `/etc/pve/ceph.conf`, we need to ensure:
|
||||
1. ML110-01 has the correct configuration
|
||||
2. Configuration syncs to r630-01
|
||||
3. Then add r630-01 monitor
|
||||
|
||||
### Option 2: Reinitialize Ceph Cluster
|
||||
If the cluster on ml110-01 is not operational or has issues:
|
||||
1. Stop Ceph on ml110-01
|
||||
2. Reinitialize with proper configuration
|
||||
3. Add r630-01 monitor
|
||||
4. Add OSDs
|
||||
|
||||
### Option 3: Use Proxmox Web UI
|
||||
The Proxmox web interface may handle the configuration and monitor addition more reliably.
|
||||
|
||||
## Current Status
|
||||
|
||||
- ML110-01: Monitor running but cluster commands timeout
|
||||
- R630-01: Cannot connect to cluster, configuration mismatch
|
||||
- Need to resolve configuration before proceeding
|
||||
|
||||
254
docs/infrastructure/COMPLETE_STATUS_REPORT.md
Normal file
254
docs/infrastructure/COMPLETE_STATUS_REPORT.md
Normal file
@@ -0,0 +1,254 @@
|
||||
# Complete Status Report - sfvalley-01 Cluster
|
||||
|
||||
**Date**: 2025-12-15 14:55 PST
|
||||
**Cluster Name**: sfvalley-01
|
||||
**Report Type**: Pre-Change Status Poll
|
||||
|
||||
## Proxmox Cluster Status
|
||||
|
||||
### ✅ **HEALTHY - Quorum Established**
|
||||
|
||||
**Cluster Information:**
|
||||
- **Name**: sfvalley-01
|
||||
- **Config Version**: 2
|
||||
- **Transport**: knet
|
||||
- **Secure Auth**: Enabled
|
||||
|
||||
**Quorum Status:**
|
||||
- **Status**: ✅ Quorate
|
||||
- **Expected Votes**: 2
|
||||
- **Total Votes**: 2
|
||||
- **Quorum**: 2
|
||||
|
||||
**Cluster Members:**
|
||||
- **Node 1**: ml110-01 (192.168.11.10) - Node ID: 0x00000001
|
||||
- **Node 2**: r630-01 (192.168.11.11) - Node ID: 0x00000002
|
||||
|
||||
**Status**: ✅ Both nodes are active and in quorum
|
||||
|
||||
---
|
||||
|
||||
## Ceph Cluster Status
|
||||
|
||||
### ⚠️ **PARTIALLY OPERATIONAL - Monitor Issues**
|
||||
|
||||
**Cluster Configuration:**
|
||||
- **FSID**: f601f0e2-cd09-402f-9e15-4b1c9a7a7b25
|
||||
- **Network**: 192.168.11.0/24 (public and cluster)
|
||||
- **Pool Settings**:
|
||||
- Default size: 3
|
||||
- Default min_size: 2
|
||||
|
||||
### ML110-01 (192.168.11.10) Status
|
||||
|
||||
**Services:**
|
||||
- ✅ **ceph-mon@ml110-01.service**: ACTIVE (running)
|
||||
- Status: Running but in "probing" state
|
||||
- Started: Mon 2025-12-15 14:39:30 PST
|
||||
- Memory: 142.7M
|
||||
- Issue: Reporting slow operations (1800+ slow ops)
|
||||
|
||||
- ✅ **ceph-mgr@ml110-01.service**: ACTIVE (running)
|
||||
- Status: Running
|
||||
- Started: Mon 2025-12-15 14:39:33 PST
|
||||
- Memory: 327.3M
|
||||
- Issue: Authentication errors with monitor
|
||||
|
||||
**Data Directories:**
|
||||
- ✅ `/var/lib/ceph/mon/ceph-ml110-01/` - Exists
|
||||
- ✅ `/var/lib/ceph/mgr/ceph-ml110-01/` - Exists
|
||||
- ⚠️ `/var/lib/ceph/osd/` - Empty (no OSDs)
|
||||
|
||||
### R630-01 (192.168.11.11) Status
|
||||
|
||||
**Services:**
|
||||
- ❌ **ceph-mon@r630-01.service**: FAILED
|
||||
- Status: Failed (exit-code)
|
||||
- Error: Monitor data directory does not exist - needs 'mkfs'
|
||||
- Last attempt: Mon 2025-12-15 14:43:39 PST
|
||||
- Issue: Monitor filesystem not created
|
||||
|
||||
- ✅ **ceph-mgr@r630-01.service**: ACTIVE (running)
|
||||
- Status: Running
|
||||
- Started: Mon 2025-12-15 14:39:47 PST
|
||||
- Memory: 326M
|
||||
- Issue: Reporting status from non-daemon mon.r630-01
|
||||
|
||||
**Data Directories:**
|
||||
- ⚠️ `/var/lib/ceph/mon/ceph-r630-01/` - Exists but EMPTY (needs mkfs)
|
||||
- ✅ `/var/lib/ceph/mgr/ceph-r630-01/` - Exists
|
||||
- ⚠️ `/var/lib/ceph/osd/` - Empty (no OSDs)
|
||||
|
||||
### Ceph Cluster Access
|
||||
|
||||
**Status**: ⚠️ **TIMEOUT ISSUES**
|
||||
- `ceph -s`: Timeout
|
||||
- `ceph mon dump`: Timeout
|
||||
- `ceph osd tree`: Timeout
|
||||
- `pveceph status`: Timeout
|
||||
|
||||
**Root Cause**: Monitor on ml110-01 is in "probing" state and cannot form quorum without r630-01 monitor. This causes all Ceph commands to timeout.
|
||||
|
||||
---
|
||||
|
||||
## Configuration Files
|
||||
|
||||
### `/etc/pve/ceph.conf` (Synced across cluster)
|
||||
|
||||
Both nodes have identical configuration:
|
||||
```
|
||||
[global]
|
||||
auth_client_required = cephx
|
||||
auth_cluster_required = cephx
|
||||
auth_service_required = cephx
|
||||
cluster_network = 192.168.11.0/24
|
||||
fsid = f601f0e2-cd09-402f-9e15-4b1c9a7a7b25
|
||||
mon_allow_pool_delete = true
|
||||
mon_host = 192.168.11.10
|
||||
ms_bind_ipv4 = true
|
||||
ms_bind_ipv6 = false
|
||||
osd_pool_default_min_size = 2
|
||||
osd_pool_default_size = 3
|
||||
public_network = 192.168.11.0/24
|
||||
|
||||
[client]
|
||||
keyring = /etc/pve/priv/$cluster.$name.keyring
|
||||
|
||||
[client.crash]
|
||||
keyring = /etc/pve/ceph/$cluster.$name.keyring
|
||||
|
||||
[mon.ml110-01]
|
||||
public_addr = 192.168.11.10
|
||||
```
|
||||
|
||||
**Note**: Configuration only lists ml110-01 in mon_host. r630-01 monitor needs to be added.
|
||||
|
||||
---
|
||||
|
||||
## Issues Identified
|
||||
|
||||
### Critical Issues
|
||||
|
||||
1. **R630-01 Monitor Not Operational**
|
||||
- Monitor service failed
|
||||
- Monitor filesystem not created
|
||||
- Needs: `ceph-mon --mkfs` to create filesystem
|
||||
- Impact: No quorum, cluster commands timeout
|
||||
|
||||
2. **Monitor Quorum Not Established**
|
||||
- Only 1 monitor running (ml110-01)
|
||||
- ml110-01 monitor in "probing" state
|
||||
- Cannot form quorum with single monitor
|
||||
- Impact: Cluster inaccessible
|
||||
|
||||
3. **Ceph Commands Timing Out**
|
||||
- All `ceph` commands timeout
|
||||
- Cannot query cluster status
|
||||
- Impact: Cannot manage cluster
|
||||
|
||||
### Warning Issues
|
||||
|
||||
1. **No OSDs Configured**
|
||||
- No OSDs on either node
|
||||
- Cluster has no storage
|
||||
- Impact: Cannot store data
|
||||
|
||||
2. **Monitor Configuration Incomplete**
|
||||
- `mon_host` only lists ml110-01
|
||||
- r630-01 monitor not in configuration
|
||||
- Impact: r630-01 monitor cannot be discovered
|
||||
|
||||
3. **Slow Operations on Monitor**
|
||||
- ml110-01 monitor reporting 1800+ slow operations
|
||||
- May indicate performance issues
|
||||
- Impact: Degraded performance
|
||||
|
||||
---
|
||||
|
||||
## Required Actions
|
||||
|
||||
### Priority 1: Fix R630-01 Monitor (CRITICAL)
|
||||
|
||||
**Steps:**
|
||||
1. Get monmap and keyring from ml110-01
|
||||
2. Create monitor filesystem on r630-01
|
||||
3. Start monitor service
|
||||
4. Verify quorum established
|
||||
|
||||
**Commands:**
|
||||
```bash
|
||||
# On ml110-01
|
||||
ceph mon getmap -o /tmp/monmap
|
||||
scp /tmp/monmap root@192.168.11.11:/tmp/
|
||||
scp /etc/pve/priv/ceph.mon.keyring root@192.168.11.11:/tmp/mon-keyring
|
||||
|
||||
# On r630-01
|
||||
mkdir -p /var/lib/ceph/mon/ceph-r630-01
|
||||
chown ceph:ceph /var/lib/ceph/mon/ceph-r630-01
|
||||
ceph-mon --mkfs -i r630-01 --monmap /tmp/monmap --keyring /tmp/mon-keyring
|
||||
chown -R ceph:ceph /var/lib/ceph/mon/ceph-r630-01
|
||||
systemctl start ceph-mon@r630-01.service
|
||||
```
|
||||
|
||||
### Priority 2: Update Monitor Configuration
|
||||
|
||||
**Steps:**
|
||||
1. Add r630-01 to mon_host in ceph.conf
|
||||
2. Add [mon.r630-01] section
|
||||
3. Verify configuration syncs to both nodes
|
||||
|
||||
### Priority 3: Verify Quorum
|
||||
|
||||
**Steps:**
|
||||
1. Check monitor quorum: `ceph quorum_status`
|
||||
2. Verify both monitors in quorum
|
||||
3. Test cluster access: `ceph -s`
|
||||
|
||||
### Priority 4: Add OSDs
|
||||
|
||||
**Steps:**
|
||||
1. Identify available disks (6x 250GB SSDs on r630-01)
|
||||
2. Create OSDs using `pveceph osd create`
|
||||
3. Verify OSD status
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
### ✅ Working
|
||||
- Proxmox cluster: Healthy, quorum established
|
||||
- Ceph packages: Installed on both nodes
|
||||
- Ceph manager: Running on both nodes
|
||||
- Ceph monitor: Running on ml110-01
|
||||
- Configuration: Synced across cluster
|
||||
|
||||
### ⚠️ Needs Attention
|
||||
- Ceph monitor: r630-01 monitor needs filesystem creation
|
||||
- Monitor quorum: Cannot form quorum with single monitor
|
||||
- Cluster access: Commands timing out
|
||||
- OSDs: No OSDs configured
|
||||
|
||||
### ❌ Blocking Issues
|
||||
- R630-01 monitor not operational (blocks quorum)
|
||||
- Cluster commands timing out (blocks management)
|
||||
|
||||
---
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. **Fix r630-01 monitor** (see Priority 1 above)
|
||||
2. **Update monitor configuration** to include r630-01
|
||||
3. **Verify quorum** and cluster access
|
||||
4. **Add OSDs** once cluster is operational
|
||||
5. **Monitor performance** for slow operations
|
||||
|
||||
---
|
||||
|
||||
## Notes
|
||||
|
||||
- Proxmox cluster "sfvalley-01" is healthy and operational
|
||||
- Ceph cluster exists but needs monitor quorum to be fully operational
|
||||
- All configuration is synced via Proxmox cluster filesystem
|
||||
- Both nodes have Ceph installed and partially configured
|
||||
- Once monitor quorum is established, cluster should become fully accessible
|
||||
|
||||
337
docs/infrastructure/DISK_INVENTORY.md
Normal file
337
docs/infrastructure/DISK_INVENTORY.md
Normal file
@@ -0,0 +1,337 @@
|
||||
# Physical Disk Inventory - ML110-01 and R630-01
|
||||
|
||||
**Date**: 2025-12-13
|
||||
**Purpose**: Document physical disk configuration for Ceph OSD planning
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
Both nodes currently have **2 physical disks each**, with one disk used for Proxmox installation and one disk used for Ceph OSD. This results in **2 OSDs total**, which is insufficient for 3-way replication.
|
||||
|
||||
**IMPORTANT UPDATE**: R630-01 actually has **8 total drive bays** with **6 additional 250GB drives** that may be available for use.
|
||||
|
||||
### Current Ceph Configuration
|
||||
- **OSD Count**: 2 (one per node)
|
||||
- **Required for 3-way replication**: 3 OSDs minimum
|
||||
- **Status**: 🔴 **INSUFFICIENT** - Cannot maintain 3-way replication
|
||||
- **Available Disks**: 6x 250GB drives on R630-01 (potential for additional OSDs)
|
||||
|
||||
---
|
||||
|
||||
## ML110-01 (HPE ML110 Gen9)
|
||||
|
||||
**IP Address**: 192.168.11.10
|
||||
**Hostname**: ml110-01
|
||||
**Form Factor**: Tower Server
|
||||
|
||||
### Physical Disks
|
||||
|
||||
| Disk | Model | Size | Type | Usage | Status |
|
||||
|------|-------|------|------|-------|--------|
|
||||
| **sda** | Seagate ST1000DM003-1ER162 | 931.5 GB | SATA HDD | Proxmox Installation (LVM) | ✅ Active |
|
||||
| **sdb** | Seagate ST1000DM003-1ER162 | 931.5 GB | SATA HDD | Ceph OSD | ✅ Active |
|
||||
|
||||
### Storage Configuration
|
||||
|
||||
**Primary Disk (sda)**:
|
||||
- **Partition Layout**:
|
||||
- `pve-swap`: 8 GB
|
||||
- `pve-root`: 96 GB
|
||||
- `pve-data`: 794.3 GB (for VMs using local-lvm)
|
||||
- **Storage Pool**: `local-lvm` (794.3 GB available)
|
||||
|
||||
**Secondary Disk (sdb)**:
|
||||
- **Usage**: Ceph OSD block device
|
||||
- **Storage Pool**: `ceph-fs` (384 GB available)
|
||||
- **Status**: Active Ceph OSD
|
||||
|
||||
### Storage Controller
|
||||
- **Type**: Intel C610/X99 series chipset 6-Port SATA Controller (AHCI mode)
|
||||
- **RAID**: Software-based (LVM, Ceph) - no hardware RAID controller
|
||||
|
||||
### Available Drive Bays
|
||||
- **Form Factor**: Tower server
|
||||
- **Drive Bays**: [To be determined - likely 2-4 internal bays]
|
||||
- **Expansion**: Limited by tower form factor
|
||||
|
||||
### Recommendations
|
||||
- ✅ **Current Configuration**: Adequate for current setup
|
||||
- ⚠️ **Ceph Expansion**: Limited expansion options (tower form factor)
|
||||
- 💡 **Option**: Add external storage or use network storage for third OSD
|
||||
|
||||
---
|
||||
|
||||
## R630-01 (Dell PowerEdge R630 High Memory)
|
||||
|
||||
**IP Address**: 192.168.11.11
|
||||
**Hostname**: r630-01
|
||||
**Form Factor**: 1U Rack Server
|
||||
**Serial Number**: HNQ3FB2
|
||||
|
||||
### Physical Disks (8 Total Bays)
|
||||
|
||||
| Bay | Disk | Model | Size | Type | Usage | Status |
|
||||
|-----|------|-------|------|------|-------|--------|
|
||||
| **1** | sda | Seagate ST9300653SS | 279.4 GB | SAS SSD | Proxmox Installation (LVM) | ✅ Active |
|
||||
| **2** | sdb | HUC106030CSS600 | 279.4 GB | SAS SSD | Ceph OSD | ✅ Active |
|
||||
| **3** | ? | [Unknown] | 250 GB | [Unknown] | **UNUSED** | ⚠️ Available? |
|
||||
| **4** | ? | [Unknown] | 250 GB | [Unknown] | **UNUSED** | ⚠️ Available? |
|
||||
| **5** | ? | [Unknown] | 250 GB | [Unknown] | **UNUSED** | ⚠️ Available? |
|
||||
| **6** | ? | [Unknown] | 250 GB | [Unknown] | **UNUSED** | ⚠️ Available? |
|
||||
| **7** | ? | [Unknown] | 250 GB | [Unknown] | **UNUSED** | ⚠️ Available? |
|
||||
| **8** | ? | [Unknown] | 250 GB | [Unknown] | **UNUSED** | ⚠️ Available? |
|
||||
|
||||
**Note**: The 6x 250GB drives need to be verified - they may be:
|
||||
- Not detected by the OS
|
||||
- Not initialized/formatted
|
||||
- Used for something else (RAID, backup, etc.)
|
||||
- Available for Ceph OSD creation
|
||||
|
||||
### Storage Configuration
|
||||
|
||||
**Primary Disk (sda - Bay 1)**:
|
||||
- **Partition Layout**:
|
||||
- `pve-swap`: 8 GB
|
||||
- `pve-root`: 79.6 GB
|
||||
- `pve-data`: 171.3 GB (for VMs using local-lvm)
|
||||
- **Storage Pool**: `local-lvm` (171.3 GB available)
|
||||
|
||||
**Secondary Disk (sdb - Bay 2)**:
|
||||
- **Usage**: Ceph OSD block device
|
||||
- **Storage Pool**: Ceph OSD
|
||||
- **Status**: Active Ceph OSD
|
||||
|
||||
**6x 250GB Drives (Bays 3-8)**:
|
||||
- **Status**: ⚠️ **NEEDS VERIFICATION**
|
||||
- **Potential Usage**:
|
||||
- Available for additional Ceph OSDs (ideal!)
|
||||
- Could create 3rd OSD immediately
|
||||
- Could create more OSDs for better performance
|
||||
|
||||
### Storage Controller
|
||||
- **Type**: Dell PERC H730 Mini (LSI MegaRAID SAS-3 3108 [Invader])
|
||||
- **RAID**: Hardware RAID controller available
|
||||
- **Mode**: Currently using individual disks (not in RAID)
|
||||
|
||||
### Available Drive Bays
|
||||
- **Form Factor**: 1U Rack Server
|
||||
- **Total Drive Bays**: **8x 2.5" hot-swappable drive bays**
|
||||
- **Currently Used**: 2 bays (sda, sdb)
|
||||
- **Additional Drives**: **6x 250GB drives** (status unknown)
|
||||
- **Storage Options**: SATA, SAS, NVMe (with riser card)
|
||||
|
||||
### Recommendations
|
||||
- ✅ **Excellent Expansion Potential**: 6 additional drives available
|
||||
- ✅ **Hot-Swappable**: Can add/configure drives without downtime
|
||||
- 🔍 **URGENT**: Verify status of 6x 250GB drives
|
||||
- 💡 **Best Option**: Use one of the 250GB drives for third OSD
|
||||
- 💡 **Future Expansion**: Can use remaining 5 drives for more OSDs
|
||||
|
||||
---
|
||||
|
||||
## Ceph OSD Status
|
||||
|
||||
### Current OSDs
|
||||
1. **OSD 0**: ML110-01 (sdb) - 931.5 GB
|
||||
2. **OSD 1**: R630-01 (sdb) - 279.4 GB
|
||||
|
||||
### Issues
|
||||
- 🔴 **Only 2 OSDs** - Cannot maintain 3-way replication
|
||||
- 🔴 **Mismatched Sizes** - 931.5 GB vs 279.4 GB (suboptimal)
|
||||
- 🔴 **Insufficient for Production** - Need minimum 3 OSDs
|
||||
|
||||
### Potential Additional OSDs
|
||||
- ✅ **6x 250GB drives available** on R630-01
|
||||
- 💡 **Can create 3rd OSD immediately** (if drives are usable)
|
||||
- 💡 **Can create more OSDs** for better performance and redundancy
|
||||
|
||||
---
|
||||
|
||||
## Recommendations for Adding Third OSD
|
||||
|
||||
### Option 1: Use Existing 250GB Drive on R630-01 (BEST) ✅
|
||||
|
||||
**Advantages**:
|
||||
- ✅ **No hardware purchase needed** - Drive already installed
|
||||
- ✅ **Hot-swappable** - Can configure without downtime
|
||||
- ✅ **Immediate solution** - Can create OSD right away
|
||||
- ✅ **Size appropriate** - 250GB is close to existing 279.4GB OSD
|
||||
|
||||
**Steps**:
|
||||
1. **Verify drive is detected**:
|
||||
```bash
|
||||
ssh root@192.168.11.11 "lsblk -o NAME,SIZE,TYPE,MOUNTPOINT,FSTYPE,MODEL"
|
||||
```
|
||||
|
||||
2. **Check if drive is initialized**:
|
||||
```bash
|
||||
ssh root@192.168.11.11 "fdisk -l | grep -E '^Disk /dev/'"
|
||||
```
|
||||
|
||||
3. **Create Ceph OSD** (if drive is available):
|
||||
```bash
|
||||
# On R630-01
|
||||
ceph-volume lvm create --data /dev/sdX # Replace sdX with actual device
|
||||
```
|
||||
|
||||
4. **Verify OSD joins cluster**:
|
||||
```bash
|
||||
ceph osd tree
|
||||
ceph health
|
||||
```
|
||||
|
||||
### Option 2: Add New Disk to R630-01
|
||||
|
||||
**Advantages**:
|
||||
- ✅ **8 empty drive bays** available (if 250GB drives not usable)
|
||||
- ✅ **Hot-swappable** - can add without downtime
|
||||
- ✅ **Better performance** - Can choose SSD
|
||||
|
||||
**Disadvantages**:
|
||||
- ⚠️ Requires hardware purchase
|
||||
- ⚠️ Takes time to procure and install
|
||||
|
||||
### Option 3: Reduce Replication Factor (TEMPORARY WORKAROUND)
|
||||
|
||||
**Action**: Change replication from 3 to 2
|
||||
|
||||
**Advantages**:
|
||||
- ✅ **Immediate** - No hardware needed
|
||||
- ✅ **Allows VM creation** - Unblocks 21 VMs
|
||||
|
||||
**Disadvantages**:
|
||||
- ⚠️ **Increased Risk** - Data loss if 1 OSD fails
|
||||
- ⚠️ **Not Production-Ready** - Should be temporary only
|
||||
- ⚠️ **Requires Monitoring** - Need to watch for failures
|
||||
|
||||
**Steps**:
|
||||
```bash
|
||||
# Reduce replication factor
|
||||
ceph osd pool set rbd size 2
|
||||
ceph osd pool set rbd min_size 1
|
||||
|
||||
# Verify
|
||||
ceph osd pool get rbd size
|
||||
ceph osd pool get rbd min_size
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Disk Size Considerations
|
||||
|
||||
### Current Mismatch
|
||||
- **ML110-01 OSD**: 931.5 GB
|
||||
- **R630-01 OSD (current)**: 279.4 GB
|
||||
- **R630-01 OSD (potential)**: 250 GB
|
||||
- **Difference**: Significant size mismatch
|
||||
|
||||
### Impact
|
||||
- ⚠️ **Uneven Data Distribution** - More data on larger OSD
|
||||
- ⚠️ **Suboptimal Performance** - Larger OSD handles more load
|
||||
- ⚠️ **Wasted Space** - Cannot use full capacity of larger OSD
|
||||
|
||||
### Recommendations
|
||||
1. **For Third OSD**: Use 250GB drive (close to existing 279.4GB)
|
||||
2. **For Future**: Consider replacing smaller OSDs with larger ones
|
||||
3. **For Production**: Use similar-sized disks for better balance
|
||||
4. **For Optimal Setup**: Consider using multiple 250GB drives for more OSDs
|
||||
|
||||
---
|
||||
|
||||
## Next Steps
|
||||
|
||||
### Immediate (To Fix Ceph Issues)
|
||||
|
||||
1. **URGENT - Verify 250GB Drives**:
|
||||
```bash
|
||||
# Check if drives are detected
|
||||
ssh root@192.168.11.11 "lsblk -o NAME,SIZE,TYPE,MOUNTPOINT,FSTYPE,MODEL"
|
||||
|
||||
# Check all block devices
|
||||
ssh root@192.168.11.11 "fdisk -l | grep -E '^Disk /dev/'"
|
||||
|
||||
# Check Ceph OSD status
|
||||
ssh root@192.168.11.11 "ceph osd tree"
|
||||
```
|
||||
|
||||
2. **Option A - Use Existing 250GB Drive** (BEST):
|
||||
- Verify drive is detected and usable
|
||||
- Create Ceph OSD on one of the 250GB drives
|
||||
- Verify cluster health improves
|
||||
|
||||
3. **Option B - Reduce Replication** (TEMPORARY):
|
||||
- Change replication from 3 to 2
|
||||
- Monitor for failures
|
||||
- Plan to add third OSD using 250GB drive
|
||||
|
||||
### Short-term (This Week)
|
||||
|
||||
1. Verify and configure 250GB drives
|
||||
2. Create third OSD on R630-01
|
||||
3. Verify Ceph cluster health improves
|
||||
4. Monitor OSD performance
|
||||
5. Consider adding more OSDs from remaining 250GB drives
|
||||
|
||||
### Long-term (This Month)
|
||||
|
||||
1. Add more OSDs from remaining 250GB drives (better performance)
|
||||
2. Balance OSD sizes for optimal distribution
|
||||
3. Plan for storage expansion
|
||||
4. Document storage architecture
|
||||
|
||||
---
|
||||
|
||||
## Verification Commands
|
||||
|
||||
### Check Current Disks on R630-01
|
||||
```bash
|
||||
# List all block devices
|
||||
ssh root@192.168.11.11 "lsblk -o NAME,SIZE,TYPE,MOUNTPOINT,FSTYPE,MODEL,SERIAL"
|
||||
|
||||
# Check all disks
|
||||
ssh root@192.168.11.11 "fdisk -l | grep -E '^Disk /dev/'"
|
||||
|
||||
# Check for uninitialized drives
|
||||
ssh root@192.168.11.11 "ls -la /dev/sd*"
|
||||
```
|
||||
|
||||
### Check Ceph OSDs
|
||||
```bash
|
||||
# On either node
|
||||
ceph osd tree
|
||||
ceph osd df
|
||||
ceph osd pool ls detail
|
||||
ceph health detail
|
||||
```
|
||||
|
||||
### Check Available Drive Bays
|
||||
```bash
|
||||
# Check via iDRAC/IPMI (if available)
|
||||
# Or physical inspection
|
||||
# Or check PERC controller status
|
||||
ssh root@192.168.11.11 "omreport storage vdisk" # If Dell tools installed
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
### Current State
|
||||
- **ML110-01**: 2 disks (1 Proxmox, 1 Ceph OSD)
|
||||
- **R630-01**: 8 total bays
|
||||
- 2x 300GB drives (1 Proxmox, 1 Ceph OSD)
|
||||
- 6x 250GB drives (status unknown - needs verification)
|
||||
- **Total OSDs**: 2 (insufficient for 3-way replication)
|
||||
|
||||
### Best Solution
|
||||
✅ **Use one of the 6x 250GB drives on R630-01** - No hardware purchase needed, immediate solution
|
||||
|
||||
### Action Required
|
||||
🔍 **URGENT**: Verify status of 6x 250GB drives and configure one for third OSD
|
||||
|
||||
### Temporary Workaround
|
||||
⚠️ **Reduce replication to 2** - Allows VM creation but increases risk (use only if 250GB drives not usable)
|
||||
|
||||
**Last Updated**: 2025-12-13
|
||||
**Status**: 🔍 **VERIFY 250GB DRIVES - POTENTIAL IMMEDIATE SOLUTION AVAILABLE**
|
||||
212
docs/infrastructure/NEXT_STEPS.md
Normal file
212
docs/infrastructure/NEXT_STEPS.md
Normal file
@@ -0,0 +1,212 @@
|
||||
# Infrastructure Next Steps
|
||||
|
||||
**Date**: 2025-12-15
|
||||
**Status**: Assessment Complete - Action Items Identified
|
||||
|
||||
## Current State Summary
|
||||
|
||||
### ✅ Completed
|
||||
1. **SSH Key Authentication**: Set up passwordless SSH access to both servers
|
||||
- ✅ r630-01 (192.168.11.11) - SSH key configured
|
||||
- ✅ ml110-01 (192.168.11.10) - SSH key configured
|
||||
2. **Disk Layout Documentation**: Comprehensive fdisk output captured
|
||||
- ✅ r630-01 disk layout documented
|
||||
- ✅ ml110-01 disk layout documented
|
||||
|
||||
### ⚠️ Current Issues Identified
|
||||
|
||||
#### 1. Ceph Installation Status (CRITICAL)
|
||||
- **Issue**: Only `ceph-common` and `ceph-fuse` packages installed, missing `ceph-mon` and `ceph-osd`
|
||||
- **Status**: `pveceph` command reports "binary not installed"
|
||||
- **Current State**:
|
||||
- `/var/lib/ceph/` directories exist but are empty (no mon/ or osd/ subdirectories)
|
||||
- Ceph client tools available but cluster services not installed
|
||||
- **Previous State**: Documentation indicates MON recovery was successful on 2025-12-13, but cluster appears to have been removed
|
||||
- **Action Required**: Install full Ceph packages or reinitialize cluster
|
||||
|
||||
#### 2. Disk Health - R630-01 `/dev/sda` (WARNING)
|
||||
- **Model**: ST9300653SS (Seagate 300GB SAS, 15K RPM)
|
||||
- **SMART Status**: OK (but showing signs of wear)
|
||||
- **Age**: Manufactured 2014, ~45,532 hours powered on (~5.2 years)
|
||||
- **Concerns**:
|
||||
- 2,186 elements in grown defect list
|
||||
- 456,913 total uncorrected read errors
|
||||
- 379,382,059 total errors corrected
|
||||
- **Risk**: Disk is aging and showing significant error correction activity
|
||||
- **Action Required**: Monitor closely, plan for replacement
|
||||
|
||||
#### 3. Ceph Cluster Accessibility
|
||||
- **Status**: Cannot access cluster (`ceph -s` fails with config errors)
|
||||
- **Possible Causes**:
|
||||
- Ceph not fully installed
|
||||
- Configuration files missing
|
||||
- Cluster not initialized
|
||||
- **Action Required**: Verify Ceph installation and configuration
|
||||
|
||||
## Immediate Next Steps
|
||||
|
||||
### Priority 1: Verify Ceph Installation Status
|
||||
|
||||
**Check if Ceph is installed:**
|
||||
```bash
|
||||
# On both servers
|
||||
ssh root@192.168.11.10 "dpkg -l | grep ceph"
|
||||
ssh root@192.168.11.11 "dpkg -l | grep ceph"
|
||||
|
||||
# Check for Ceph packages
|
||||
ssh root@192.168.11.10 "apt list --installed | grep ceph"
|
||||
ssh root@192.168.11.11 "apt list --installed | grep ceph"
|
||||
```
|
||||
|
||||
**If Ceph is not installed:**
|
||||
- Follow Proxmox Ceph installation guide
|
||||
- Or use existing scripts in `scripts/` directory
|
||||
|
||||
**If Ceph is installed but binaries missing:**
|
||||
- Reinstall Ceph packages: `apt install --reinstall ceph-common ceph-mon ceph-osd`
|
||||
- Verify Proxmox Ceph integration: `pveceph install`
|
||||
|
||||
### Priority 2: Assess Disk Health and Plan Replacement
|
||||
|
||||
**Monitor disk health:**
|
||||
```bash
|
||||
# Check SMART attributes
|
||||
ssh root@192.168.11.11 "smartctl -a /dev/sda"
|
||||
|
||||
# Check for I/O errors in dmesg
|
||||
ssh root@192.168.11.11 "dmesg | grep -i 'sda.*error' | tail -20"
|
||||
|
||||
# Check filesystem health
|
||||
ssh root@192.168.11.11 "fsck -n /dev/mapper/pve-root"
|
||||
```
|
||||
|
||||
**Plan disk replacement:**
|
||||
- R630-01 has `/dev/sdb` (279.4GB) available as secondary disk
|
||||
- Consider moving critical services (like Ceph MON) to `/dev/sdb`
|
||||
- Or plan full disk replacement for `/dev/sda`
|
||||
|
||||
### Priority 3: Verify Ceph Cluster State
|
||||
|
||||
**If Ceph is installed, check cluster status:**
|
||||
```bash
|
||||
# Check if Ceph services exist
|
||||
ssh root@192.168.11.10 "systemctl list-units | grep ceph"
|
||||
ssh root@192.168.11.11 "systemctl list-units | grep ceph"
|
||||
|
||||
# Check for Ceph configuration
|
||||
ssh root@192.168.11.10 "ls -la /etc/ceph/ /etc/pve/ceph.conf"
|
||||
ssh root@192.168.11.11 "ls -la /etc/ceph/ /etc/pve/ceph.conf"
|
||||
|
||||
# Check for MON data directories
|
||||
ssh root@192.168.11.10 "ls -la /var/lib/ceph/mon/"
|
||||
ssh root@192.168.11.11 "ls -la /var/lib/ceph/mon/"
|
||||
|
||||
# Check for OSD data
|
||||
ssh root@192.168.11.10 "ls -la /var/lib/ceph/osd/"
|
||||
ssh root@192.168.11.11 "ls -la /var/lib/ceph/osd/"
|
||||
```
|
||||
|
||||
## Medium-Term Actions
|
||||
|
||||
### 1. Ceph Cluster Setup/Recovery
|
||||
|
||||
**If cluster needs to be initialized:**
|
||||
- Use existing scripts: `scripts/fresh-ceph-install-r630.sh`
|
||||
- Or follow Proxmox Ceph setup guide
|
||||
|
||||
**If cluster exists but is inaccessible:**
|
||||
- Review recovery documentation: `docs/ceph/MON_RECOVERY_SUCCESS.md`
|
||||
- Check if MON services need to be recreated
|
||||
- Verify network connectivity between nodes
|
||||
|
||||
### 2. Add Additional Ceph OSDs
|
||||
|
||||
**R630-01 has 6x 250GB SSDs available:**
|
||||
- `/dev/sdc` through `/dev/sdh` (232.89 GiB each)
|
||||
- Total capacity: ~1.4TB
|
||||
- Use script: `scripts/setup-ceph-osds-r630.sh`
|
||||
- Or manual setup following: `docs/ceph/ADD_THIRD_OSD_GUIDE.md`
|
||||
|
||||
**Benefits:**
|
||||
- Increase cluster capacity
|
||||
- Improve redundancy (currently only 2 OSDs)
|
||||
- Fix "undersized PGs" warnings
|
||||
|
||||
### 3. Disk Replacement Planning
|
||||
|
||||
**For R630-01 `/dev/sda`:**
|
||||
- Monitor SMART attributes weekly
|
||||
- Plan replacement when errors increase
|
||||
- Consider moving MON to `/dev/sdb` as interim solution
|
||||
- Document replacement procedure
|
||||
|
||||
## Long-Term Improvements
|
||||
|
||||
### 1. Infrastructure Monitoring
|
||||
- Set up SMART monitoring for all disks
|
||||
- Configure alerts for disk health degradation
|
||||
- Monitor Ceph cluster health automatically
|
||||
|
||||
### 2. Backup Strategy
|
||||
- Regular backups of Ceph MON data directories
|
||||
- Backup Proxmox configurations
|
||||
- Document recovery procedures
|
||||
|
||||
### 3. Redundancy Improvements
|
||||
- Add third Ceph MON (if possible)
|
||||
- Ensure OSD count meets replication requirements
|
||||
- Plan for hardware failures
|
||||
|
||||
### 4. Documentation Updates
|
||||
- Update disk layout docs as changes occur
|
||||
- Document all recovery procedures
|
||||
- Maintain runbook for common issues
|
||||
|
||||
## Available Scripts and Resources
|
||||
|
||||
### Ceph Setup Scripts
|
||||
- `scripts/fresh-ceph-install-r630.sh` - Fresh Ceph installation
|
||||
- `scripts/setup-ceph-osds-r630.sh` - OSD setup for R630-01
|
||||
- `scripts/reinitialize-ceph-cluster.sh` - Cluster reinitialization
|
||||
|
||||
### Disk Management Scripts
|
||||
- `scripts/analyze-r630-disk-layout.sh` - Disk analysis
|
||||
- `scripts/verify-r630-250gb-drives.sh` - SSD verification
|
||||
|
||||
### SSH Helper Scripts
|
||||
- `scripts/ssh-r630-01.sh` - SSH to R630-01
|
||||
- `scripts/ssh-ml110-01.sh` - SSH to ML110-01
|
||||
|
||||
### Documentation
|
||||
- `docs/infrastructure/` - Infrastructure documentation
|
||||
- `docs/ceph/` - Ceph-specific documentation
|
||||
|
||||
## Recommended Action Order
|
||||
|
||||
1. **Verify Ceph installation** (15 minutes)
|
||||
- Check packages, binaries, configuration
|
||||
- Determine if reinstall needed
|
||||
|
||||
2. **Assess cluster state** (30 minutes)
|
||||
- Check services, data directories
|
||||
- Determine if recovery needed
|
||||
|
||||
3. **Monitor disk health** (ongoing)
|
||||
- Set up monitoring
|
||||
- Plan replacement timeline
|
||||
|
||||
4. **Add OSDs** (1-2 hours)
|
||||
- Once cluster is stable
|
||||
- Use 6x SSDs on R630-01
|
||||
|
||||
5. **Plan disk replacement** (planning)
|
||||
- Schedule replacement
|
||||
- Document procedure
|
||||
|
||||
## Notes
|
||||
|
||||
- All SSH access is now configured with key authentication
|
||||
- Disk layouts are fully documented
|
||||
- Previous recovery documentation exists but cluster state needs verification
|
||||
- Disk health on R630-01 `/dev/sda` requires attention
|
||||
|
||||
155
docs/infrastructure/ml110-01-fdisk-output.md
Normal file
155
docs/infrastructure/ml110-01-fdisk-output.md
Normal file
@@ -0,0 +1,155 @@
|
||||
# ML110-01 fdisk Output - Detailed Disk Information
|
||||
|
||||
**Date**: Generated from `fdisk -l` command
|
||||
**Host**: ml110-01 (192.168.11.10)
|
||||
|
||||
## Complete fdisk Output
|
||||
|
||||
```
|
||||
Disk /dev/sda: 931.51 GiB, 1000204886016 bytes, 1953525168 sectors
|
||||
Disk model: ST1000DM003-1ER1
|
||||
Units: sectors of 1 * 512 = 512 bytes
|
||||
Sector size (logical/physical): 512 bytes / 4096 bytes
|
||||
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
|
||||
Disklabel type: gpt
|
||||
Disk identifier: F9CFB0F5-ED1C-477B-9656-5D2651A864E1
|
||||
|
||||
Device Start End Sectors Size Type
|
||||
/dev/sda1 34 2047 2014 1007K BIOS boot
|
||||
/dev/sda2 2048 2099199 2097152 1G EFI System
|
||||
/dev/sda3 2099200 1953525134 1951425935 930.5G Linux LVM
|
||||
|
||||
Partition 1 does not start on physical sector boundary.
|
||||
|
||||
|
||||
Disk /dev/sdb: 931.51 GiB, 1000204886016 bytes, 1953525168 sectors
|
||||
Disk model: ST1000DM003-1ER1
|
||||
Units: sectors of 1 * 512 = 512 bytes
|
||||
Sector size (logical/physical): 512 bytes / 4096 bytes
|
||||
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
|
||||
|
||||
|
||||
Disk /dev/mapper/ceph--ba672668--8341--42e0--a8fd--277cee5491e4-osd--block--fcaaaa49--68ea--4664--8559--7f76088f246d: 931.51 GiB, 1000203091968 bytes, 1953521664 sectors
|
||||
Units: sectors of 1 * 512 = 512 bytes
|
||||
Sector size (logical/physical): 512 bytes / 4096 bytes
|
||||
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
|
||||
|
||||
|
||||
Disk /dev/mapper/pve-swap: 8 GiB, 8589934592 bytes, 16777216 sectors
|
||||
Units: sectors of 1 * 512 = 512 bytes
|
||||
Sector size (logical/physical): 512 bytes / 4096 bytes
|
||||
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
|
||||
|
||||
|
||||
Disk /dev/mapper/pve-root: 96 GiB, 103079215104 bytes, 201326592 sectors
|
||||
Units: sectors of 1 * 512 = 512 bytes
|
||||
Sector size (logical/physical): 512 bytes / 4096 bytes
|
||||
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
|
||||
|
||||
|
||||
Disk /dev/sdc: 28.91 GiB, 31042043904 bytes, 60628992 sectors
|
||||
Disk model: USB 2.0 FD
|
||||
Units: sectors of 1 * 512 = 512 bytes
|
||||
Sector size (logical/physical): 512 bytes / 512 bytes
|
||||
I/O size (minimum/optimal): 512 bytes / 512 bytes
|
||||
Disklabel type: gpt
|
||||
Disk identifier: B9083D28-1E94-45BC-A8A3-DA68BE09A9A9
|
||||
|
||||
Device Start End Sectors Size Type
|
||||
/dev/sdc1 64 635 572 286K Microsoft basic data
|
||||
/dev/sdc2 636 17019 16384 8M EFI System
|
||||
/dev/sdc3 17020 3577255 3560236 1.7G Apple HFS/HFS+
|
||||
/dev/sdc4 3577256 3577855 600 300K Microsoft basic data
|
||||
GPT PMBR size mismatch (3577903 != 60628991) will be corrected by write.
|
||||
```
|
||||
|
||||
## Disk Analysis
|
||||
|
||||
### System Disk: `/dev/sda` (931.51 GiB / 1TB)
|
||||
- **Model**: ST1000DM003-1ER1 (Seagate Barracuda 1TB HDD)
|
||||
- **Partition Table**: GPT
|
||||
- **Disk Identifier**: F9CFB0F5-ED1C-477B-9656-5D2651A864E1
|
||||
- **Sector Size**: 512 bytes logical / 4096 bytes physical (4K native)
|
||||
- **I/O Size**: 4096 bytes (optimal)
|
||||
- **Partitions**:
|
||||
- `/dev/sda1`: 1007K (sectors 34-2047) - BIOS boot partition
|
||||
- **Note**: Partition 1 does not start on physical sector boundary (minor alignment issue)
|
||||
- `/dev/sda2`: 1G (sectors 2048-2099199) - EFI System partition
|
||||
- `/dev/sda3`: 930.5G (sectors 2099200-1953525134) - Linux LVM
|
||||
- **LVM Volumes**:
|
||||
- `/dev/mapper/pve-swap`: 8 GiB
|
||||
- `/dev/mapper/pve-root`: 96 GiB
|
||||
- Additional space likely in `pve-data` volume (not shown in fdisk output)
|
||||
|
||||
### Ceph OSD Disk: `/dev/sdb` (931.51 GiB / 1TB)
|
||||
- **Model**: ST1000DM003-1ER1 (Seagate Barracuda 1TB HDD)
|
||||
- **Status**: Used by Ceph OSD
|
||||
- **Sector Size**: 512 bytes logical / 4096 bytes physical (4K native)
|
||||
- **I/O Size**: 4096 bytes (optimal)
|
||||
- **Ceph OSD Block Device**:
|
||||
- `/dev/mapper/ceph--ba672668--8341--42e0--a8fd--277cee5491e4-osd--block--fcaaaa49--68ea--4664--8559--7f76088f246d`
|
||||
- Size: 931.51 GiB (full disk used for OSD)
|
||||
- **OSD ID**: Likely OSD 0 or 1 (check with `ceph osd tree`)
|
||||
|
||||
### USB Drive: `/dev/sdc` (28.91 GiB)
|
||||
- **Model**: USB 2.0 FD (USB Flash Drive)
|
||||
- **Partition Table**: GPT
|
||||
- **Disk Identifier**: B9083D28-1E94-45BC-A8A3-DA68BE09A9A9
|
||||
- **Sector Size**: 512 bytes (logical/physical)
|
||||
- **Partitions**:
|
||||
- `/dev/sdc1`: 286K - Microsoft basic data
|
||||
- `/dev/sdc2`: 8M - EFI System
|
||||
- `/dev/sdc3`: 1.7G - Apple HFS/HFS+
|
||||
- `/dev/sdc4`: 300K - Microsoft basic data
|
||||
- **Status**: Appears to be a bootable USB drive (possibly macOS installer or boot media)
|
||||
- **Warning**: GPT PMBR size mismatch detected (partition table may need correction)
|
||||
|
||||
## Key Observations
|
||||
|
||||
1. **System Disk (`/dev/sda`)**:
|
||||
- 1TB HDD with Proxmox VE installation
|
||||
- Properly partitioned with GPT
|
||||
- Has both BIOS boot and EFI System partitions
|
||||
- LVM configured with swap and root filesystem
|
||||
- 4K native physical sector size (modern HDD standard)
|
||||
|
||||
2. **Ceph OSD Disk (`/dev/sdb`)**:
|
||||
- 1TB HDD dedicated to Ceph OSD
|
||||
- Full disk used for Ceph block device
|
||||
- Matches system disk model (same hardware)
|
||||
- Active Ceph OSD in use
|
||||
|
||||
3. **USB Drive (`/dev/sdc`)**:
|
||||
- 28.91GB USB flash drive
|
||||
- Contains bootable partitions (EFI, HFS+)
|
||||
- Likely installation media or boot drive
|
||||
- GPT partition table has size mismatch warning
|
||||
|
||||
4. **LVM Configuration**:
|
||||
- `pve-swap`: 8GB swap space
|
||||
- `pve-root`: 96GB root filesystem (larger than r630-01's 79.5GB)
|
||||
- Remaining space in LVM likely allocated to `pve-data`
|
||||
|
||||
## Comparison with R630-01
|
||||
|
||||
| Feature | ML110-01 | R630-01 |
|
||||
|---------|----------|---------|
|
||||
| System Disk | 1TB HDD (sda) | 300GB SAS (sda) |
|
||||
| Ceph OSD Disk | 1TB HDD (sdb) - Active | 6x 250GB SSDs (sdc-sdh) - Available |
|
||||
| Root Filesystem | 96GB | 79.5GB |
|
||||
| Swap | 8GB | 8GB |
|
||||
| Additional Storage | USB drive (sdc) | Secondary 300GB SAS (sdb) |
|
||||
|
||||
## Recommendations
|
||||
|
||||
- **System Disk**: Keep `/dev/sda` as-is for Proxmox VE OS
|
||||
- **Ceph OSD**: `/dev/sdb` is already configured and active
|
||||
- **USB Drive**: `/dev/sdc` appears to be temporary/installation media - can be removed if not needed
|
||||
- **Storage Expansion**: ML110-01 has less available storage than R630-01 (which has 6x SSDs available)
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [R630-01 fdisk Output](./r630-01-fdisk-output.md) - Comparison disk layout
|
||||
- [R630-01 Disk Layout Reference](./r630-01-disk-layout.md) - R630-01 overview
|
||||
- [Ceph OSD Setup Documentation](../ceph/) - Ceph configuration guides
|
||||
|
||||
117
docs/infrastructure/r630-01-disk-layout.md
Normal file
117
docs/infrastructure/r630-01-disk-layout.md
Normal file
@@ -0,0 +1,117 @@
|
||||
# R630-01 Disk Layout Reference
|
||||
|
||||
## Overview
|
||||
This document describes the disk layout for the R630-01 Proxmox VE host based on `lsblk` output.
|
||||
|
||||
## Disk Inventory
|
||||
|
||||
### System Disk: `/dev/sda` (279.4G)
|
||||
- **Type**: System/OS disk with LVM
|
||||
- **Partitions**:
|
||||
- `sda1`: 1007K (BIOS boot partition)
|
||||
- `sda2`: 1G (EFI boot partition, mounted at `/boot/efi`)
|
||||
- `sda3`: 278G (LVM physical volume)
|
||||
- **LVM Volumes**:
|
||||
- `pve-swap`: 8G (swap)
|
||||
- `pve-root`: 79.5G (root filesystem, mounted at `/`)
|
||||
- `pve-data`: 171G (Proxmox data storage)
|
||||
- **Status**: ✅ Active system disk
|
||||
- **Usage**: Proxmox VE host OS and base storage
|
||||
|
||||
### Secondary Disk: `/dev/sdb` (279.4G)
|
||||
- **Type**: Standalone LVM physical volume (not in volume group)
|
||||
- **Partitions**: None (has LVM2_member signature)
|
||||
- **Status**: ✅ Available for use (needs LVM signature removal for Ceph)
|
||||
- **Model**: HUC106030CSS600
|
||||
- **Recommendations**:
|
||||
- Ceph OSD (wipe LVM signature first)
|
||||
- Proxmox storage pool
|
||||
- Backup storage
|
||||
|
||||
### Ceph Candidate Disks: `/dev/sdc` through `/dev/sdh` (6x 232.9G)
|
||||
- **Type**: SSDs - Ready for Ceph OSDs
|
||||
- **Model**: CT250MX500SSD1 (Crucial MX500)
|
||||
- **Size**: ~232.9G each
|
||||
- **Total Capacity**: ~1.4TB
|
||||
- **Status**: ✅ All 6 disks available (no partitions, unmounted)
|
||||
- **Typical Usage**: Ceph OSD nodes for distributed storage
|
||||
|
||||
## Disk Layout Summary
|
||||
|
||||
```
|
||||
sda (279.4G) - System Disk
|
||||
├─ sda1 (1007K) - BIOS boot
|
||||
├─ sda2 (1G) - EFI boot [/boot/efi]
|
||||
└─ sda3 (278G) - LVM PV
|
||||
├─ pve-swap (8G)
|
||||
├─ pve-root (79.5G) [/]
|
||||
└─ pve-data (171G)
|
||||
|
||||
sdb (279.4G) - Available
|
||||
|
||||
sdc-sdh (6x 232.9G) - Storage/Ceph candidates
|
||||
```
|
||||
|
||||
## Storage Recommendations
|
||||
|
||||
### For Ceph OSD Setup (Recommended)
|
||||
|
||||
**Quick Setup (Automated):**
|
||||
```bash
|
||||
# Run on r630-01 directly
|
||||
./scripts/setup-ceph-osds-r630.sh
|
||||
|
||||
# Or run remotely
|
||||
./scripts/run-ceph-osd-setup-remote.sh L@kers2010
|
||||
|
||||
# To include sdb disk as well
|
||||
./scripts/run-ceph-osd-setup-remote.sh L@kers2010 --include-sdb
|
||||
```
|
||||
|
||||
**Manual Setup:**
|
||||
1. **Verify disk status**: Run `analyze-r630-disk-layout.sh`
|
||||
2. **Wipe disks** (if needed): `wipefs -a /dev/sdX`
|
||||
3. **Create OSDs**: `ceph-volume lvm create --data /dev/sdX`
|
||||
4. **Verify**: `ceph osd tree`
|
||||
|
||||
**What the automated script does:**
|
||||
- Checks prerequisites (Ceph installed, cluster accessible)
|
||||
- Verifies all 6 SSDs (sdc-sdh) are available
|
||||
- Optionally prepares sdb (removes LVM signature)
|
||||
- Creates OSDs on all available disks
|
||||
- Enables and starts OSD services
|
||||
- Verifies Ceph health and OSD status
|
||||
|
||||
### For Proxmox Storage
|
||||
1. **Format disks**: Use `format-ssds-and-check-proxmox.sh`
|
||||
2. **Add to Proxmox**: Datacenter > Storage > Add
|
||||
3. **Choose type**: Directory, LVM, or ZFS
|
||||
|
||||
### For Mixed Use
|
||||
- Use `sdb` for Proxmox storage
|
||||
- Use `sdc-sdh` for Ceph OSDs
|
||||
- Keep `sda` as system disk
|
||||
|
||||
## Related Scripts
|
||||
|
||||
### Analysis & Verification
|
||||
- `scripts/analyze-r630-disk-layout.sh` - Comprehensive disk analysis
|
||||
- `scripts/check-proxmox-storage.sh` - Quick Proxmox storage check
|
||||
- `scripts/verify-r630-disks.sh` - Disk verification for Ceph
|
||||
|
||||
### Ceph OSD Setup
|
||||
- `scripts/setup-ceph-osds-r630.sh` - **Main setup script** (run on r630-01)
|
||||
- `scripts/run-ceph-osd-setup-remote.sh` - Remote execution helper
|
||||
- `scripts/create-ceph-osds-all-drives.sh` - Alternative OSD creation script
|
||||
- `scripts/create-all-osds-efficient.sh` - Efficient OSD creation
|
||||
|
||||
### Proxmox Storage
|
||||
- `scripts/format-ssds-and-check-proxmox.sh` - Format disks for Proxmox
|
||||
|
||||
## Notes
|
||||
|
||||
- All disk sizes are approximate (decimal vs binary)
|
||||
- Check actual status with `lsblk` or analysis script
|
||||
- Always verify before making changes to disk layout
|
||||
- Backup important data before repartitioning
|
||||
|
||||
151
docs/infrastructure/r630-01-fdisk-output.md
Normal file
151
docs/infrastructure/r630-01-fdisk-output.md
Normal file
@@ -0,0 +1,151 @@
|
||||
# R630-01 fdisk Output - Detailed Disk Information
|
||||
|
||||
**Date**: Generated from `fdisk -l` command
|
||||
**Host**: r630-01 (192.168.11.11)
|
||||
|
||||
## Complete fdisk Output
|
||||
|
||||
```
|
||||
Disk /dev/sda: 279.4 GiB, 300000000000 bytes, 585937500 sectors
|
||||
Disk model: ST9300653SS
|
||||
Units: sectors of 1 * 512 = 512 bytes
|
||||
Sector size (logical/physical): 512 bytes / 512 bytes
|
||||
I/O size (minimum/optimal): 512 bytes / 512 bytes
|
||||
Disklabel type: gpt
|
||||
Disk identifier: 3BB9B517-B595-4B69-9D31-468622AD51F9
|
||||
|
||||
Device Start End Sectors Size Type
|
||||
/dev/sda1 34 2047 2014 1007K BIOS boot
|
||||
/dev/sda2 2048 2099199 2097152 1G EFI System
|
||||
/dev/sda3 2099200 585105408 583006209 278G Linux LVM
|
||||
|
||||
|
||||
Disk /dev/sdb: 279.4 GiB, 300000000000 bytes, 585937500 sectors
|
||||
Disk model: HUC106030CSS600
|
||||
Units: sectors of 1 * 512 = 512 bytes
|
||||
Sector size (logical/physical): 512 bytes / 512 bytes
|
||||
I/O size (minimum/optimal): 512 bytes / 512 bytes
|
||||
|
||||
|
||||
Disk /dev/sde: 232.89 GiB, 250059350016 bytes, 488397168 sectors
|
||||
Disk model: CT250MX500SSD1
|
||||
Units: sectors of 1 * 512 = 512 bytes
|
||||
Sector size (logical/physical): 512 bytes / 4096 bytes
|
||||
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
|
||||
|
||||
|
||||
Disk /dev/sdc: 232.89 GiB, 250059350016 bytes, 488397168 sectors
|
||||
Disk model: CT250MX500SSD1
|
||||
Units: sectors of 1 * 512 = 512 bytes
|
||||
Sector size (logical/physical): 512 bytes / 4096 bytes
|
||||
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
|
||||
|
||||
|
||||
Disk /dev/sdd: 232.89 GiB, 250059350016 bytes, 488397168 sectors
|
||||
Disk model: CT250MX500SSD1
|
||||
Units: sectors of 1 * 512 = 512 bytes
|
||||
Sector size (logical/physical): 512 bytes / 4096 bytes
|
||||
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
|
||||
|
||||
|
||||
Disk /dev/sdh: 232.89 GiB, 250059350016 bytes, 488397168 sectors
|
||||
Disk model: CT250MX500SSD1
|
||||
Units: sectors of 1 * 512 = 512 bytes
|
||||
Sector size (logical/physical): 512 bytes / 4096 bytes
|
||||
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
|
||||
|
||||
|
||||
Disk /dev/sdg: 232.89 GiB, 250059350016 bytes, 488397168 sectors
|
||||
Disk model: CT250MX500SSD1
|
||||
Units: sectors of 1 * 512 = 512 bytes
|
||||
Sector size (logical/physical): 512 bytes / 4096 bytes
|
||||
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
|
||||
|
||||
|
||||
Disk /dev/sdf: 232.89 GiB, 250059350016 bytes, 488397168 sectors
|
||||
Disk model: CT250MX500SSD1
|
||||
Units: sectors of 1 * 512 = 512 bytes
|
||||
Sector size (logical/physical): 512 bytes / 4096 bytes
|
||||
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
|
||||
|
||||
|
||||
Disk /dev/mapper/pve-swap: 8 GiB, 8589934592 bytes, 16777216 sectors
|
||||
Units: sectors of 1 * 512 = 512 bytes
|
||||
Sector size (logical/physical): 512 bytes / 512 bytes
|
||||
I/O size (minimum/optimal): 512 bytes / 512 bytes
|
||||
|
||||
|
||||
Disk /dev/mapper/pve-root: 79.5 GiB, 85358280704 bytes, 166715392 sectors
|
||||
Units: sectors of 1 * 512 = 512 bytes
|
||||
Sector size (logical/physical): 512 bytes / 512 bytes
|
||||
I/O size (minimum/optimal): 512 bytes / 512 bytes
|
||||
```
|
||||
|
||||
## Disk Analysis
|
||||
|
||||
### System Disk: `/dev/sda` (279.4 GiB)
|
||||
- **Model**: ST9300653SS (Seagate 300GB SAS)
|
||||
- **Partition Table**: GPT
|
||||
- **Disk Identifier**: 3BB9B517-B595-4B69-9D31-468622AD51F9
|
||||
- **Sector Size**: 512 bytes (logical/physical)
|
||||
- **Partitions**:
|
||||
- `/dev/sda1`: 1007K (sectors 34-2047) - BIOS boot partition
|
||||
- `/dev/sda2`: 1G (sectors 2048-2099199) - EFI System partition
|
||||
- `/dev/sda3`: 278G (sectors 2099200-585105408) - Linux LVM
|
||||
- **LVM Volumes**:
|
||||
- `/dev/mapper/pve-swap`: 8 GiB
|
||||
- `/dev/mapper/pve-root`: 79.5 GiB
|
||||
|
||||
### Secondary Disk: `/dev/sdb` (279.4 GiB)
|
||||
- **Model**: HUC106030CSS600 (Hitachi/HGST 300GB SAS)
|
||||
- **Status**: No partition table (raw disk)
|
||||
- **Sector Size**: 512 bytes (logical/physical)
|
||||
- **Available**: Ready for use (may have LVM signature)
|
||||
|
||||
### SSD Storage Disks: `/dev/sdc` through `/dev/sdh` (6x 232.89 GiB)
|
||||
- **Model**: CT250MX500SSD1 (Crucial MX500 250GB SSD)
|
||||
- **Total Capacity**: ~1.4TB (6 x 232.89 GiB)
|
||||
- **Sector Size**: 512 bytes logical / 4096 bytes physical (4K native)
|
||||
- **I/O Size**: 4096 bytes (optimal for SSDs)
|
||||
- **Status**: All 6 disks are raw (no partition tables)
|
||||
- **Devices**:
|
||||
- `/dev/sdc`: 232.89 GiB
|
||||
- `/dev/sdd`: 232.89 GiB
|
||||
- `/dev/sde`: 232.89 GiB
|
||||
- `/dev/sdf`: 232.89 GiB
|
||||
- `/dev/sdg`: 232.89 GiB
|
||||
- `/dev/sdh`: 232.89 GiB
|
||||
|
||||
## Key Observations
|
||||
|
||||
1. **System Disk (`/dev/sda`)**:
|
||||
- Properly partitioned with GPT
|
||||
- Contains Proxmox VE installation with LVM
|
||||
- Has both BIOS boot and EFI System partitions (dual-boot capable)
|
||||
|
||||
2. **Secondary Disk (`/dev/sdb`)**:
|
||||
- No partition table detected
|
||||
- May contain LVM signature that needs to be wiped for Ceph use
|
||||
|
||||
3. **SSD Array (`/dev/sdc-sdh`)**:
|
||||
- All 6 SSDs are identical (Crucial MX500 250GB)
|
||||
- 4K native physical sector size (optimal for modern SSDs)
|
||||
- No partitions - ready for Ceph OSD creation
|
||||
- Total usable capacity: ~1.4TB
|
||||
|
||||
4. **LVM Volumes**:
|
||||
- `pve-swap`: 8GB swap space
|
||||
- `pve-root`: 79.5GB root filesystem
|
||||
- Additional space likely in `pve-data` volume (not shown in fdisk output)
|
||||
|
||||
## Recommendations
|
||||
|
||||
- **For Ceph**: Use all 6 SSDs (`sdc-sdh`) for OSDs
|
||||
- **For `/dev/sdb`**: Wipe LVM signature if present before using for Ceph
|
||||
- **System Disk**: Keep `/dev/sda` as-is for Proxmox VE OS
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [R630-01 Disk Layout Reference](./r630-01-disk-layout.md) - Overview and recommendations
|
||||
- [Ceph OSD Setup Documentation](../ceph/) - Ceph configuration guides
|
||||
|
||||
Reference in New Issue
Block a user