fbda1b4beb
Some checks failed
Deploy to Phoenix / deploy (push) Has been cancelled
- ADD_CHAIN138_TO_LEDGER_LIVE: Ledger form done; public code review repo bis-innovations/LedgerLive; init/push commands - CONTRACT_DEPLOYMENT_RUNBOOK: Chain 138 gas price 1 gwei, 36-addr check, TransactionMirror workaround - CONTRACT_*: AddressMapper, MirrorManager deployed 2026-02-12; 36-address on-chain check - NEXT_STEPS_FOR_YOU: Ledger done; steps completable now (no LAN); run-completable-tasks-from-anywhere - MASTER_INDEX, OPERATOR_OPTIONAL, SMART_CONTRACTS_INVENTORY_SIMPLE: updates - LEDGER_BLOCKCHAIN_INTEGRATION_COMPLETE: bis-innovations/LedgerLive reference Co-authored-by: Cursor <cursoragent@cursor.com>
244 lines
6.5 KiB
Markdown
244 lines
6.5 KiB
Markdown
# r630-02 Status Report
|
|
|
|
**Last Updated:** 2026-01-31
|
|
**Document Version:** 1.0
|
|
**Status:** Active Documentation
|
|
|
|
---
|
|
|
|
**Date:** 2026-01-15
|
|
**Host:** r630-02 (192.168.11.12)
|
|
**Status:** ✅ **OPERATIONAL** - Some Concerns
|
|
|
|
---
|
|
|
|
## System Information
|
|
|
|
### Host Details
|
|
- **Hostname:** r630-02
|
|
- **IP Address:** 192.168.11.12/24
|
|
- **OS:** Proxmox VE 9.1.4
|
|
- **Kernel:** 6.17.4-1-pve
|
|
- **Uptime:** 1 day, 6 hours, 40 minutes
|
|
- **Architecture:** x86_64
|
|
|
|
### Resource Usage
|
|
- **Memory:** 251GB total, 8.6GB used, 235GB free, 243GB available
|
|
- **CPU:** Intel Xeon E5-2660 v4 @ 2.00GHz (2 sockets, 28 cores, 56 threads)
|
|
- **CPU Usage:** 4.9% user, 5.5% system, 89.2% idle
|
|
- **Load Average:** ⚠️ **12.03, 11.65, 11.51** (High - concerning)
|
|
- **Swap:** 0B (no swap configured)
|
|
|
|
---
|
|
|
|
## Proxmox Status
|
|
|
|
### Services
|
|
- **pve-cluster:** ✅ Active (running) - 1 day 6h uptime
|
|
- **pvedaemon:** ✅ Active (running) - 1 day 6h uptime
|
|
- **pveproxy:** ✅ Active (running)
|
|
|
|
### Version
|
|
- **Proxmox VE:** 9.1.4/5ac30304265fbd8e
|
|
- **Kernel:** 6.17.4-1-pve
|
|
|
|
### VMs and Containers
|
|
- **Containers:** 4 total
|
|
- ✅ **5000** - blockscout-1 (running)
|
|
- ✅ **6200** - firefly-1 (running)
|
|
- ⏸️ **6201** - firefly-ali-1 (stopped)
|
|
- ✅ **7811** - mim-api-1 (running)
|
|
- **VMs:** 0
|
|
|
|
---
|
|
|
|
## Network Configuration
|
|
|
|
### Interfaces
|
|
- **Primary:** vmbr0 (bridge)
|
|
- IP: 192.168.11.12/24
|
|
- Gateway: 192.168.11.1
|
|
- Bridge Port: nic2
|
|
- **Physical:** nic2 (UP, connected to vmbr0)
|
|
- **Other NICs:** nic0, nic1, nic3 (DOWN)
|
|
|
|
### VLAN Configuration
|
|
- **Bridge:** vmbr0 (VLAN-aware)
|
|
- **Native VLAN:** 1 (untagged)
|
|
- **Container VLANs:** All on VLAN 1 (untagged)
|
|
- **Note:** ⚠️ Containers are not using VLAN tags - all on native VLAN
|
|
|
|
### Routing
|
|
- **Default Gateway:** 192.168.11.1 (VLAN 11)
|
|
- **Local Network:** 192.168.11.0/24
|
|
|
|
---
|
|
|
|
## Cluster Status
|
|
|
|
### Cluster Membership
|
|
- **Cluster Name:** h
|
|
- **Status:** ✅ **Quorate** (3/3 nodes)
|
|
- **Nodes:**
|
|
- Node 1: ml110 (192.168.11.10)
|
|
- Node 2: r630-01 (192.168.11.11)
|
|
- Node 3: r630-02 (192.168.11.12) - **Local**
|
|
|
|
### Quorum
|
|
- **Expected Votes:** 3
|
|
- **Total Votes:** 3
|
|
- **Quorum:** 2
|
|
- **Status:** ✅ Quorate
|
|
|
|
---
|
|
|
|
## Storage Status
|
|
|
|
### Storage Pools
|
|
|
|
| Name | Type | Status | Total | Used | Available | Usage % |
|
|
|------|------|--------|-------|------|-----------|---------|
|
|
| local | dir | ✅ Active | 220GB | 7.3GB | 212GB | 3.31% |
|
|
| thin1-r630-02 | lvmthin | ✅ Active | 226GB | 200GB | 26GB | ⚠️ **88.51%** |
|
|
| thin2 | lvmthin | ✅ Active | 226GB | 200GB | 26GB | ⚠️ **88.33%** |
|
|
| thin3 | lvmthin | ✅ Active | 226GB | 0GB | 226GB | 0.00% |
|
|
| thin4 | lvmthin | ✅ Active | 226GB | 29GB | 197GB | 12.69% |
|
|
| thin5 | lvmthin | ✅ Active | 226GB | 0GB | 226GB | 0.00% |
|
|
| thin6 | lvmthin | ✅ Active | 226GB | 0GB | 226GB | 0.00% |
|
|
|
|
**Concerns:**
|
|
- ⚠️ thin1-r630-02: 88.51% full (26GB free)
|
|
- ⚠️ thin2: 88.33% full (26GB free)
|
|
- ⚠️ Consider cleanup or expansion
|
|
|
|
---
|
|
|
|
## System Logs
|
|
|
|
### Recent Activity
|
|
- ✅ Cluster filesystem verification: Successful (hourly)
|
|
- ✅ Container operations: Normal (start, stop, backup)
|
|
- ✅ System updates: Regular apt updates
|
|
|
|
### Errors/Warnings
|
|
|
|
**⚠️ CRITICAL: Memory OOM (Out of Memory) Kills**
|
|
|
|
Multiple containers have been killed due to memory exhaustion:
|
|
|
|
| Date | Process | Container | Issue |
|
|
|------|---------|-----------|-------|
|
|
| Jan 14 01:39 | npm exec func s | UID:100000 | OOM killed (708MB) |
|
|
| Jan 14 07:42 | systemd-journal | UID:100000 | OOM killed (39MB) |
|
|
| Jan 14 07:42 | npm exec func s | UID:100000 | OOM killed (633MB) |
|
|
| Jan 14 09:37 | apt-get | UID:100000 | OOM killed (88MB) |
|
|
| Jan 14 11:10 | node | UID:100000 | OOM killed (331MB) |
|
|
| Jan 14 13:01 | python3 | UID:100000 | OOM killed (38MB) |
|
|
| Jan 14 16:06 | npm exec func s | UID:100000 | OOM killed (633MB) |
|
|
| Jan 14 16:40 | systemd-journal | UID:100000 | OOM killed (31MB) |
|
|
| Jan 14 16:48 | networkd-dispat | UID:100000 | OOM killed (29MB) |
|
|
|
|
**Analysis:**
|
|
- Multiple containers experiencing OOM kills
|
|
- Pattern suggests memory limits may be too restrictive
|
|
- Some containers (npm/node processes) consuming significant memory
|
|
- Systemd-journal and networkd-dispat also being killed
|
|
|
|
**Recommendation:**
|
|
- Review container memory limits
|
|
- Increase memory allocation for affected containers
|
|
- Monitor memory usage patterns
|
|
- Consider adding swap (currently 0B)
|
|
|
|
**Other Issues:**
|
|
- ⚠️ Subscription check failed: DNS resolution issue (non-critical)
|
|
|
|
---
|
|
|
|
## Firewall Status
|
|
|
|
### Proxmox Firewall
|
|
- **Status:** ✅ Enabled/Running
|
|
- **Configuration:** Active
|
|
|
|
### Firewall Rules
|
|
- ✅ **Rule:** Allow Default Network (192.168.0.0/24)
|
|
- Direction: IN
|
|
- Action: ACCEPT
|
|
- Source: 192.168.0.0/24
|
|
- Logging: Disabled
|
|
|
|
**Note:** Firewall is properly configured to allow access from Default network.
|
|
|
|
---
|
|
|
|
## Summary
|
|
|
|
### ✅ Operational Status
|
|
|
|
**System:** ✅ **OPERATIONAL**
|
|
- Proxmox services running
|
|
- Cluster quorate (3/3 nodes)
|
|
- Network configured correctly
|
|
- Firewall enabled and configured
|
|
|
|
### ⚠️ Concerns
|
|
|
|
1. **High Load Average:**
|
|
- Load: 12.03, 11.65, 11.51
|
|
- With 56 CPU threads, this is moderate but worth monitoring
|
|
- May indicate CPU-intensive workloads
|
|
|
|
2. **Memory OOM Kills:**
|
|
- Multiple containers killed due to memory exhaustion
|
|
- Pattern suggests memory limits may be too restrictive
|
|
- Need to review and adjust container memory allocations
|
|
|
|
3. **Storage Usage:**
|
|
- thin1-r630-02: 88.51% full (26GB free)
|
|
- thin2: 88.33% full (26GB free)
|
|
- Consider cleanup or expansion
|
|
|
|
4. **Container VLAN Assignment:**
|
|
- All containers on native VLAN (untagged)
|
|
- Not utilizing VLAN plan yet
|
|
- Ready for VLAN migration
|
|
|
|
### 📋 Recommendations
|
|
|
|
1. **Immediate:**
|
|
- Review container memory limits
|
|
- Increase memory for containers experiencing OOM kills
|
|
- Monitor load average trends
|
|
|
|
2. **Short-term:**
|
|
- Clean up or expand thin1-r630-02 and thin2 storage pools
|
|
- Migrate containers to appropriate VLANs
|
|
- Add swap space (currently 0B)
|
|
|
|
3. **Long-term:**
|
|
- Optimize container resource allocation
|
|
- Plan storage expansion
|
|
- Complete VLAN migration
|
|
|
|
### 🎯 Overall Assessment
|
|
|
|
**Status:** ✅ **OPERATIONAL WITH CONCERNS**
|
|
|
|
The system is functional and stable, but requires attention to:
|
|
- Memory management (OOM kills)
|
|
- Storage capacity (thin pools nearly full)
|
|
- Load monitoring (high but manageable)
|
|
|
|
**Ready for:**
|
|
- ✅ Normal operations
|
|
- ✅ Container management
|
|
- ✅ VLAN migration (when ready)
|
|
- ⚠️ Memory optimization needed
|
|
|
|
---
|
|
|
|
**Last Updated:** 2026-01-15
|
|
**Last Reviewed:** 2026-01-15
|
|
**Actions Taken:** Memory limits adjusted, status re-checked
|