loc_az_hci/docs/temporary/DEPLOYMENT_WITHOUT_AZURE.md

Deployment Guide - Without Azure Arc

This guide covers deploying the complete infrastructure stack without Azure Arc integration. Azure Arc can be added later once subscription issues are resolved.

✅ What Works Without Azure

• ✅ Proxmox VE cluster and VM management
• ✅ Cloudflare Tunnel for secure external access
• ✅ Kubernetes (K3s) cluster deployment
• ✅ GitOps with self-hosted Git server
• ✅ All HC Stack services (Besu, Firefly, Chainlink, etc.)
• ✅ Monitoring and observability stack
• ✅ Network configuration and VLANs
• ✅ Storage management

⏸️ What's Deferred (Until Azure Available)

• ⏸️ Azure Arc onboarding
• ⏸️ Azure Policy enforcement
• ⏸️ Azure Monitor integration
• ⏸️ Azure Defender
• ⏸️ Azure Update Management

🚀 Deployment Phases (Without Azure)

Phase 1: Proxmox Cluster Setup

Verify/Configure Cluster:

# On ML110 (192.168.1.206)
ssh root@192.168.1.206
pvecm status
pvecm nodes

# On R630 (192.168.1.49)
ssh root@192.168.1.49
pvecm status
pvecm nodes

If not clustered, create cluster:

# On ML110 (first node)
pvecm create hc-cluster

# On R630 (join cluster)
pvecm add 192.168.1.206

Phase 2: Create Service VMs

Option A: Using Terraform

cd terraform/proxmox

# Create terraform.tfvars from .env
source <(grep -v '^#' ../.env | grep -v '^$' | sed 's/#.*$//' | sed 's/^[[:space:]]*//;s/[[:space:]]*$//' | grep '=')

cat > terraform.tfvars <<EOF
proxmox_host     = "192.168.1.206"
proxmox_username = "root@pam"
proxmox_password = "$PVE_ROOT_PASS"
proxmox_node     = "pve"
EOF

terraform init
terraform plan
terraform apply

Option B: Manual VM Creation via Proxmox UI

Access Proxmox web UI:

• ML110: https://192.168.1.206:8006
• R630: https://192.168.1.49:8006

Create VMs for:

1. K3s Master (Kubernetes)

   • 4 vCPU, 8GB RAM, 80GB disk
   • Ubuntu 22.04 LTS
   • IP: 192.168.1.188 (adjust as needed)

2. Cloudflare Tunnel VM

   • 2 vCPU, 4GB RAM, 40GB disk
   • Ubuntu 22.04 LTS
   • IP: 192.168.1.60 (VLAN 99/DMZ)

3. Git Server (Gitea/GitLab)

   • 4 vCPU, 8GB RAM, 100GB disk
   • Ubuntu 22.04 LTS
   • IP: 192.168.1.121

4. Observability VM (Prometheus/Grafana)

   • 4 vCPU, 8GB RAM, 200GB disk
   • Ubuntu 22.04 LTS
   • IP: 192.168.1.82

Phase 3: Cloudflare Tunnel Configuration

On Cloudflare Tunnel VM:

# SSH to tunnel VM
ssh ubuntu@192.168.1.60

# Install cloudflared
curl -L https://github.com/cloudflare/cloudflared/releases/latest/download/cloudflared-linux-amd64 -o /usr/local/bin/cloudflared
chmod +x /usr/local/bin/cloudflared

# Configure tunnel using tunnel token from .env
# Load environment variables
source <(grep -v '^#' /path/to/.env | grep -v '^$' | sed 's/#.*$//' | sed 's/^[[:space:]]*//;s/[[:space:]]*$//' | grep '=')

# Create tunnel config directory
sudo mkdir -p /etc/cloudflared

# Create config.yml using tunnel token
cat > /tmp/config.yml <<EOF
tunnel: $(echo $CLOUDFLARE_TUNNEL_TOKEN | cut -d'_' -f1)
credentials-file: /etc/cloudflared/credentials.json

ingress:
  # Proxmox UI - ML110
  - hostname: proxmox-ml110.d-bis.org
    service: https://192.168.1.206:8006
    originRequest:
      noHappyEyeballs: true
      tcpKeepAlive: 30

  # Proxmox UI - R630
  - hostname: proxmox-r630.d-bis.org
    service: https://192.168.1.49:8006
    originRequest:
      noHappyEyeballs: true
      tcpKeepAlive: 30

  # Kubernetes Dashboard (will add after K3s deployment)
  - hostname: k8s.d-bis.org
    service: http://192.168.1.188:8000
    originRequest:
      noHappyEyeballs: true

  # Git Server (will add after Git deployment)
  - hostname: git.d-bis.org
    service: http://192.168.1.121:3000
    originRequest:
      noHappyEyeballs: true

  # Grafana (will add after observability deployment)
  - hostname: grafana.d-bis.org
    service: http://192.168.1.82:3000
    originRequest:
      noHappyEyeballs: true

  # Catch-all
  - service: http_status:404
EOF

sudo mv /tmp/config.yml /etc/cloudflared/config.yml

# Create credentials file from tunnel token
# Note: Tunnel token format may vary, adjust as needed
echo "{\"AccountTag\":\"$CLOUDFLARE_ACCOUNT_ID\",\"TunnelSecret\":\"$CLOUDFLARE_TUNNEL_TOKEN\"}" | sudo tee /etc/cloudflared/credentials.json
sudo chmod 600 /etc/cloudflared/credentials.json

# Create systemd service
sudo tee /etc/systemd/system/cloudflared.service > /dev/null <<EOF
[Unit]
Description=Cloudflare Tunnel
After=network.target

[Service]
Type=simple
User=cloudflared
ExecStart=/usr/local/bin/cloudflared tunnel --config /etc/cloudflared/config.yml run
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target
EOF

# Create cloudflared user
sudo useradd -r -s /usr/sbin/nologin cloudflared
sudo chown -R cloudflared:cloudflared /etc/cloudflared

# Enable and start
sudo systemctl enable cloudflared
sudo systemctl start cloudflared
sudo systemctl status cloudflared

Phase 4: Kubernetes (K3s) Deployment

On K3s VM:

# SSH to K3s VM
ssh ubuntu@192.168.1.188

# Install K3s
curl -sfL https://get.k3s.io | sh -

# Verify installation
sudo k3s kubectl get nodes

# Get kubeconfig
sudo cat /etc/rancher/k3s/k3s.yaml

# Copy kubeconfig to local machine
mkdir -p ~/.kube
sudo cat /etc/rancher/k3s/k3s.yaml | sed 's/127.0.0.1/192.168.1.188/g' > ~/.kube/config
chmod 600 ~/.kube/config

# Test access
kubectl get nodes

Deploy Base Infrastructure:

# Create namespaces
kubectl create namespace blockchain
kubectl create namespace monitoring
kubectl create namespace hc-stack

# Deploy NGINX Ingress Controller
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/cloud/deploy.yaml

# Deploy Cert-Manager (optional, for TLS)
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.13.0/cert-manager.yaml

Phase 5: Git Server Deployment

On Git Server VM:

# SSH to Git server VM
ssh ubuntu@192.168.1.121

# Option A: Deploy Gitea (Recommended)
docker run -d --name=gitea \
  -p 3000:3000 \
  -p 2222:22 \
  -v gitea_data:/data \
  -e USER_UID=1000 \
  -e USER_GID=1000 \
  gitea/gitea:latest

# Access Gitea at http://192.168.1.121:3000
# Complete initial setup
# Create repository for GitOps

Or use deployment script:

cd /path/to/loc_az_hci
./infrastructure/gitops/gitea-deploy.sh

Phase 6: Observability Stack

On Observability VM or Kubernetes:

Option A: Deploy in Kubernetes (Recommended)

# Deploy Prometheus
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install prometheus prometheus-community/kube-prometheus-stack -n monitoring

# Deploy Grafana (if not included in kube-prometheus-stack)
helm repo add grafana https://grafana.github.io/helm-charts
helm install grafana grafana/grafana -n monitoring

# Get Grafana admin password
kubectl get secret --namespace monitoring grafana -o jsonpath="{.data.admin-password}" | base64 --decode

Option B: Deploy on VM

# On observability VM
ssh ubuntu@192.168.1.82

# Install Prometheus
# Install Grafana
# Configure data sources

Phase 7: Deploy HC Stack Services

Deploy Blockchain Services:

# Ensure you're in project directory
cd /path/to/loc_az_hci

# Deploy Besu
helm install besu ./gitops/apps/besu -n blockchain

# Deploy Firefly
helm install firefly ./gitops/apps/firefly -n blockchain

# Deploy Chainlink
helm install chainlink-ccip ./gitops/apps/chainlink-ccip -n blockchain

# Deploy Blockscout
helm install blockscout ./gitops/apps/blockscout -n blockchain

# Deploy Cacti (monitoring)
helm install cacti ./gitops/apps/cacti -n monitoring

# Deploy NGINX Proxy
helm install nginx-proxy ./gitops/apps/nginx-proxy -n hc-stack

Phase 8: Configure Ingress

Update Cloudflare Tunnel config with service endpoints:

# On Cloudflare Tunnel VM
sudo nano /etc/cloudflared/config.yml

# Add ingress rules for:
# - besu.d-bis.org → Kubernetes service
# - firefly.d-bis.org → Kubernetes service
# - blockscout.d-bis.org → Kubernetes service
# - grafana.d-bis.org → Grafana service

# Restart tunnel
sudo systemctl restart cloudflared

Create Kubernetes Ingress resources:

# Create ingress for services
kubectl apply -f - <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: blockchain-ingress
  namespace: blockchain
  annotations:
    kubernetes.io/ingress.class: nginx
spec:
  rules:
  - host: besu.d-bis.org
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: besu
            port:
              number: 8545
EOF

Phase 9: Network Configuration

Configure VLANs on Proxmox (if not done):

# On each Proxmox host
# Configure VLAN bridges
# See: infrastructure/proxmox/configure-proxmox-vlans.sh

Configure OpenWrt (if router server exists):

# Configure VLANs, routing, firewall
# See: infrastructure/network/configure-openwrt-network.ps1

Phase 10: Monitoring Setup

Configure Prometheus Targets:

# Add Proxmox exporters
# Add node exporters
# Configure scrape configs

Create Grafana Dashboards:

# Import dashboards for:
# - Kubernetes cluster
# - Proxmox hosts
# - Network metrics
# - Application metrics

🔧 Useful Commands (Without Azure)

Proxmox Operations:

# List VMs
pvesh get /nodes/pve/qemu

# Create VM via API
# Use Terraform or Proxmox UI

Kubernetes Operations:

# Check cluster status
kubectl get nodes
kubectl get pods --all-namespaces
kubectl get services --all-namespaces

# Check ingress
kubectl get ingress --all-namespaces

Cloudflare Tunnel:

# Check tunnel status
sudo systemctl status cloudflared
sudo journalctl -u cloudflared -f

# Test tunnel connectivity
cloudflared tunnel info

📋 Deployment Checklist (Without Azure)

• Verify Proxmox cluster status
• Create service VMs (K3s, Tunnel, Git, Observability)
• Configure Cloudflare Tunnel
• Deploy Kubernetes (K3s)
• Deploy Git server (Gitea/GitLab)
• Deploy observability stack
• Deploy HC Stack services
• Configure ingress and routing
• Set up monitoring dashboards
• Test all services
• Configure backups
• Document IPs and configurations

🎯 Adding Azure Arc Later

Once Azure subscription is enabled:

1. Install Arc agents on Proxmox hosts:

   # On each Proxmox host
   ./scripts/azure-arc/onboard-proxmox-hosts.sh
   

2. Install Arc agents on VMs:

   # On each VM
   ./scripts/azure-arc/onboard-vms.sh
   

3. Onboard Kubernetes to Arc:

   # On K3s VM
   ./infrastructure/kubernetes/arc-onboard-k8s.sh
   

🚨 Important Notes

• No Azure dependencies: All services work independently
• Local monitoring: Use Prometheus/Grafana instead of Azure Monitor
• Local GitOps: Use self-hosted Git server instead of Azure DevOps
• Cloudflare for access: Use Cloudflare Tunnel instead of Azure VPN
• Manual updates: Update management without Azure Update Management

📚 Reference Documentation

• Deployment Guide - General deployment steps
• Cloudflare Integration - Tunnel setup
• Proxmox Operations - Proxmox management
• Network Topology - Network configuration