Initial commit: loc_az_hci (smom-dbis-138 excluded via .gitignore)
Some checks failed
Test / test (push) Has been cancelled
Some checks failed
Test / test (push) Has been cancelled
Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
defiQUG
485
docs/deployment/deployment-guide.md
Normal file
485
docs/deployment/deployment-guide.md
Normal file
@@ -0,0 +1,485 @@
|
||||
# Deployment Guide
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Before starting the deployment, ensure you have:
|
||||
|
||||
1. **Two Proxmox VE hosts** with:
|
||||
- Proxmox VE 7.0+ installed
|
||||
- Static IP addresses configured
|
||||
- At least 8GB RAM per node
|
||||
- Network connectivity between nodes
|
||||
- Root or sudo access
|
||||
|
||||
2. **Azure Subscription** with:
|
||||
- Azure CLI installed and authenticated
|
||||
- Contributor role on subscription
|
||||
- Resource group creation permissions
|
||||
|
||||
3. **Network Requirements**:
|
||||
- Static IP addresses for all nodes
|
||||
- DNS resolution (or hosts file)
|
||||
- Internet access for Azure Arc connectivity
|
||||
- NFS server (optional, for shared storage)
|
||||
|
||||
4. **Tools Installed**:
|
||||
- SSH client
|
||||
- kubectl
|
||||
- helm (optional)
|
||||
- terraform (optional)
|
||||
|
||||
5. **Environment Configuration**:
|
||||
- Copy `.env.example` to `.env` and fill in all credentials
|
||||
- See [Configuration](#configuration) section for details
|
||||
|
||||
## Configuration
|
||||
|
||||
### Environment Variables Setup
|
||||
|
||||
Before starting deployment, configure your environment variables:
|
||||
|
||||
1. **Copy the template:**
|
||||
```bash
|
||||
cp .env.example .env
|
||||
```
|
||||
|
||||
2. **Edit `.env` with your credentials:**
|
||||
- Azure credentials: `AZURE_SUBSCRIPTION_ID`, `AZURE_TENANT_ID`
|
||||
- Cloudflare: `CLOUDFLARE_API_TOKEN`
|
||||
- Proxmox: `PVE_ROOT_PASS` (shared root password for all instances)
|
||||
- Proxmox ML110: `PROXMOX_ML110_URL`
|
||||
- Proxmox R630: `PROXMOX_R630_URL`
|
||||
|
||||
**Note**: The username `root@pam` is implied and should not be stored. For production operations, use RBAC accounts and API tokens instead of root credentials.
|
||||
|
||||
3. **Load environment variables:**
|
||||
```bash
|
||||
# Source the .env file
|
||||
export $(cat .env | grep -v '^#' | xargs)
|
||||
```
|
||||
|
||||
**Note**: All scripts in this guide will use environment variables from `.env` if available. You can also set them manually using `export` commands.
|
||||
|
||||
## Deployment Phases
|
||||
|
||||
### Phase 1: Proxmox Cluster Setup
|
||||
|
||||
#### Step 1.1: Configure Network on Both Nodes
|
||||
|
||||
On each Proxmox node:
|
||||
|
||||
```bash
|
||||
# Option 1: Use .env file (recommended)
|
||||
# Load environment variables from .env
|
||||
export $(cat .env | grep -v '^#' | xargs)
|
||||
|
||||
# Option 2: Set environment variables manually
|
||||
export NODE_IP=192.168.1.10 # Use appropriate IP for each node
|
||||
export NODE_GATEWAY=192.168.1.1
|
||||
export NODE_NETMASK=24
|
||||
export NODE_HOSTNAME=pve-node-1 # Use appropriate hostname
|
||||
|
||||
# Run network configuration script
|
||||
cd /path/to/loc_az_hci
|
||||
./infrastructure/proxmox/network-config.sh
|
||||
```
|
||||
|
||||
**For Node 2**, repeat with appropriate values:
|
||||
```bash
|
||||
export NODE_IP=192.168.1.11
|
||||
export NODE_HOSTNAME=pve-node-2
|
||||
./infrastructure/proxmox/network-config.sh
|
||||
```
|
||||
|
||||
#### Step 1.2: Update Proxmox Repositories
|
||||
|
||||
On both nodes:
|
||||
|
||||
```bash
|
||||
# Update to subscription-free repos
|
||||
sed -i 's/enterprise/no-subscription/g' /etc/apt/sources.list.d/pve-enterprise.list
|
||||
apt update && apt dist-upgrade -y
|
||||
```
|
||||
|
||||
#### Step 1.3: Configure Shared Storage (NFS)
|
||||
|
||||
**Option A: Using existing NFS server**
|
||||
|
||||
On both Proxmox nodes:
|
||||
|
||||
```bash
|
||||
export NFS_SERVER=192.168.1.100
|
||||
export NFS_PATH=/mnt/proxmox-storage
|
||||
export STORAGE_NAME=nfs-shared
|
||||
|
||||
./infrastructure/proxmox/nfs-storage.sh
|
||||
```
|
||||
|
||||
**Option B: Set up NFS server**
|
||||
|
||||
If you need to set up an NFS server, install and configure it on a separate machine or VM.
|
||||
|
||||
#### Step 1.4: Create Proxmox Cluster
|
||||
|
||||
**On Node 1** (cluster creator):
|
||||
|
||||
```bash
|
||||
export NODE_ROLE=create
|
||||
export CLUSTER_NAME=hc-cluster
|
||||
|
||||
./infrastructure/proxmox/cluster-setup.sh
|
||||
```
|
||||
|
||||
**On Node 2** (join cluster):
|
||||
|
||||
```bash
|
||||
export NODE_ROLE=join
|
||||
export CLUSTER_NODE_IP=192.168.1.10 # IP of Node 1
|
||||
export ROOT_PASSWORD=your-root-password # Optional, will prompt if not set
|
||||
|
||||
./infrastructure/proxmox/cluster-setup.sh
|
||||
```
|
||||
|
||||
**Verify cluster**:
|
||||
|
||||
```bash
|
||||
pvecm status
|
||||
pvecm nodes
|
||||
```
|
||||
|
||||
### Phase 2: Azure Arc Integration
|
||||
|
||||
#### Step 2.1: Prepare Azure Environment
|
||||
|
||||
```bash
|
||||
# Load environment variables from .env (if using .env file)
|
||||
export $(cat .env | grep -v '^#' | xargs)
|
||||
|
||||
# Login to Azure
|
||||
az login
|
||||
|
||||
# Set subscription (use from .env or set manually)
|
||||
az account set --subscription "${AZURE_SUBSCRIPTION_ID:-your-subscription-id}"
|
||||
|
||||
# Create resource group (if not exists)
|
||||
az group create --name "${AZURE_RESOURCE_GROUP:-HC-Stack}" --location "${AZURE_LOCATION:-eastus}"
|
||||
```
|
||||
|
||||
#### Step 2.2: Onboard Proxmox Hosts to Azure Arc
|
||||
|
||||
On each Proxmox node:
|
||||
|
||||
```bash
|
||||
# Load environment variables from .env (if using .env file)
|
||||
export $(cat .env | grep -v '^#' | xargs)
|
||||
|
||||
# Set Azure variables (use from .env or get from Azure CLI)
|
||||
export RESOURCE_GROUP="${AZURE_RESOURCE_GROUP:-HC-Stack}"
|
||||
export TENANT_ID="${AZURE_TENANT_ID:-$(az account show --query tenantId -o tsv)}"
|
||||
export SUBSCRIPTION_ID="${AZURE_SUBSCRIPTION_ID:-$(az account show --query id -o tsv)}"
|
||||
export LOCATION="${AZURE_LOCATION:-eastus}"
|
||||
export TAGS="type=proxmox,environment=hybrid"
|
||||
|
||||
./scripts/azure-arc/onboard-proxmox-hosts.sh
|
||||
```
|
||||
|
||||
**Verify in Azure Portal**:
|
||||
- Navigate to: Azure Portal → Azure Arc → Servers
|
||||
- You should see both Proxmox nodes
|
||||
|
||||
#### Step 2.3: Create VMs for Kubernetes and Git
|
||||
|
||||
Create VMs in Proxmox web UI or using Terraform:
|
||||
|
||||
```bash
|
||||
# Load environment variables from .env
|
||||
export $(cat .env | grep -v '^#' | xargs)
|
||||
|
||||
cd terraform/proxmox
|
||||
# Create terraform.tfvars from environment variables or edit manually
|
||||
cat > terraform.tfvars <<EOF
|
||||
proxmox_host = "${PROXMOX_ML110_URL#https://}"
|
||||
proxmox_username = "root@pam" # Hardcoded, not from env (best practice)
|
||||
proxmox_password = "${PVE_ROOT_PASS}"
|
||||
proxmox_node = "pve-node-1"
|
||||
EOF
|
||||
|
||||
terraform init
|
||||
terraform plan
|
||||
terraform apply
|
||||
```
|
||||
|
||||
#### Step 2.4: Onboard VMs to Azure Arc
|
||||
|
||||
For each VM:
|
||||
|
||||
```bash
|
||||
# Load environment variables from .env
|
||||
export $(cat .env | grep -v '^#' | xargs)
|
||||
|
||||
export VM_IP=192.168.1.188
|
||||
export VM_USER=ubuntu
|
||||
export RESOURCE_GROUP="${AZURE_RESOURCE_GROUP:-HC-Stack}"
|
||||
export TENANT_ID="${AZURE_TENANT_ID:-$(az account show --query tenantId -o tsv)}"
|
||||
export SUBSCRIPTION_ID="${AZURE_SUBSCRIPTION_ID:-$(az account show --query id -o tsv)}"
|
||||
export LOCATION="${AZURE_LOCATION:-eastus}"
|
||||
|
||||
./scripts/azure-arc/onboard-vms.sh
|
||||
```
|
||||
|
||||
### Phase 3: Kubernetes Setup
|
||||
|
||||
#### Step 3.1: Install K3s
|
||||
|
||||
On the VM designated for Kubernetes:
|
||||
|
||||
```bash
|
||||
export INSTALL_MODE=local
|
||||
export K3S_VERSION=latest
|
||||
|
||||
./infrastructure/kubernetes/k3s-install.sh
|
||||
```
|
||||
|
||||
**Or install remotely**:
|
||||
|
||||
```bash
|
||||
export INSTALL_MODE=remote
|
||||
export REMOTE_IP=192.168.1.188
|
||||
export REMOTE_USER=ubuntu
|
||||
|
||||
./infrastructure/kubernetes/k3s-install.sh
|
||||
```
|
||||
|
||||
#### Step 3.2: Onboard Kubernetes to Azure Arc
|
||||
|
||||
```bash
|
||||
# Load environment variables from .env
|
||||
export $(cat .env | grep -v '^#' | xargs)
|
||||
|
||||
export RESOURCE_GROUP="${AZURE_RESOURCE_GROUP:-HC-Stack}"
|
||||
export TENANT_ID="${AZURE_TENANT_ID:-$(az account show --query tenantId -o tsv)}"
|
||||
export SUBSCRIPTION_ID="${AZURE_SUBSCRIPTION_ID:-$(az account show --query id -o tsv)}"
|
||||
export LOCATION="${AZURE_LOCATION:-eastus}"
|
||||
export CLUSTER_NAME=proxmox-k3s-cluster
|
||||
|
||||
# Ensure kubeconfig is set
|
||||
export KUBECONFIG=~/.kube/config
|
||||
|
||||
./infrastructure/kubernetes/arc-onboard-k8s.sh
|
||||
```
|
||||
|
||||
**Verify in Azure Portal**:
|
||||
- Navigate to: Azure Portal → Azure Arc → Kubernetes
|
||||
- You should see your cluster
|
||||
|
||||
#### Step 3.3: Install Base Infrastructure
|
||||
|
||||
```bash
|
||||
# Apply namespace and base infrastructure
|
||||
kubectl apply -f gitops/infrastructure/namespace.yaml
|
||||
kubectl apply -f gitops/infrastructure/ingress-controller.yaml
|
||||
kubectl apply -f gitops/infrastructure/cert-manager.yaml
|
||||
```
|
||||
|
||||
### Phase 4: Git/DevOps Setup
|
||||
|
||||
#### Option A: Deploy Gitea (Recommended for small deployments)
|
||||
|
||||
```bash
|
||||
export GITEA_DOMAIN=git.local
|
||||
export GITEA_PORT=3000
|
||||
|
||||
./infrastructure/gitops/gitea-deploy.sh
|
||||
```
|
||||
|
||||
Access Gitea at `http://git.local:3000` and complete initial setup.
|
||||
|
||||
#### Option B: Deploy GitLab CE
|
||||
|
||||
```bash
|
||||
export GITLAB_DOMAIN=gitlab.local
|
||||
export GITLAB_PORT=8080
|
||||
|
||||
./infrastructure/gitops/gitlab-deploy.sh
|
||||
```
|
||||
|
||||
**Note**: GitLab requires at least 8GB RAM.
|
||||
|
||||
#### Option C: Azure DevOps Self-Hosted Agent
|
||||
|
||||
On a VM:
|
||||
|
||||
```bash
|
||||
# Load environment variables from .env
|
||||
export $(cat .env | grep -v '^#' | xargs)
|
||||
|
||||
export AZP_URL="${AZP_URL:-https://dev.azure.com/yourorg}"
|
||||
export AZP_TOKEN="${AZP_TOKEN:-your-personal-access-token}"
|
||||
export AZP_AGENT_NAME=proxmox-agent-1
|
||||
export AZP_POOL=Default
|
||||
|
||||
./infrastructure/gitops/azure-devops-agent.sh
|
||||
```
|
||||
|
||||
### Phase 5: Configure GitOps
|
||||
|
||||
#### Step 5.1: Create Git Repository
|
||||
|
||||
1. Create a new repository in your Git server (Gitea/GitLab)
|
||||
2. Clone the repository locally
|
||||
3. Copy the `gitops/` directory to your repository
|
||||
4. Commit and push:
|
||||
|
||||
```bash
|
||||
git clone http://git.local:3000/user/gitops-repo.git
|
||||
cd gitops-repo
|
||||
cp -r /path/to/loc_az_hci/gitops/* .
|
||||
git add .
|
||||
git commit -m "Initial GitOps configuration"
|
||||
git push
|
||||
```
|
||||
|
||||
#### Step 5.2: Connect GitOps to Azure Arc
|
||||
|
||||
In Azure Portal:
|
||||
|
||||
1. Navigate to: Azure Arc → Kubernetes → Your cluster
|
||||
2. Go to "GitOps" section
|
||||
3. Click "Add configuration"
|
||||
4. Configure:
|
||||
- Repository URL: `http://git.local:3000/user/gitops-repo.git`
|
||||
- Branch: `main`
|
||||
- Path: `gitops/`
|
||||
- Authentication: Configure as needed
|
||||
|
||||
### Phase 6: Deploy HC Stack Services
|
||||
|
||||
#### Option A: Deploy via GitOps (Recommended)
|
||||
|
||||
1. Update Helm chart values in your Git repository
|
||||
2. Commit and push changes
|
||||
3. Flux will automatically deploy updates
|
||||
|
||||
#### Option B: Deploy Manually with Helm
|
||||
|
||||
```bash
|
||||
# Add Helm charts
|
||||
helm install besu ./gitops/apps/besu -n blockchain
|
||||
helm install firefly ./gitops/apps/firefly -n blockchain
|
||||
helm install chainlink-ccip ./gitops/apps/chainlink-ccip -n blockchain
|
||||
helm install blockscout ./gitops/apps/blockscout -n blockchain
|
||||
helm install cacti ./gitops/apps/cacti -n monitoring
|
||||
helm install nginx-proxy ./gitops/apps/nginx-proxy -n hc-stack
|
||||
```
|
||||
|
||||
#### Option C: Deploy with Terraform
|
||||
|
||||
```bash
|
||||
cd terraform/kubernetes
|
||||
terraform init
|
||||
terraform plan
|
||||
terraform apply
|
||||
```
|
||||
|
||||
### Phase 7: Verify Deployment
|
||||
|
||||
#### Check Proxmox Cluster
|
||||
|
||||
```bash
|
||||
pvecm status
|
||||
pvesm status
|
||||
```
|
||||
|
||||
#### Check Azure Arc
|
||||
|
||||
```bash
|
||||
# List Arc-enabled servers
|
||||
az connectedmachine list --resource-group HC-Stack -o table
|
||||
|
||||
# List Arc-enabled Kubernetes clusters
|
||||
az arc kubernetes list --resource-group HC-Stack -o table
|
||||
```
|
||||
|
||||
#### Check Kubernetes
|
||||
|
||||
```bash
|
||||
kubectl get nodes
|
||||
kubectl get pods --all-namespaces
|
||||
kubectl get services --all-namespaces
|
||||
```
|
||||
|
||||
#### Check Applications
|
||||
|
||||
```bash
|
||||
# Check Besu
|
||||
kubectl get pods -n blockchain -l app=besu
|
||||
|
||||
# Check Firefly
|
||||
kubectl get pods -n blockchain -l app=firefly
|
||||
|
||||
# Check all services
|
||||
kubectl get all --all-namespaces
|
||||
```
|
||||
|
||||
## Post-Deployment Configuration
|
||||
|
||||
### 1. Configure Ingress
|
||||
|
||||
Update ingress configurations for external access:
|
||||
|
||||
```bash
|
||||
# Edit ingress resources
|
||||
kubectl edit ingress -n blockchain
|
||||
```
|
||||
|
||||
### 2. Set Up Monitoring
|
||||
|
||||
- Configure Cacti to monitor your infrastructure
|
||||
- Set up Azure Monitor alerts
|
||||
- Configure log aggregation
|
||||
|
||||
### 3. Configure Backup
|
||||
|
||||
- Set up Proxmox backup schedules
|
||||
- Configure Kubernetes backup (Velero)
|
||||
- Set up Azure Backup for Arc resources
|
||||
|
||||
### 4. Security Hardening
|
||||
|
||||
- Enable Azure Policy for compliance
|
||||
- Configure network policies
|
||||
- Set up RBAC
|
||||
- Enable Defender for Cloud
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
1. **Cluster creation fails**:
|
||||
- Check network connectivity between nodes
|
||||
- Verify firewall rules
|
||||
- Check Corosync configuration
|
||||
|
||||
2. **Azure Arc connection fails**:
|
||||
- Verify internet connectivity
|
||||
- Check Azure credentials
|
||||
- Review agent logs: `journalctl -u azcmagent`
|
||||
|
||||
3. **Kubernetes pods not starting**:
|
||||
- Check resource limits
|
||||
- Verify storage classes
|
||||
- Review pod logs: `kubectl logs <pod-name>`
|
||||
|
||||
4. **GitOps not syncing**:
|
||||
- Check Flux logs: `kubectl logs -n flux-system -l app=flux`
|
||||
- Verify repository access
|
||||
- Check GitOps configuration in Azure Portal
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. Review architecture documentation
|
||||
2. Set up monitoring and alerting
|
||||
3. Configure backup and disaster recovery
|
||||
4. Implement security policies
|
||||
5. Plan for scaling and expansion
|
||||
|
||||
Reference in New Issue
Block a user