proxmox/docs/04-configuration/HSM_MIGRATION_GUIDE.md

HSM Key Vault Migration Guide

Last Updated: 2026-01-31
Document Version: 1.0
Status: Active Documentation

---

Date: 2026-01-26
Status: 🔴 CRITICAL - IMMEDIATE ACTION REQUIRED
Purpose: Guide for migrating private keys and secrets from files to HSM/Key Vault

---

Executive Summary

Current Risk: 🔴 CRITICAL

Private keys and sensitive secrets are currently stored in .env files and documentation, creating a significant security risk. This guide provides step-by-step instructions for migrating all secrets to a Hardware Security Module (HSM) or Key Vault system.

---

🔴 Critical Secrets Identified

Private Keys (CRITICAL - Highest Priority)

Locations Found:

1. smom-dbis-138/.env - Deployer private key
2. no_five/.env - Private key (same as deployer)
3. 237-combo/.env - Different private key
4. loc_az_hci/smom-dbis-138/.env - Deployer private key
5. smom-dbis-138/services/*/.env - Multiple service files
6. docs/06-besu/T1_2_CREDENTIALS_VERIFIED.md - Documented in markdown

Risk Level: 🔴 CRITICAL

• Complete compromise of blockchain accounts
• Unauthorized transaction signing
• Financial loss
• Reputation damage

---

Migration Strategy

Phase 1: Immediate Actions (Day 1)

Step 1: Identify All Private Keys

# Search for private keys in .env files
find . -name ".env" -type f -exec grep -l "PRIVATE_KEY" {} \;

# Search for private keys in documentation
find docs -name "*.md" -type f -exec grep -l "0x[0-9a-fA-F]\{64\}" {} \;

Step 2: Document Current Keys

Create a secure inventory (encrypted or in secure location):

# Create encrypted inventory
cat > /tmp/key-inventory.txt <<EOF
# Private Key Inventory - DO NOT COMMIT
# Created: $(date)

# Key 1: Deployer Key (ChainID 138)
Location: smom-dbis-138/.env
Address: [REDACTED - check blockchain explorer]
Purpose: Contract deployment
Status: ACTIVE - MIGRATE IMMEDIATELY

# Key 2: Service Key
Location: smom-dbis-138/services/*/.env
Purpose: Service operations
Status: ACTIVE - MIGRATE IMMEDIATELY

EOF

# Encrypt the inventory
gpg --encrypt --recipient your-email@example.com /tmp/key-inventory.txt
rm /tmp/key-inventory.txt

Step 3: Set Up HSM/Key Vault

Option A: AWS KMS

# Create KMS key
aws kms create-key --description "Blockchain Deployer Key" --key-usage SIGN_VERIFY

# Create alias
aws kms create-alias --alias-name alias/blockchain-deployer --target-key-id <key-id>

Option B: Azure Key Vault

# Create key vault
az keyvault create --name blockchain-keyvault --resource-group your-rg

# Create key
az keyvault key create --vault-name blockchain-keyvault --name deployer-key --kty EC --curve P-256

Option C: HashiCorp Vault

# Enable transit secrets engine
vault secrets enable transit

# Create encryption key
vault write -f transit/keys/blockchain-deployer

---

Phase 2: Key Migration (Days 2-3)

Step 4: Import Keys to HSM

⚠️ IMPORTANT: Never export private keys from HSM. Use HSM for all operations.

For AWS KMS:

# Import key material (if supported by your HSM)
# Note: AWS KMS doesn't support importing EC private keys directly
# You may need to use AWS CloudHSM or generate new keys in KMS

For Azure Key Vault:

# Import key from file (if supported)
az keyvault key import \
  --vault-name blockchain-keyvault \
  --name deployer-key \
  --pem-file private-key.pem

For HashiCorp Vault:

# Import key (if supported)
vault write transit/keys/blockchain-deployer \
  type=ecdsa-p256 \
  exportable=false

Step 5: Update Application Code

Before (Insecure):

const privateKey = process.env.PRIVATE_KEY; // ❌ Insecure
const wallet = new ethers.Wallet(privateKey);

After (Secure):

// Use HSM for signing
import { KMSClient } from '@aws-sdk/client-kms';

const kmsClient = new KMSClient({ region: 'us-east-1' });
const keyId = 'arn:aws:kms:us-east-1:123456789012:key/12345678-1234-1234-1234-123456789012';

// Sign transaction using HSM
async function signWithHSM(message: string) {
  const signCommand = new SignCommand({
    KeyId: keyId,
    Message: Buffer.from(message),
    MessageType: 'RAW',
    SigningAlgorithm: 'ECDSA_SHA_256',
  });
  
  const response = await kmsClient.send(signCommand);
  return response.Signature;
}

Step 6: Remove Keys from Files

⚠️ CRITICAL: Only remove keys after:

1. Keys are successfully imported to HSM
2. Application code is updated to use HSM
3. All operations are tested with HSM
4. Backup of keys is securely stored (encrypted, offline)

# Remove private keys from .env files
find . -name ".env" -type f -exec sed -i '/^PRIVATE_KEY=/d' {} \;

# Remove from documentation
find docs -name "*.md" -type f -exec sed -i 's/0x[0-9a-fA-F]\{64\}/[PRIVATE_KEY_REDACTED]/g' {} \;

---

Phase 3: Key Rotation (Days 4-5)

Step 7: Generate New Keys in HSM

If keys were exposed, generate new keys:

# AWS KMS - Generate new key
aws kms create-key --description "New Blockchain Deployer Key"

# Azure Key Vault - Generate new key
az keyvault key create --vault-name blockchain-keyvault --name deployer-key-new --kty EC

# HashiCorp Vault - Generate new key
vault write -f transit/keys/blockchain-deployer-new

Step 8: Update Deployed Contracts

If using new keys, update contract ownership:

// Transfer ownership to new address
contract.transferOwnership(newOwnerAddress);

---

API Tokens and Passwords Migration

Cloudflare API Credentials

Current Locations:

• proxmox/.env - API key and tunnel token
• scripts/fix-certbot-dns-propagation.sh - Hardcoded token
• scripts/install-shared-tunnel-token.sh - Hardcoded tunnel token

Migration Steps:

1. Create API tokens in Cloudflare (not global API key)

   • Limit permissions to specific operations
   • Set expiration dates

2. Store in Key Vault:

# AWS Secrets Manager
aws secretsmanager create-secret \
  --name cloudflare/api-token \
  --secret-string "your-token"

# Azure Key Vault
az keyvault secret set \
  --vault-name blockchain-keyvault \
  --name cloudflare-api-token \
  --value "your-token"

# HashiCorp Vault
vault kv put secret/cloudflare api-token="your-token"

3. Update scripts to use Key Vault:

# Before
CLOUDFLARE_API_TOKEN="hardcoded-token"

# After
CLOUDFLARE_API_TOKEN=$(aws secretsmanager get-secret-value \
  --secret-id cloudflare/api-token \
  --query SecretString --output text)

---

NPM (Nginx Proxy Manager) Credentials

Current Locations:

• proxmox/.env - Plain text password
• scripts/create-npmplus-proxy.sh - Hardcoded password hash
• scripts/nginx-proxy-manager/update-npmplus-proxy-hosts-api.sh - Hardcoded password

Migration Steps:

1. Store in Key Vault
2. Update scripts (already done for some scripts)
3. Use API tokens instead of passwords (if NPM supports it)

---

Database Credentials

Current Locations:

• dbis_core/.env - DATABASE_URL with embedded password
• explorer-monorepo/.env - Database credentials

Migration Steps:

1. Use Vault database secrets engine (dynamic credentials)
2. Separate password from connection string
3. Rotate credentials regularly

---

Implementation Checklist

Immediate (Day 1)

• Identify all private keys
• Document current keys (encrypted)
• Set up HSM/Key Vault
• Create secure backup of keys

Short Term (Days 2-3)

• Import keys to HSM
• Update application code to use HSM
• Test all operations with HSM
• Remove keys from files
• Remove keys from documentation

Medium Term (Days 4-5)

• Generate new keys (if keys were exposed)
• Update contract ownership
• Rotate API tokens
• Update all scripts to use Key Vault

Long Term (Week 2+)

• Set up automated key rotation
• Implement key rotation policies
• Set up monitoring and alerts
• Document HSM operations procedures

---

Security Best Practices

Key Management

1. Never export private keys from HSM
2. Use HSM for all cryptographic operations
3. Rotate keys regularly (annually or after exposure)
4. Limit key access (principle of least privilege)
5. Monitor key usage (audit logs)

Access Control

1. Use IAM roles (not API keys) where possible
2. Limit permissions (minimum required)
3. Set expiration dates on tokens
4. Rotate credentials regularly

Monitoring

1. Enable audit logging for all key operations
2. Set up alerts for unusual key usage
3. Review access logs regularly
4. Monitor for exposed credentials (GitHub, logs, etc.)

---

Tools and Resources

HSM/Key Vault Options

• AWS KMS - Managed key service
• AWS CloudHSM - Dedicated HSM
• Azure Key Vault - Managed key service
• HashiCorp Vault - Self-hosted secrets management
• Google Cloud KMS - Managed key service

Migration Tools

• git-secrets - Prevent committing secrets
• truffleHog - Scan for secrets in git history
• detect-secrets - Detect secrets in codebase
• AWS Secrets Manager Migration - Migrate to AWS

Documentation

• AWS KMS Best Practices
• Azure Key Vault Best Practices
• HashiCorp Vault Best Practices

---

Emergency Procedures

If Keys Are Exposed

1. Immediately rotate keys
2. Revoke access for exposed keys
3. Monitor accounts for unauthorized activity
4. Review transaction history
5. Update all systems with new keys
6. Document incident and lessons learned

If HSM Becomes Unavailable

1. Use backup HSM (if available)
2. Use encrypted backup keys (last resort)
3. Document downtime and impact
4. Review and improve redundancy

---

Status Tracking

Current Status: 🔴 CRITICAL - IMMEDIATE ACTION REQUIRED

Next Review Date: After Phase 1 completion

Responsible Party: [To be assigned]

---

⚠️ WARNING: Do not delay this migration. Private keys in files represent a critical security risk that could result in complete compromise of blockchain accounts and financial loss.