Production Deployment

Best practices and considerations for deploying NCN Network in production.

---

Production Checklist

Security

• TLS certificates configured for all endpoints

• Private keys stored in secure vault (HSM recommended)

• Firewall rules configured

• Rate limiting enabled

• Security audit completed

Infrastructure

• High availability setup (multiple replicas)

• Load balancer configured

• Auto-scaling enabled

• Persistent storage for data

• Backup strategy implemented

Monitoring

• Prometheus metrics enabled

• Grafana dashboards configured

• Alerting rules set up

• Log aggregation configured

• Health checks implemented

Operations

• Runbooks documented

• Incident response plan

• Update/rollback procedures

• On-call rotation established

---

TLS Configuration

Generate Certificates

# Using Let's Encrypt with certbot
sudo apt install certbot
sudo certbot certonly --standalone -d api.ncn-network.io

# Certificates will be at:
# /etc/letsencrypt/live/api.ncn-network.io/fullchain.pem
# /etc/letsencrypt/live/api.ncn-network.io/privkey.pem

Configure Gateway TLS

# gateway.env
TLS_ENABLED=true
TLS_CERT_PATH=/etc/ncn/certs/fullchain.pem
TLS_KEY_PATH=/etc/ncn/certs/privkey.pem

Certificate Renewal

# Add to crontab
0 0 1 * * certbot renew --quiet && systemctl reload ncn-gateway

---

High Availability

Gateway HA

                    Load Balancer
                    ┌───────────────────┐
                    │   HAProxy/NGINX   │
                    │   (health checks) │
                    └─────────┬─────────┘
                              │
              ┌───────────────┼───────────────┐
              │               │               │
              ▼               ▼               ▼
        ┌──────────┐   ┌──────────┐   ┌──────────┐
        │Gateway 1 │   │Gateway 2 │   │Gateway 3 │
        │ Server A │   │ Server B │   │ Server C │
        └──────────┘   └──────────┘   └──────────┘

HAProxy Configuration

# /etc/haproxy/haproxy.cfg
frontend ncn-http
    bind *:80
    bind *:443 ssl crt /etc/haproxy/certs/ncn.pem
    default_backend ncn-gateway

backend ncn-gateway
    balance roundrobin
    option httpchk GET /health
    server gateway1 10.0.1.1:8080 check
    server gateway2 10.0.1.2:8080 check
    server gateway3 10.0.1.3:8080 check

frontend ncn-grpc
    bind *:50051
    mode tcp
    default_backend ncn-gateway-grpc

backend ncn-gateway-grpc
    mode tcp
    balance roundrobin
    server gateway1 10.0.1.1:50051 check
    server gateway2 10.0.1.2:50051 check
    server gateway3 10.0.1.3:50051 check

---

Security Hardening

Firewall Rules

# Public-facing (Gateway only)
sudo ufw allow 443/tcp      # HTTPS
sudo ufw allow 50051/tcp    # gRPC (if public)

# Internal only
sudo ufw deny 50050/tcp     # Registry gRPC (internal)
sudo ufw deny 8828/tcp      # Registry P2P (internal)

# Block all other inbound
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw enable

Rate Limiting (NGINX)

# /etc/nginx/conf.d/rate-limit.conf
limit_req_zone $binary_remote_addr zone=api:10m rate=100r/m;

server {
    location /api/ {
        limit_req zone=api burst=20 nodelay;
        proxy_pass http://ncn-gateway;
    }
}

Key Management

Development:

• Environment variables (acceptable)

Staging:

• Secrets manager (AWS Secrets Manager, HashiCorp Vault)

Production:

• Hardware Security Module (HSM)

• Key Management Service (AWS KMS, GCP KMS)

# Using AWS Secrets Manager
GATEWAY_WALLET_PRIVATE_KEY=$(aws secretsmanager get-secret-value \
  --secret-id ncn/gateway-key \
  --query SecretString \
  --output text)

---

Monitoring Setup

Prometheus Configuration

# prometheus.yml
scrape_configs:
  - job_name: 'ncn-gateway'
    static_configs:
      - targets: ['gateway1:8080', 'gateway2:8080', 'gateway3:8080']
    metrics_path: /metrics

  - job_name: 'ncn-registry'
    static_configs:
      - targets: ['registry1:50050', 'registry2:50050', 'registry3:50050']

Key Metrics to Monitor

Metric	Alert Threshold
Request latency p99	> 5s
Error rate	> 5%
CPU usage	> 80%
Memory usage	> 85%
Disk usage	> 90%
Active connections	> 1000

Alerting Rules

# alerts.yml
groups:
- name: ncn-alerts
  rules:
  - alert: HighErrorRate
    expr: rate(ncn_requests_failed_total[5m]) / rate(ncn_requests_total[5m]) > 0.05
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "High error rate on NCN Gateway"
      
  - alert: HighLatency
    expr: histogram_quantile(0.99, rate(ncn_request_duration_seconds_bucket[5m])) > 5
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "High latency on NCN Gateway"

---

Backup Strategy

What to Backup

Data	Frequency	Retention
Configuration files	Daily	30 days
Registry DHT data	Every 6 hours	7 days
Logs	Daily	90 days
Wallet keys	Once (secure offsite)	Forever

Backup Script

#!/bin/bash
# /opt/ncn/scripts/backup.sh

DATE=$(date +%Y%m%d_%H%M%S)
BACKUP_DIR=/backup/ncn/$DATE

mkdir -p $BACKUP_DIR

# Backup configuration
cp -r /opt/ncn/config $BACKUP_DIR/

# Backup data
tar -czf $BACKUP_DIR/data.tar.gz /opt/ncn/data/

# Upload to S3
aws s3 sync $BACKUP_DIR s3://ncn-backups/$DATE/

# Cleanup old backups
find /backup/ncn -type d -mtime +30 -exec rm -rf {} \;

---

Disaster Recovery

Recovery Time Objective (RTO)

Component	RTO
Gateway	5 minutes
Registry	15 minutes
Compute	30 minutes

Recovery Procedures

Gateway Failure:

1. Health check detects failure

2. Load balancer removes from pool

3. Alert sent to ops team

4. Auto-restart or manual intervention

5. Verify health, add back to pool

Registry Failure:

1. P2P network continues with remaining nodes

2. Failed node restarts automatically

3. DHT data syncs from peers

4. Verify consensus capability

Full Cluster Recovery:

# 1. Start infrastructure
systemctl start ncn-registry
sleep 30

# 2. Start gateways
systemctl start ncn-gateway
sleep 10

# 3. Start compute nodes
systemctl start ncn-compute

# 4. Verify health
curl http://localhost:8080/health

---

Performance Tuning

System Limits

# /etc/security/limits.conf
ncn soft nofile 65535
ncn hard nofile 65535
ncn soft nproc 32768
ncn hard nproc 32768

Kernel Parameters

# /etc/sysctl.conf
net.core.somaxconn = 65535
net.ipv4.tcp_max_syn_backlog = 65535
net.ipv4.ip_local_port_range = 1024 65535
net.ipv4.tcp_tw_reuse = 1

Application Tuning

# Gateway tuning
TOKIO_WORKER_THREADS=4
HTTP_KEEP_ALIVE_TIMEOUT=60
GRPC_MAX_CONNECTIONS=1000

---

Update Procedures

Rolling Update

# 1. Update one gateway at a time
for host in gateway1 gateway2 gateway3; do
  ssh $host "systemctl stop ncn-gateway"
  ssh $host "cp /tmp/gateway_node /opt/ncn/bin/"
  ssh $host "systemctl start ncn-gateway"
  sleep 30
  # Verify health before continuing
  curl -f http://$host:8080/health || exit 1
done

Rollback Procedure

# Keep previous version
cp /opt/ncn/bin/gateway_node /opt/ncn/bin/gateway_node.backup

# Rollback if needed
systemctl stop ncn-gateway
cp /opt/ncn/bin/gateway_node.backup /opt/ncn/bin/gateway_node
systemctl start ncn-gateway

---

Compliance Considerations

Data Handling

• Minimize data retention

• Encrypt data at rest

• Log access to sensitive data

• GDPR compliance for EU users

Audit Logging

# Enable audit logging
AUDIT_LOG_ENABLED=true
AUDIT_LOG_PATH=/var/log/ncn/audit.log

---

Next Steps

• Monitoring - Detailed monitoring setup

• Troubleshooting - Common issues

• Security - Security documentation