> For the complete documentation index, see [llms.txt](https://docs.neurochain.ai/nc/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.neurochain.ai/nc/neurochainai-guides/operators/monitoring.md).

# Monitoring

Complete guide for monitoring NCN Network services.

***

## Overview

Effective monitoring includes:

* **Metrics**: Numerical data (latency, throughput, errors)
* **Logs**: Event records and debugging info
* **Alerts**: Notifications for issues
* **Dashboards**: Visual status overview

***

## Prometheus Setup

### Install Prometheus

```bash
# Download Prometheus
wget https://github.com/prometheus/prometheus/releases/download/v2.48.0/prometheus-2.48.0.linux-amd64.tar.gz
tar xvfz prometheus-*.tar.gz
sudo mv prometheus-*/prometheus /usr/local/bin/
sudo mv prometheus-*/promtool /usr/local/bin/
```

### Configure Prometheus

Create `/etc/prometheus/prometheus.yml`:

```yaml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'ncn-gateway'
    static_configs:
      - targets: ['localhost:8080']
    metrics_path: /metrics

  - job_name: 'ncn-registry'
    static_configs:
      - targets: ['localhost:50050']
    metrics_path: /metrics

  - job_name: 'node-exporter'
    static_configs:
      - targets: ['localhost:9100']

alerting:
  alertmanagers:
    - static_configs:
        - targets: ['localhost:9093']

rule_files:
  - /etc/prometheus/rules/*.yml
```

### Start Prometheus

```bash
prometheus --config.file=/etc/prometheus/prometheus.yml
```

***

## Key Metrics

### Gateway Metrics

| Metric                         | Type      | Description              |
| ------------------------------ | --------- | ------------------------ |
| `ncn_requests_total`           | Counter   | Total requests           |
| `ncn_requests_failed_total`    | Counter   | Failed requests          |
| `ncn_request_duration_seconds` | Histogram | Request latency          |
| `ncn_active_connections`       | Gauge     | Active connections       |
| `ncn_compute_nodes_registered` | Gauge     | Registered compute nodes |

### Registry Metrics

| Metric                        | Type    | Description          |
| ----------------------------- | ------- | -------------------- |
| `ncn_validators_active`       | Gauge   | Active validators    |
| `ncn_validations_total`       | Counter | Total validations    |
| `ncn_consensus_reached_total` | Counter | Successful consensus |
| `ncn_mempool_size`            | Gauge   | Pending requests     |
| `ncn_p2p_peers`               | Gauge   | Connected peers      |

### Compute Metrics

| Metric                      | Type      | Description         |
| --------------------------- | --------- | ------------------- |
| `ncn_tasks_completed_total` | Counter   | Completed tasks     |
| `ncn_task_duration_seconds` | Histogram | Task execution time |
| `ncn_gpu_utilization`       | Gauge     | GPU usage           |
| `ncn_memory_used_bytes`     | Gauge     | Memory usage        |

***

## Grafana Dashboards

### Install Grafana

```bash
# Add repository
sudo apt-get install -y software-properties-common
sudo add-apt-repository "deb https://packages.grafana.com/oss/deb stable main"
wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
sudo apt-get update
sudo apt-get install grafana

# Start Grafana
sudo systemctl start grafana-server
sudo systemctl enable grafana-server
```

### Access Grafana

* URL: <http://localhost:3000>
* Default credentials: admin/admin

### NCN Dashboard JSON

Create dashboard with panels:

```json
{
  "title": "NCN Network Overview",
  "panels": [
    {
      "title": "Request Rate",
      "type": "graph",
      "targets": [
        {
          "expr": "rate(ncn_requests_total[5m])",
          "legendFormat": "{{instance}}"
        }
      ]
    },
    {
      "title": "Error Rate",
      "type": "graph",
      "targets": [
        {
          "expr": "rate(ncn_requests_failed_total[5m]) / rate(ncn_requests_total[5m]) * 100",
          "legendFormat": "Error %"
        }
      ]
    },
    {
      "title": "P99 Latency",
      "type": "graph",
      "targets": [
        {
          "expr": "histogram_quantile(0.99, rate(ncn_request_duration_seconds_bucket[5m]))",
          "legendFormat": "p99"
        }
      ]
    },
    {
      "title": "Active Validators",
      "type": "stat",
      "targets": [
        {
          "expr": "ncn_validators_active"
        }
      ]
    },
    {
      "title": "Compute Nodes",
      "type": "stat",
      "targets": [
        {
          "expr": "ncn_compute_nodes_registered"
        }
      ]
    }
  ]
}
```

***

## Alerting

### Alert Rules

Create `/etc/prometheus/rules/ncn-alerts.yml`:

```yaml
groups:
- name: ncn-critical
  rules:
  - alert: HighErrorRate
    expr: rate(ncn_requests_failed_total[5m]) / rate(ncn_requests_total[5m]) > 0.05
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "High error rate on NCN Gateway"
      description: "Error rate is {{ $value | humanizePercentage }}"

  - alert: GatewayDown
    expr: up{job="ncn-gateway"} == 0
    for: 1m
    labels:
      severity: critical
    annotations:
      summary: "NCN Gateway is down"

  - alert: NoComputeNodes
    expr: ncn_compute_nodes_registered == 0
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "No compute nodes registered"

- name: ncn-warning
  rules:
  - alert: HighLatency
    expr: histogram_quantile(0.99, rate(ncn_request_duration_seconds_bucket[5m])) > 5
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "High latency on NCN Gateway"
      description: "P99 latency is {{ $value }}s"

  - alert: LowValidators
    expr: ncn_validators_active < 5
    for: 10m
    labels:
      severity: warning
    annotations:
      summary: "Low number of active validators"

  - alert: HighMemoryUsage
    expr: (node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / node_memory_MemTotal_bytes > 0.85
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "High memory usage"
```

### Alertmanager Configuration

Create `/etc/alertmanager/alertmanager.yml`:

```yaml
global:
  resolve_timeout: 5m

route:
  group_by: ['alertname', 'severity']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: 'default'
  routes:
    - match:
        severity: critical
      receiver: 'critical'

receivers:
  - name: 'default'
    email_configs:
      - to: 'ops@example.com'
        from: 'alerts@ncn-network.io'
        smarthost: 'smtp.example.com:587'

  - name: 'critical'
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/xxx'
        channel: '#alerts-critical'
    pagerduty_configs:
      - service_key: 'xxx'
```

***

## Log Aggregation

### Using Loki + Promtail

```yaml
# promtail-config.yml
server:
  http_listen_port: 9080

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: ncn-gateway
    static_configs:
      - targets:
          - localhost
        labels:
          job: ncn-gateway
          __path__: /var/log/ncn/gateway.log

  - job_name: ncn-registry
    static_configs:
      - targets:
          - localhost
        labels:
          job: ncn-registry
          __path__: /var/log/ncn/registry.log

  - job_name: ncn-compute
    static_configs:
      - targets:
          - localhost
        labels:
          job: ncn-compute
          __path__: /var/log/ncn/compute.log
```

### Log Queries (LogQL)

```logql
# Error logs
{job="ncn-gateway"} |= "error"

# Slow requests
{job="ncn-gateway"} | json | duration > 5s

# Failed validations
{job="ncn-registry"} |= "validation failed"
```

***

## Health Checks

### HTTP Health Check

```bash
# Gateway health
curl -f http://localhost:8080/health || echo "Gateway unhealthy"

# Expected response
{
  "status": "healthy",
  "version": "0.1.0",
  "uptime_seconds": 3600
}
```

### Custom Health Script

Create `/opt/ncn/health-check.sh`:

```bash
#!/bin/bash

# Check gateway
if ! curl -sf http://localhost:8080/health > /dev/null; then
  echo "Gateway unhealthy"
  exit 1
fi

# Check registry
if ! nc -z localhost 50050; then
  echo "Registry unhealthy"
  exit 1
fi

# Check compute (optional)
if systemctl is-active --quiet ncn-compute; then
  echo "All services healthy"
  exit 0
else
  echo "Compute node not running"
  exit 1
fi
```

### Systemd Health Integration

```ini
[Service]
ExecStartPost=/opt/ncn/health-check.sh
```

***

## Uptime Monitoring

### External Monitoring

Use services like:

* Uptime Robot
* Pingdom
* Better Uptime

Configure to monitor:

* `https://api.ncn-network.io/health`
* Expected response code: 200
* Check interval: 1 minute

***

## Resource Monitoring

### Node Exporter

```bash
# Install node exporter
wget https://github.com/prometheus/node_exporter/releases/download/v1.7.0/node_exporter-1.7.0.linux-amd64.tar.gz
tar xvfz node_exporter-*.tar.gz
sudo mv node_exporter-*/node_exporter /usr/local/bin/

# Run as service
sudo systemctl start node_exporter
```

### GPU Monitoring (NVIDIA)

```bash
# Install DCGM exporter for GPU metrics
docker run -d --gpus all \
  -p 9400:9400 \
  nvidia/dcgm-exporter
```

***

## Dashboard Examples

### Key Dashboard Panels

1. **Overview**
   * Request rate
   * Error rate
   * Active nodes
2. **Performance**
   * Latency percentiles
   * Throughput
   * Queue depth
3. **Resources**
   * CPU usage
   * Memory usage
   * Disk I/O
   * Network I/O
4. **Validators**
   * Active validators
   * Consensus success rate
   * Reputation distribution

***

## Best Practices

1. **Retention**
   * Metrics: 15-30 days
   * Logs: 90 days
   * Alerts history: 1 year
2. **Alerting**
   * Alert on symptoms, not causes
   * Have actionable runbooks
   * Avoid alert fatigue
3. **Dashboards**
   * Start with high-level overview
   * Drill-down capability
   * Include links to runbooks
4. **On-Call**
   * Define escalation paths
   * Regular rotation
   * Post-incident reviews

***

## Next Steps

* [Production Deployment](/nc/neurochainai-guides/deployment/production.md) - Production setup
* [Troubleshooting](/nc/neurochainai-guides/troubleshooting/troubleshooting.md) - Common issues