Debugging

Complete guide for debugging NCN Network services and issues.

---

Quick Reference

# Enable debug logging
export RUST_LOG=debug
export RUST_BACKTRACE=1

# Run with debug output
cargo run --bin gateway_node

---

Logging

Log Levels

Level	Use For
error	Critical failures
warn	Warnings, non-critical issues
info	General operational info (default)
debug	Detailed debugging info
trace	Very detailed tracing

Configure Logging

# All components at debug
export RUST_LOG=debug

# Specific component
export RUST_LOG=gateway_node=debug

# Multiple components
export RUST_LOG=gateway_node=debug,compute_node=info

# Specific module
export RUST_LOG=gateway_node::payment=trace

# With dependencies
export RUST_LOG=debug,hyper=warn,tonic=warn

Logging in Code

use log::{debug, info, warn, error, trace};

fn process_request(req: &InferenceRequest) {
    debug!("Processing request: {:?}", req);
    
    if let Err(e) = validate(&req) {
        error!("Validation failed: {}", e);
        return;
    }
    
    info!("Request {} processed successfully", req.request_id);
}

---

Backtraces

Enable Backtraces

# Basic backtrace
export RUST_BACKTRACE=1

# Full backtrace
export RUST_BACKTRACE=full

Example Backtrace

thread 'main' panicked at 'called `Option::unwrap()` on a `None` value'
stack backtrace:
   0: std::panicking::begin_panic
   1: core::option::Option<T>::unwrap
   2: gateway_node::payment::process_payment
             at ./src/payment.rs:42
   3: gateway_node::handler::handle_request
             at ./src/handler.rs:88

---

Debugging Tools

GDB/LLDB

# Build with debug symbols
cargo build

# Run with LLDB (macOS)
lldb ./target/debug/gateway_node

# Run with GDB (Linux)
gdb ./target/debug/gateway_node

# Set breakpoint
(lldb) b gateway_node::payment::process_payment
(lldb) run

VS Code Debugging

.vscode/launch.json:

{
  "version": "0.2.0",
  "configurations": [
    {
      "type": "lldb",
      "request": "launch",
      "name": "Debug Gateway",
      "cargo": {
        "args": ["build", "--bin=gateway_node"]
      },
      "args": [],
      "cwd": "${workspaceFolder}",
      "env": {
        "RUST_LOG": "debug",
        "RUST_BACKTRACE": "1"
      }
    },
    {
      "type": "lldb",
      "request": "launch",
      "name": "Debug Tests",
      "cargo": {
        "args": ["test", "--no-run", "-p", "gateway_node"],
        "filter": {
          "kind": "test"
        }
      }
    }
  ]
}

rust-analyzer

Enable in VS Code for inline error checking:

{
  "rust-analyzer.checkOnSave.command": "clippy"
}

---

Network Debugging

gRPC Debugging

# List services
grpcurl -plaintext localhost:50051 list

# Describe service
grpcurl -plaintext localhost:50051 describe common_types.GatewayClientService

# Call method
grpcurl -plaintext \
  -d '{"model_uuid": "test", "input_data": "{}"}' \
  localhost:50051 common_types.GatewayClientService/SubmitInferenceTask

HTTP Debugging

# Verbose curl
curl -v http://localhost:8080/health

# With request body
curl -v -X POST http://localhost:8080/api/v1/inference \
  -H "Content-Type: application/json" \
  -d '{"model_uuid": "test", "input_data": "{}"}'

WebSocket Debugging

# Install wscat
npm install -g wscat

# Connect
wscat -c ws://localhost:9000

# Send message
{"type": "subscribe", "request_id": "test-123"}

Network Capture

# Capture gRPC traffic
tcpdump -i lo -w grpc.pcap port 50051

# Analyze with Wireshark
wireshark grpc.pcap

---

Component-Specific Debugging

Gateway Debugging

# Full debug logging
RUST_LOG=gateway_node=trace cargo run --bin gateway_node

# Payment module only
RUST_LOG=gateway_node::payment=trace cargo run --bin gateway_node

# Check connections
curl http://localhost:8080/health

Common Issues:

• Registry connection failed → Check registry is running

• No compute nodes → Check compute node registration

• Payment failed → Check blockchain RPC connectivity

Compute Node Debugging

# Full debug with sandbox disabled
RUST_LOG=compute_node=trace \
SANDBOX_MODE=disabled \
cargo run --bin compute_node -- --gateway-addr http://127.0.0.1:50051

# Model execution only
RUST_LOG=compute_node::executor=trace cargo run --bin compute_node

Common Issues:

• Sandbox failed → Use SANDBOX_MODE=disabled

• Model not found → Check --model-path

• Gateway connection → Verify gateway is running

Registry Debugging

# Full debug logging
RUST_LOG=p2p_registry_node=trace cargo run --bin p2p_registry_node

# P2P networking
RUST_LOG=libp2p=debug,p2p_registry_node=trace cargo run --bin p2p_registry_node

Common Issues:

• No validators → Check validator registration

• P2P bootstrap → Verify bootstrap node addresses

• Consensus failed → Check minimum validators

---

Memory Debugging

Detect Memory Issues

# Install valgrind (Linux)
sudo apt install valgrind

# Run with valgrind
valgrind --leak-check=full ./target/debug/gateway_node

Heap Profiling

# Install heaptrack
sudo apt install heaptrack

# Profile
heaptrack ./target/release/gateway_node

# Analyze
heaptrack_gui heaptrack.gateway_node.*.gz

---

Performance Debugging

CPU Profiling

# Using perf (Linux)
perf record -g ./target/release/gateway_node
perf report

# Using flamegraph
cargo install flamegraph
cargo flamegraph --bin gateway_node

Timing Analysis

use std::time::Instant;

fn process_request(req: &InferenceRequest) {
    let start = Instant::now();
    
    // ... processing ...
    
    let duration = start.elapsed();
    debug!("Request {} processed in {:?}", req.request_id, duration);
}

---

Contract Debugging

Foundry Debugging

cd contracts

# Verbose test output
forge test -vvvv

# Debug specific test
forge test --debug testInitiateInferenceRequest

# Gas profiling
forge test --gas-report

Trace Transactions

# Trace a transaction
cast run <tx_hash> --rpc-url $RPC_URL

# Decode call data
cast 4byte-decode <calldata>

---

Log Analysis

Search Logs

# Find errors
grep -i error logs/gateway.log

# Find specific request
grep "req-123" logs/gateway.log

# Tail with filter
tail -f logs/gateway.log | grep -i payment

Structured Logging

Enable JSON logging for analysis:

export RUST_LOG_FORMAT=json
cargo run --bin gateway_node 2>&1 | jq .

---

Common Debug Scenarios

Connection Issues

# Check ports
netstat -tlnp | grep -E "(50050|50051|8080)"

# Check process
ps aux | grep gateway

# Test connectivity
nc -zv localhost 50051

Request Not Processing

# Enable request tracing
RUST_LOG=gateway_node::handler=trace cargo run --bin gateway_node

# Check request flow
curl -v -X POST http://localhost:8080/api/v1/inference \
  -H "Content-Type: application/json" \
  -d '{"model_uuid": "test", "input_data": "{}"}'

Signature Verification Failures

# Enable crypto debugging
RUST_LOG=common_types::crypto=trace cargo run --bin gateway_node

# Check signature format
# Ensure 0x prefix, 65 bytes (130 hex chars + 0x)

Blockchain Issues

# Test RPC connection
curl -X POST $RPC_URL \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}'

# Check contract
cast call $CONTRACT_ADDRESS "minValidatorSignatures()" --rpc-url $RPC_URL

---

Debug Checklist

Before Reporting Issues

• Enable debug logging (RUST_LOG=debug)

• Enable backtraces (RUST_BACKTRACE=1)

• Reproduce the issue consistently

• Capture relevant logs

• Note environment details (OS, versions)

• Check for existing issues

Information to Collect

1. Full error message and backtrace

2. Relevant log output

3. Steps to reproduce

4. Configuration (sanitized)

5. Environment details

---

Next Steps

• Testing - Run tests

• Troubleshooting - Common issues

• FAQ - Frequently asked questions