# Debugging Complete guide for debugging NCN Network services and issues. *** ## Quick Reference ```bash # Enable debug logging export RUST_LOG=debug export RUST_BACKTRACE=1 # Run with debug output cargo run --bin gateway_node ``` *** ## Logging ### Log Levels | Level | Use For | | ------- | ---------------------------------- | | `error` | Critical failures | | `warn` | Warnings, non-critical issues | | `info` | General operational info (default) | | `debug` | Detailed debugging info | | `trace` | Very detailed tracing | ### Configure Logging ```bash # All components at debug export RUST_LOG=debug # Specific component export RUST_LOG=gateway_node=debug # Multiple components export RUST_LOG=gateway_node=debug,compute_node=info # Specific module export RUST_LOG=gateway_node::payment=trace # With dependencies export RUST_LOG=debug,hyper=warn,tonic=warn ``` ### Logging in Code ```rust use log::{debug, info, warn, error, trace}; fn process_request(req: &InferenceRequest) { debug!("Processing request: {:?}", req); if let Err(e) = validate(&req) { error!("Validation failed: {}", e); return; } info!("Request {} processed successfully", req.request_id); } ``` *** ## Backtraces ### Enable Backtraces ```bash # Basic backtrace export RUST_BACKTRACE=1 # Full backtrace export RUST_BACKTRACE=full ``` ### Example Backtrace ``` thread 'main' panicked at 'called `Option::unwrap()` on a `None` value' stack backtrace: 0: std::panicking::begin_panic 1: core::option::Option::unwrap 2: gateway_node::payment::process_payment at ./src/payment.rs:42 3: gateway_node::handler::handle_request at ./src/handler.rs:88 ``` *** ## Debugging Tools ### GDB/LLDB ```bash # Build with debug symbols cargo build # Run with LLDB (macOS) lldb ./target/debug/gateway_node # Run with GDB (Linux) gdb ./target/debug/gateway_node # Set breakpoint (lldb) b gateway_node::payment::process_payment (lldb) run ``` ### VS Code Debugging `.vscode/launch.json`: ```json { "version": "0.2.0", "configurations": [ { "type": "lldb", "request": "launch", "name": "Debug Gateway", "cargo": { "args": ["build", "--bin=gateway_node"] }, "args": [], "cwd": "${workspaceFolder}", "env": { "RUST_LOG": "debug", "RUST_BACKTRACE": "1" } }, { "type": "lldb", "request": "launch", "name": "Debug Tests", "cargo": { "args": ["test", "--no-run", "-p", "gateway_node"], "filter": { "kind": "test" } } } ] } ``` ### rust-analyzer Enable in VS Code for inline error checking: ```json { "rust-analyzer.checkOnSave.command": "clippy" } ``` *** ## Network Debugging ### gRPC Debugging ```bash # List services grpcurl -plaintext localhost:50051 list # Describe service grpcurl -plaintext localhost:50051 describe common_types.GatewayClientService # Call method grpcurl -plaintext \ -d '{"model_uuid": "test", "input_data": "{}"}' \ localhost:50051 common_types.GatewayClientService/SubmitInferenceTask ``` ### HTTP Debugging ```bash # Verbose curl curl -v http://localhost:8080/health # With request body curl -v -X POST http://localhost:8080/api/v1/inference \ -H "Content-Type: application/json" \ -d '{"model_uuid": "test", "input_data": "{}"}' ``` ### WebSocket Debugging ```bash # Install wscat npm install -g wscat # Connect wscat -c ws://localhost:9000 # Send message {"type": "subscribe", "request_id": "test-123"} ``` ### Network Capture ```bash # Capture gRPC traffic tcpdump -i lo -w grpc.pcap port 50051 # Analyze with Wireshark wireshark grpc.pcap ``` *** ## Component-Specific Debugging ### Gateway Debugging ```bash # Full debug logging RUST_LOG=gateway_node=trace cargo run --bin gateway_node # Payment module only RUST_LOG=gateway_node::payment=trace cargo run --bin gateway_node # Check connections curl http://localhost:8080/health ``` **Common Issues:** * Registry connection failed → Check registry is running * No compute nodes → Check compute node registration * Payment failed → Check blockchain RPC connectivity ### Compute Node Debugging ```bash # Full debug with sandbox disabled RUST_LOG=compute_node=trace \ SANDBOX_MODE=disabled \ cargo run --bin compute_node -- --gateway-addr http://127.0.0.1:50051 # Model execution only RUST_LOG=compute_node::executor=trace cargo run --bin compute_node ``` **Common Issues:** * Sandbox failed → Use `SANDBOX_MODE=disabled` * Model not found → Check `--model-path` * Gateway connection → Verify gateway is running ### Registry Debugging ```bash # Full debug logging RUST_LOG=p2p_registry_node=trace cargo run --bin p2p_registry_node # P2P networking RUST_LOG=libp2p=debug,p2p_registry_node=trace cargo run --bin p2p_registry_node ``` **Common Issues:** * No validators → Check validator registration * P2P bootstrap → Verify bootstrap node addresses * Consensus failed → Check minimum validators *** ## Memory Debugging ### Detect Memory Issues ```bash # Install valgrind (Linux) sudo apt install valgrind # Run with valgrind valgrind --leak-check=full ./target/debug/gateway_node ``` ### Heap Profiling ```bash # Install heaptrack sudo apt install heaptrack # Profile heaptrack ./target/release/gateway_node # Analyze heaptrack_gui heaptrack.gateway_node.*.gz ``` *** ## Performance Debugging ### CPU Profiling ```bash # Using perf (Linux) perf record -g ./target/release/gateway_node perf report # Using flamegraph cargo install flamegraph cargo flamegraph --bin gateway_node ``` ### Timing Analysis ```rust use std::time::Instant; fn process_request(req: &InferenceRequest) { let start = Instant::now(); // ... processing ... let duration = start.elapsed(); debug!("Request {} processed in {:?}", req.request_id, duration); } ``` *** ## Contract Debugging ### Foundry Debugging ```bash cd contracts # Verbose test output forge test -vvvv # Debug specific test forge test --debug testInitiateInferenceRequest # Gas profiling forge test --gas-report ``` ### Trace Transactions ```bash # Trace a transaction cast run --rpc-url $RPC_URL # Decode call data cast 4byte-decode ``` *** ## Log Analysis ### Search Logs ```bash # Find errors grep -i error logs/gateway.log # Find specific request grep "req-123" logs/gateway.log # Tail with filter tail -f logs/gateway.log | grep -i payment ``` ### Structured Logging Enable JSON logging for analysis: ```bash export RUST_LOG_FORMAT=json cargo run --bin gateway_node 2>&1 | jq . ``` *** ## Common Debug Scenarios ### Connection Issues ```bash # Check ports netstat -tlnp | grep -E "(50050|50051|8080)" # Check process ps aux | grep gateway # Test connectivity nc -zv localhost 50051 ``` ### Request Not Processing ```bash # Enable request tracing RUST_LOG=gateway_node::handler=trace cargo run --bin gateway_node # Check request flow curl -v -X POST http://localhost:8080/api/v1/inference \ -H "Content-Type: application/json" \ -d '{"model_uuid": "test", "input_data": "{}"}' ``` ### Signature Verification Failures ```bash # Enable crypto debugging RUST_LOG=common_types::crypto=trace cargo run --bin gateway_node # Check signature format # Ensure 0x prefix, 65 bytes (130 hex chars + 0x) ``` ### Blockchain Issues ```bash # Test RPC connection curl -X POST $RPC_URL \ -H "Content-Type: application/json" \ -d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}' # Check contract cast call $CONTRACT_ADDRESS "minValidatorSignatures()" --rpc-url $RPC_URL ``` *** ## Debug Checklist ### Before Reporting Issues * [ ] Enable debug logging (`RUST_LOG=debug`) * [ ] Enable backtraces (`RUST_BACKTRACE=1`) * [ ] Reproduce the issue consistently * [ ] Capture relevant logs * [ ] Note environment details (OS, versions) * [ ] Check for existing issues ### Information to Collect 1. Full error message and backtrace 2. Relevant log output 3. Steps to reproduce 4. Configuration (sanitized) 5. Environment details *** ## Next Steps * [Testing](https://docs.neurochain.ai/nc/neurochainai-guides/development/testing) - Run tests * [Troubleshooting](https://docs.neurochain.ai/nc/neurochainai-guides/troubleshooting/troubleshooting) - Common issues * [FAQ](https://docs.neurochain.ai/nc/neurochainai-guides/troubleshooting/faq) - Frequently asked questions