# Debugging Complete guide for debugging NCN Network services and issues. *** ## Quick Reference ```bash # Enable debug logging export RUST_LOG=debug export RUST_BACKTRACE=1 # Run with debug output cargo run --bin gateway_node ``` *** ## Logging ### Log Levels | Level | Use For | | ------- | ---------------------------------- | | `error` | Critical failures | | `warn` | Warnings, non-critical issues | | `info` | General operational info (default) | | `debug` | Detailed debugging info | | `trace` | Very detailed tracing | ### Configure Logging ```bash # All components at debug export RUST_LOG=debug # Specific component export RUST_LOG=gateway_node=debug # Multiple components export RUST_LOG=gateway_node=debug,compute_node=info # Specific module export RUST_LOG=gateway_node::payment=trace # With dependencies export RUST_LOG=debug,hyper=warn,tonic=warn ``` ### Logging in Code ```rust use log::{debug, info, warn, error, trace}; fn process_request(req: &InferenceRequest) { debug!("Processing request: {:?}", req); if let Err(e) = validate(&req) { error!("Validation failed: {}", e); return; } info!("Request {} processed successfully", req.request_id); } ``` *** ## Backtraces ### Enable Backtraces ```bash # Basic backtrace export RUST_BACKTRACE=1 # Full backtrace export RUST_BACKTRACE=full ``` ### Example Backtrace ``` thread 'main' panicked at 'called `Option::unwrap()` on a `None` value' stack backtrace: 0: std::panicking::begin_panic 1: core::option::Option::unwrap 2: gateway_node::payment::process_payment at ./src/payment.rs:42 3: gateway_node::handler::handle_request at ./src/handler.rs:88 ``` *** ## Debugging Tools ### GDB/LLDB ```bash # Build with debug symbols cargo build # Run with LLDB (macOS) lldb ./target/debug/gateway_node # Run with GDB (Linux) gdb ./target/debug/gateway_node # Set breakpoint (lldb) b gateway_node::payment::process_payment (lldb) run ``` ### VS Code Debugging `.vscode/launch.json`: ```json { "version": "0.2.0", "configurations": [ { "type": "lldb", "request": "launch", "name": "Debug Gateway", "cargo": { "args": ["build", "--bin=gateway_node"] }, "args": [], "cwd": "${workspaceFolder}", "env": { "RUST_LOG": "debug", "RUST_BACKTRACE": "1" } }, { "type": "lldb", "request": "launch", "name": "Debug Tests", "cargo": { "args": ["test", "--no-run", "-p", "gateway_node"], "filter": { "kind": "test" } } } ] } ``` ### rust-analyzer Enable in VS Code for inline error checking: ```json { "rust-analyzer.checkOnSave.command": "clippy" } ``` *** ## Network Debugging ### gRPC Debugging ```bash # List services grpcurl -plaintext localhost:50051 list # Describe service grpcurl -plaintext localhost:50051 describe common_types.GatewayClientService # Call method grpcurl -plaintext \ -d '{"model_uuid": "test", "input_data": "{}"}' \ localhost:50051 common_types.GatewayClientService/SubmitInferenceTask ``` ### HTTP Debugging ```bash # Verbose curl curl -v http://localhost:8080/health # With request body curl -v -X POST http://localhost:8080/api/v1/inference \ -H "Content-Type: application/json" \ -d '{"model_uuid": "test", "input_data": "{}"}' ``` ### WebSocket Debugging ```bash # Install wscat npm install -g wscat # Connect wscat -c ws://localhost:9000 # Send message {"type": "subscribe", "request_id": "test-123"} ``` ### Network Capture ```bash # Capture gRPC traffic tcpdump -i lo -w grpc.pcap port 50051 # Analyze with Wireshark wireshark grpc.pcap ``` *** ## Component-Specific Debugging ### Gateway Debugging ```bash # Full debug logging RUST_LOG=gateway_node=trace cargo run --bin gateway_node # Payment module only RUST_LOG=gateway_node::payment=trace cargo run --bin gateway_node # Check connections curl http://localhost:8080/health ``` **Common Issues:** * Registry connection failed → Check registry is running * No compute nodes → Check compute node registration * Payment failed → Check blockchain RPC connectivity ### Compute Node Debugging ```bash # Full debug with sandbox disabled RUST_LOG=compute_node=trace \ SANDBOX_MODE=disabled \ cargo run --bin compute_node -- --gateway-addr http://127.0.0.1:50051 # Model execution only RUST_LOG=compute_node::executor=trace cargo run --bin compute_node ``` **Common Issues:** * Sandbox failed → Use `SANDBOX_MODE=disabled` * Model not found → Check `--model-path` * Gateway connection → Verify gateway is running ### Registry Debugging ```bash # Full debug logging RUST_LOG=p2p_registry_node=trace cargo run --bin p2p_registry_node # P2P networking RUST_LOG=libp2p=debug,p2p_registry_node=trace cargo run --bin p2p_registry_node ``` **Common Issues:** * No validators → Check validator registration * P2P bootstrap → Verify bootstrap node addresses * Consensus failed → Check minimum validators *** ## Memory Debugging ### Detect Memory Issues ```bash # Install valgrind (Linux) sudo apt install valgrind # Run with valgrind valgrind --leak-check=full ./target/debug/gateway_node ``` ### Heap Profiling ```bash # Install heaptrack sudo apt install heaptrack # Profile heaptrack ./target/release/gateway_node # Analyze heaptrack_gui heaptrack.gateway_node.*.gz ``` *** ## Performance Debugging ### CPU Profiling ```bash # Using perf (Linux) perf record -g ./target/release/gateway_node perf report # Using flamegraph cargo install flamegraph cargo flamegraph --bin gateway_node ``` ### Timing Analysis ```rust use std::time::Instant; fn process_request(req: &InferenceRequest) { let start = Instant::now(); // ... processing ... let duration = start.elapsed(); debug!("Request {} processed in {:?}", req.request_id, duration); } ``` *** ## Contract Debugging ### Foundry Debugging ```bash cd contracts # Verbose test output forge test -vvvv # Debug specific test forge test --debug testInitiateInferenceRequest # Gas profiling forge test --gas-report ``` ### Trace Transactions ```bash # Trace a transaction cast run --rpc-url $RPC_URL # Decode call data cast 4byte-decode ``` *** ## Log Analysis ### Search Logs ```bash # Find errors grep -i error logs/gateway.log # Find specific request grep "req-123" logs/gateway.log # Tail with filter tail -f logs/gateway.log | grep -i payment ``` ### Structured Logging Enable JSON logging for analysis: ```bash export RUST_LOG_FORMAT=json cargo run --bin gateway_node 2>&1 | jq . ``` *** ## Common Debug Scenarios ### Connection Issues ```bash # Check ports netstat -tlnp | grep -E "(50050|50051|8080)" # Check process ps aux | grep gateway # Test connectivity nc -zv localhost 50051 ``` ### Request Not Processing ```bash # Enable request tracing RUST_LOG=gateway_node::handler=trace cargo run --bin gateway_node # Check request flow curl -v -X POST http://localhost:8080/api/v1/inference \ -H "Content-Type: application/json" \ -d '{"model_uuid": "test", "input_data": "{}"}' ``` ### Signature Verification Failures ```bash # Enable crypto debugging RUST_LOG=common_types::crypto=trace cargo run --bin gateway_node # Check signature format # Ensure 0x prefix, 65 bytes (130 hex chars + 0x) ``` ### Blockchain Issues ```bash # Test RPC connection curl -X POST $RPC_URL \ -H "Content-Type: application/json" \ -d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}' # Check contract cast call $CONTRACT_ADDRESS "minValidatorSignatures()" --rpc-url $RPC_URL ``` *** ## Debug Checklist ### Before Reporting Issues * [ ] Enable debug logging (`RUST_LOG=debug`) * [ ] Enable backtraces (`RUST_BACKTRACE=1`) * [ ] Reproduce the issue consistently * [ ] Capture relevant logs * [ ] Note environment details (OS, versions) * [ ] Check for existing issues ### Information to Collect 1. Full error message and backtrace 2. Relevant log output 3. Steps to reproduce 4. Configuration (sanitized) 5. Environment details *** ## Next Steps * [Testing](/nc/neurochainai-guides/development/testing.md) - Run tests * [Troubleshooting](/nc/neurochainai-guides/troubleshooting/troubleshooting.md) - Common issues * [FAQ](/nc/neurochainai-guides/troubleshooting/faq.md) - Frequently asked questions --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://docs.neurochain.ai/nc/neurochainai-guides/development/debugging.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.