Aranya Observability Documentation

This section covers observability and debugging specifications for Aranya.

Overview

The Aranya observability system targets remote debugging of deployed systems rather than real-time monitoring. It focuses on diagnosing production issues when direct access is limited.

Documentation Structure

Core Concepts

• Logging Configuration - Log levels, filters, output configuration for daemon and clients

Component-Specific Observability

• Sync Observability - Debugging sync operations, performance, stalls, and topology
• AFC Observability - Fast Channels logging, SHM operations, seal/open debugging
• Policy Observability - Policy evaluation logging, authorization failures, role debugging

Implementation

• Implementation Roadmap - Phased rollout plan

Quick Start

Add logging to daemon code:

info!(
    duration_ms = start.elapsed().as_millis(),
    cmd_count,
    effects_count,
    "Sync completed successfully"
);    

Configure daemon log level:

# daemon.toml
log_filter = "info,aranya_daemon::sync=debug"

Initialize logging in your application:

use tracing_subscriber::{prelude::*, EnvFilter};

fn main() {
    tracing_subscriber::registry()
        .with(
            tracing_subscriber::fmt::layer()
                .with_filter(EnvFilter::from_env("MY_APP_LOG")),
        )
        .init();
    
    // Client library logs are now captured
    let client = aranya_client::Client::connect(...).await?;
}

Enable AFC debugging:

APP_LOG=aranya_client::afc=debug ./application

Design Principles

Principles:

1. Remote-First: Designed for production environments with limited access
2. File-Based: Write locally, collect and analyze offline
3. Runtime Configurable: Enable/disable without recompilation
4. Performance Conscious: Avoid hot-path overhead; use TRACE sparingly
5. Sync-Focused: Most production issues are sync-related, so prioritize sync debugging
6. Structured Output: JSON for machine parsing, text for human reading

Security Considerations

• Logs may contain sensitive data
• Debug endpoints should be disabled in production or protected by authentication
• Avoid logging cryptographic material (keys, passphrases)

Contributing

When adding observability features:

1. Follow structured logging guidelines in Logging Configuration
2. Use appropriate log levels (ERROR/WARN/INFO/DEBUG/TRACE)
3. Include device_id/team_id context where relevant
4. Add timing information for operations
5. Update relevant documentation in this directory