Observability Span Naming Convention

Observability Span Naming Convention

Overview

To enable quick visual identification of which component is performing which operation in distributed traces, AgentHub follows a consistent span naming convention across all agents and services.

Naming Standard

Format

All manually-created spans MUST follow this format:

{component}.{operation}

Where:

• component: The agent or service name (lowercase, underscores for multi-word)
• operation: The operation being performed (lowercase, underscores for multi-word)

Examples

Component Names

Standard component names for AgentHub:

For new agents, use the agent’s ID or a short descriptive name.

Implementation

Creating Spans

When creating spans, use the component name as prefix:

// Good: Clear component identification
ctx, span := traceManager.StartSpan(ctx, "cortex.handle_message")
defer span.End()

// Bad: Missing component prefix
ctx, span := traceManager.StartSpan(ctx, "handle_message")  // ❌
defer span.End()

Component Attribute

In addition to the span name prefix, ALWAYS add the component attribute for filtering and querying:

ctx, span := traceManager.StartSpan(ctx, "cortex.handle_message")
defer span.End()

// Add component attribute
traceManager.AddComponentAttribute(span, "cortex")

This enables:

1. Visual identification: Span name shows component in trace waterfall
2. Query filtering: Component attribute enables filtering traces by component

A2A Message Spans

For spans specifically tracking A2A message handling, use the specialized method:

ctx, span := traceManager.StartA2AMessageSpan(
    ctx,
    "cortex.handle_message",  // Note: includes component prefix
    message.GetMessageId(),
    message.GetRole().String(),
)
defer span.End()

// Add component attribute
traceManager.AddComponentAttribute(span, "cortex")

Operation Naming Guidelines

Use Action Verbs

Operations should describe what the component is doing:

• handle_message - Processing an incoming message
• publish_event - Publishing an event
• route_event - Routing an event to subscribers
• execute_actions - Executing a list of actions
• send_response - Sending a response

Be Specific

When possible, be specific about what kind of operation:

• ✅ cortex.llm_decide (specific: LLM decision making)

• ❌ cortex.decide (too generic)

• ✅ echo_agent.handle_echo_request (specific: echo request handling)

• ❌ echo_agent.handle (too generic)

Use Underscores

Separate words with underscores, not hyphens or camelCase:

• ✅ broker.publish_event
• ❌ broker.publish-event (hyphens)
• ❌ broker.publishEvent (camelCase)

Auto-Generated Spans

Some spans are auto-generated by instrumentation libraries (e.g., gRPC):

agenthub.AgentHub/PublishMessage
agenthub.AgentHub/SubscribeToMessages
agenthub.AgentHub/RegisterAgent

Why These Are Acceptable

These auto-generated gRPC spans are acceptable and should NOT be changed because:

1. Standard Format: They follow gRPC’s OpenTelemetry standard naming convention: package.Service/Method
2. Automatic Instrumentation: Generated automatically by gRPC’s built-in OpenTelemetry interceptors
3. Breaking Changes: Modifying them would break standard gRPC tracing and break compatibility with observability tools
4. Clear Indication: The format clearly indicates these are RPC calls (the / separator is distinctive)
5. Component Context: Parent spans provide the component context

Visual Example in Traces

In practice, you’ll see this pattern:

cortex.handle_message                              ← Manual span (component prefix)
├─ cortex.llm_decide                              ← Manual span (component prefix)
├─ cortex.execute_actions                         ← Manual span (component prefix)
│  └─ cortex.send_chat_response                   ← Manual span (component prefix)
│     └─ agenthub.AgentHub/PublishMessage         ← Auto-generated gRPC span (standard)
│        └─ broker.publish_event                  ← Manual span (component prefix)
│           └─ broker.route_event                 ← Manual span (component prefix)
└─ echo_agent.handle_request                      ← Manual span (component prefix)
   └─ echo_agent.publish_response                 ← Manual span (component prefix)
      └─ agenthub.AgentHub/PublishMessage         ← Auto-generated gRPC span (standard)

Key Point: The gRPC spans (agenthub.AgentHub/*) are nested within component-prefixed spans, so the component context is always clear from the parent span.

Best Practice: Create Parent Spans

For clarity, always create a parent span with your component prefix that wraps gRPC calls:

// Good: Parent span with component prefix
ctx, span := traceManager.StartSpan(ctx, "cortex.send_chat_response")
defer span.End()
traceManager.AddComponentAttribute(span, "cortex")

// Child span will be auto-created by gRPC instrumentation
_, err := client.Client.PublishMessage(ctx, request)
// Creates child span: agenthub.AgentHub/PublishMessage

// Result in trace:
// cortex.send_chat_response (your span)
//   └─ agenthub.AgentHub/PublishMessage (gRPC auto-span)

This pattern ensures:

• Component identification at the operation level
• Standard gRPC tracing compatibility
• Clear parent-child relationships
• No modification of auto-generated spans

Benefits

Visual Clarity

In trace visualizations, you can immediately identify components:

broker.publish_event
├─ broker.route_event
│  ├─ cortex.handle_message
│  │  ├─ cortex.llm_decide
│  │  └─ cortex.execute_actions
│  │     └─ cortex.send_chat_response
│  └─ echo_agent.handle_request
│     └─ echo_agent.publish_response
└─ broker.route_event
   └─ chat_cli.display_response

Query & Filter

Filter traces by component using attributes:

# All cortex operations
component = "cortex"

# All message handling across components
span.name LIKE "%.handle_message"

# Echo agent operations only
component = "echo_agent"

Debugging

When debugging issues:

1. Look at span name to identify which component failed
2. No need to expand span details to find component
3. Quickly trace request flow across components

Migration

Existing code should be updated to follow this convention:

Before

// Inconsistent naming
ctx, span := traceManager.StartSpan(ctx, "handle_message")  // Who is handling?
ctx, span := traceManager.StartSpan(ctx, "cortex_chat_request")  // Inconsistent separator
ctx, span := traceManager.StartSpan(ctx, "cli_publish_user_message")  // Inconsistent prefix

After

// Consistent naming
ctx, span := traceManager.StartSpan(ctx, "cortex.handle_message")
ctx, span := traceManager.StartSpan(ctx, "cortex.chat_request")
ctx, span := traceManager.StartSpan(ctx, "chat_cli.publish_message")

Validation

To ensure compliance, span names should be validated:

1. Code Review: Check span names follow {component}.{operation} format
2. Testing: Verify component attribute is set on all spans
3. Trace Review: Inspect actual traces to confirm naming consistency

See Also

• Distributed Tracing Guide
• OpenTelemetry Best Practices
• Semantic Conventions