azure-skills telemetry

azure-skills-telemetry.md

Azure MCP Server - Azure-Skills Plugin Telemetry Feature

Overview

We need visibility into which skills and tools are being used across Copilot CLI, VS Code, and Claude Code to understand adoption and prioritize development. This feature adds telemetry tracking for the azure-skills plugin execution. It enables collection of plugin usage metrics while maintaining privacy through PII removal and allowlisting.

Purpose

• Track skill invocations, Azure MCP tool usage, and reference files accessed by the skills
• Maintain user privacy by removing PII from file paths
• Security-first approach with allowlist-based validation for both file references and skill names
• Track client type (copilot-cli, vscode, claude-code) for cross-platform analytics

Architecture

┌─────────────────────────────────────────────────────────────┐
│                    Agent Hook Layer                          │
│  (PostToolUse hook in Copilot CLI / VS Code / Claude Code)  │
│                                                             │
│  track-telemetry.ps1 (Copilot CLI)                          │
│  track-telemetry.sh  (Claude Code)                          │
│  track-telemetry.ps1 (VS Code)                              │
│                                                             │
│  - Receives JSON from stdin on every tool call              │
│  - Detects: skill invocations, MCP tool calls, file reads   │
│  - Strips PII from file paths                               │
│  - Calls: azmcp server plugin-telemetry [options]           │
└──────────────────────┬──────────────────────────────────────┘
                       │
                       ▼
┌─────────────────────────────────────────────────────────────┐
│              PluginTelemetryCommand                          │
│  [HiddenCommand] — not in main tool list                    │
│                                                             │
│  1. Validate required options                               │
│  2. Validate skill-name against skill allowlist             │
│     → HTTP 403 if not allowed                               │
│  3. Validate file-reference against file allowlist           │
│     → HTTP 403 if not allowed                               │
│  4. Emit OpenTelemetry activity with Plugin_* tags          │
└──────────────────────┬──────────────────────────────────────┘
                       │
                       ▼
┌─────────────────────────────────────────────────────────────┐
│              OpenTelemetry Pipeline                          │
│                                                             │
│  Activity: PluginExecuted                                   │
│  Tags: Plugin_EventType, Plugin_SessionId,                  │
│        Plugin_ClientType, Plugin_PluginName,                │
│        Plugin_PluginVersion, Plugin_SkillName,              │
│        Plugin_SkillVersion, Plugin_ToolName,                │
│        Plugin_Timestamp, Plugin_FileReference               │
└─────────────────────────────────────────────────────────────┘

Key Components

1. PluginTelemetryCommand

A hidden command ([HiddenCommand]) for programmatic telemetry publishing from agent hooks. Not exposed in the main tool list — intended for use by PostToolUse hooks only.

Invocation: azmcp server plugin-telemetry [options]

File Location: core/Microsoft.Mcp.Core/src/Areas/Server/Commands/ToolLoading/PluginTelemetryCommand.cs

Command Options

Option	Type	Required	Description
--timestamp	string	Yes	ISO 8601 timestamp of the event
--event-type	string	Yes	skill_invocation, tool_invocation, or reference_file_read
--session-id	string	Yes	Session correlation ID
--client-type	string	Yes	Client identifier: copilot-cli, vscode, or claude-code
--plugin-name	string	No	Name of the plugin (e.g., azure-skills)
--plugin-version	string	No	Version of the plugin
--skill-name	string	No	Name of the skill being invoked (validated against allowlist)
--skill-version	string	No	Version of the skill
--tool-name	string	No	Name of the Azure MCP tool being invoked
--file-reference	string	No	Relative file path from skills folder (validated against allowlist)

Event Types

• skill_invocation — When a skill is invoked (via skill tool) or its SKILL.md is read
• tool_invocation — When an Azure MCP tool is called (e.g., azure-documentation, mcp__plugin_azure_azure__deploy)
• reference_file_read — When a reference file from the skills folder is accessed

Security Features

1. Dual Allowlist Validation

Both file references and skill names are validated against embedded JSON allowlists before telemetry is emitted.

File Reference Allowlist (IPluginFileReferenceAllowlistProvider)

• Only logs file references from an approved allowlist embedded in the assembly
• Fail-closed: Returns HTTP 403 if path not in allowlist
• Case-insensitive path matching

Skill Name Allowlist (IPluginSkillNameAllowlistProvider)

• Only logs skill names from an approved allowlist embedded in the assembly
• Fail-closed: Returns HTTP 403 if skill name not in allowlist
• Case-insensitive matching

2. PII Removal

• File paths are stripped to relative paths after the skills folder
• Example: C:\Users\username\...azure\skills\azure-prepare\references\plan.md → azure-prepare\references\plan.md
• No user directory paths or system-specific information is logged

Usage Examples

Skill invocation from Copilot CLI:

azmcp server plugin-telemetry \
  --timestamp "2026-03-23T10:30:00Z" \
  --event-type "skill_invocation" \
  --session-id "f951a31d-06df-4d01-9a74-4c915b98abce" \
  --client-type "copilot-cli" \
  --plugin-name "azure-skills" \
  --plugin-version "1.0.1" \
  --skill-name "azure-prepare"

File reference read from VS Code:

azmcp server plugin-telemetry \
  --timestamp "2026-03-23T10:31:00Z" \
  --event-type "reference_file_read" \
  --session-id "f951a31d-06df-4d01-9a74-4c915b98abce" \
  --client-type "vscode" \
  --plugin-name "azure-skills" \
  --skill-name "azure-prepare" \
  --file-reference "azure-prepare\references\architecture.md"

Execution Workflow

1. PostToolUse hook fires, reads JSON from stdin
2. Hook detects trackable event, strips PII from paths
3. Hook calls: azmcp server plugin-telemetry [options]
4. Command validates skill name and file reference against allowlists (403 if rejected)
5. Emits OpenTelemetry activity with Plugin_* tags (200 on success)

Appendix: Telemetry Tags

Emitted as OpenTelemetry activity tags with Plugin_ prefix. Only non-empty fields are added.

Tag	Source Option
Plugin_EventType	--event-type
Plugin_SessionId	--session-id
Plugin_ClientType	--client-type
Plugin_PluginName	--plugin-name
Plugin_PluginVersion	--plugin-version
Plugin_SkillName	--skill-name
Plugin_SkillVersion	--skill-version
Plugin_ToolName	--tool-name
Plugin_Timestamp	--timestamp
Plugin_FileReference	--file-reference