Zhentar

ETW Stack Caching

"Stack Caching" (or Stack Compression as PerfView calls it) is a feature of ETW designed to reduce trace buffer & etl file sizes by de-duplicating stack traces. Naturally, as an ETW feature it is documented solely through obtuse (likely accidental) references and hints in Microsoft tooling. And so the documentation is left to ~~stubborn reverse engineers~~ dedicated ETW enthusiasts such as myself.

The Windows version studied for this was Windows 10 1809 64-bit. I do not think this feature has changed significantly since its introduction, but I have not verified that.

Basics

In trace buffers, the compressed stacks are emitted with the Stackwalk task guid, like regular stackwalks, but with opcodes for events (as labeled by WPA) like "Stack Walk: Delete Definition" and "Stack Walk: Reference [User]". "Reference" entries contain a 'StackKey' value that uniquely identifies a stack trace definition. "Stack Walk: Delete Definition" is logged when cached stacks are evicted; from the MOF def

I have a program that parses data from both delimited files and Excel spreadsheets. I was trying out Span to speed up parsing the delimited files, but the ref struct restrictions mean I can't just hide the two different file formats behind an interface (without the small added overhead of repeatedly pulling Spans from Memory).

But what if I just wrote the ASCII strings from the Excel spreadsheets into a byte buffer, so that the same Span based parser could be used with both file formats? Seems like the overhead cost could be fairly low, and the Excel parsing is already intrinsically slower because of the decompression & XML parsing costs, so I'd be willing to take a small performance hit there for a big gain on the delimited files.

BenchmarkDotNet=v0.10.14, OS=Windows 10.0.17134
Intel Core i7-6600U CPU 2.60GHz (Skylake), 1 CPU, 4 logical and 2 physical cores
.NET Core SDK=2.1.301
[Host] : .NET Core 2.1.1 (CoreCLR 4.6.26606.02, CoreFX 4.6.26606.05), 64bit RyuJIT

There are 11 new "PatchOperation" actions we can perform to modify Defs without copying the whole thing. This in turn improves mod compatability dramatically - it allows two mods to modify different parts of the same Def without conflict. This guide only describes the basics of each operation available to us.

Patches go in xml files in a subfolder named "Patches" in your mod folder (So "MyMod\Patches", just like you might have "MyMod\Defs" or "MyMod\Textures"). Example.

Each PatchOperation has an "xpath" node, which should contain an xpath selector for the xml node(s) that the operation should affect. This is not an xpath tutorial. If you don't know xpath, not my problem. But minimurgle has you covered!

To illustrate the operations, I'll use this simple example def:

  <DesignationCategoryDef>

	//------------------------------------------------
	//--- 010 Editor v9.0.2 Binary Template
	//
	// File: ETL.bt
	// Authors: Zhentar
	// Version: 1.0
	// Purpose: Microsoft Event Tracing for Windows ETL file format
	// Category: Misc
	// File Mask: *.etl
	// History:

	using System.ComponentModel;
	using System.Runtime.InteropServices;

	public static unsafe ulong StartTraceHelper(string loggerName, string filename)
	{
	var trace_props = NewEventTraceProperties(false);
	trace_props.LogFileMode = 0x08000000; //EVENT_TRACE_INDEPENDENT_SESSION_MODE

	trace_props.BufferSize = 1024;
	for (int i = 0; i < loggerName.Length; i++)

	using BenchmarkDotNet.Attributes;
	using BenchmarkDotNet.Running;

	namespace BDNTest
	{
	public class Program
	{
	public static void Main() => BenchmarkRunner.Run<OverheadTests>();
	}

	function Set-ItemPropertyCreatePath {
	Param(
	[string]$Path,
	[string]$Name,
	[object]$Value,
	$Type
	)
	If (!(Test-Path $Path)) {
	New-Item -Path $Path -Force \| Out-Null
	}

	using System;
	using System.Runtime.InteropServices;

	namespace StartTraceExtended
	{
	static class Program
	{
	static unsafe void Main()
	{
	var trace_props = new EVENT_TRACE_PROPERTIES();

	using System;
	using System.Collections;
	using System.Collections.Generic;
	using System.Linq;
	using System.Runtime.CompilerServices;

	public abstract class BaseDictionary<TKey, TValue>
	{
	protected BaseDictionary() { }

	/* This Source Code Form is subject to the terms of the Mozilla Public
	* License, v. 2.0. If a copy of the MPL was not distributed with this
	* file, You can obtain one at http://mozilla.org/MPL/2.0/.
	*
	* The Initial Developer of the Original Code is Rune Skovbo Johansen.
	* Portions created by the Initial Developer are Copyright (C) 2015
	* the Initial Developer. All Rights Reserved.
	*/

	using System;