Expensive `memory.stat` + `cpu.stat`

README.md

Expensive memory.stat + cpu.stat

We're seeing issues where cadvisor spends a lot of time on CPU.

Repro

The following should run on any system with systemd (code is in this gist):

go run main.go

The output for is is something like this:

completed:  4.32s [manual / mem-stat + cpu-stat]
completed:  4.25s [manual / cpu-stat + mem-stat]
completed:  0.47s [manual / mem-stat]
completed:  0.04s [manual / cpu-stat]
completed:  5.17s [manual / mem-stat + cpu-stat]
completed:  5.59s [manual / cpu-stat + mem-stat]
completed:  0.52s [manual / mem-stat]
completed:  0.04s [manual / cpu-stat]

The first three runs are a warm-up, and the second three runs is the meat and potatoes.

In cpu-stat + mem-stat we do effectively the following:

for _ in $(seq 1 1000); do cat /sys/fs/cgroup/system.slice/memory.stat /sys/fs/cgroup/system.slice/cpu.stat > /dev/null

In cpu-stat it's just cpu.stat file:

for _ in $(seq 1 1000); do cat /sys/fs/cgroup/system.slice/cpu.stat > /dev/null; done

In memory.stat it's just memory.stat file:

for _ in $(seq 1 1000); do cat /sys/fs/cgroup/system.slice/memory.stat > /dev/null; done

The weird thing is: combining the latter two is much faster than the former and there's easily 10x difference in the time spent in kernel between them.

Unfortunately, the former is how monitoring software normally works: all metrics are fetched for one service before proceeding to the next one. One prominent example that prompted this examination is cAdvisor:

• https://github.com/google/cadvisor

Flamegraphs for a single combined loop (one-loop.svg) and for two loops back to back (two-loops.svg) are attached to this gist.

This doesn't happen on a mostly idle machine, but we do see this on loaded servers (128 busy logical CPUs, lots of memory churn).

main.go

package main

import (
	"fmt"
	"os"
	"time"
)

var suffixes = map[string][]string{
	"mem-all+cpu-stat":  []string{"memory.stat", "memory.current", "memory.max", "memory.swap.current", "memory.swap.max", "cpu.stat"},
	"mem-stat+cpu-stat": []string{"memory.stat", "cpu.stat"},
	"cpu-stat+mem-stat": []string{"cpu.stat", "memory.stat"},
	"mem-all":           []string{"memory.stat", "memory.current", "memory.max", "memory.swap.current", "memory.swap.max"},
	"mem-stat":          []string{"memory.stat"},
	"cpu-stat":          []string{"cpu.stat"},
}

func main() {
	onePathRepeated := make([]string, 1000)
	for i := range onePathRepeated {
		onePathRepeated[i] = "system.slice"
	}

	// Warm up round
	runBenchmarks(onePathRepeated)

	// Real run
	runBenchmarks(onePathRepeated)
}

func runBenchmarks(paths []string) {
	// benchmarkFunc("manual / mem-all + cpu-stat", func() {
	// 	manualBenchmark(paths, suffixes["mem-all+cpu-stat"])
	// })

	benchmarkFunc("manual / mem-stat + cpu-stat", func() {
		manualBenchmark(paths, suffixes["mem-stat+cpu-stat"])
	})

	benchmarkFunc("manual / cpu-stat + mem-stat", func() {
		manualBenchmark(paths, suffixes["cpu-stat+mem-stat"])
	})

	// benchmarkFunc("manual / mem-all", func() {
	// 	manualBenchmark(paths, suffixes["mem-all"])
	// })

	benchmarkFunc("manual / mem-stat", func() {
		manualBenchmark(paths, suffixes["mem-stat"])
	})

	benchmarkFunc("manual / cpu-stat", func() {
		manualBenchmark(paths, suffixes["cpu-stat"])
	})
}

func benchmarkFunc(name string, bench func()) {
	started := time.Now()

	bench()

	fmt.Printf("completed: %5.2fs [%s]\n", time.Since(started).Seconds(), name)
}

func manualBenchmark(paths []string, suffixes []string) {
	for _, path := range paths {
		_, err := getStatsManual("/sys/fs/cgroup/"+path, suffixes)
		if err != nil {
			fmt.Printf("Error getting stats: %v\n", err)
		}
	}
}

func getStatsManual(path string, suffixes []string) ([][]byte, error) {
	results := [][]byte{}

	for _, suffix := range suffixes {
		data, err := os.ReadFile(path + "/" + suffix)
		if err != nil {
			return nil, err
		}

		results = append(results, data)
	}

	return results, nil
}

one-loop.svg

two-loops.svg

<?xml version="1.0" standalone="no"?><!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd"><svg version="1.1" width="1440" height="646" onload="init(evt)" viewBox="0 0 1440 646" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:fg="http://github.com/jonhoo/inferno"><!--Flame graph stack visualization. See https://github.com/brendangregg/FlameGraph for latest version, and http://www.brendangregg.com/flamegraphs.html for examples.--><!--NOTES: --><defs><linearGradient id="background" y1="0" y2="1" x1="0" x2="0"><stop stop-color="#eeeeee" offset="5%"/><stop stop-color="#eeeeb0" offset="95%"/></linearGradient></defs><style type="text/css">
text { font-family:"Verdana"; font-size:12px; fill:rgb(0,0,0); }
#title { text-anchor:middle; font-size:17px; }
#search { opacity:0.1; cursor:pointer; }
#search:hover, #search.show { opacity:1; }
#subtitle { text-anchor:middle; font-color:rgb(160,160,160); }
#unzoom { cursor:pointer; }
#frames > *:hover { stroke:black; stroke-width:0.5; cursor:pointer; }
.hide { display:none; }
.parent { opacity:0.5; }
</style><script type="text/ecmascript"><![CDATA[
        var nametype = 'Function:';
        var fontsize = 12;
        var fontwidth = 0.59;
        var xpad = 10;
        var inverted = false;
        var searchcolor = 'rgb(230,0,230)';
        var fluiddrawing = false;
        var truncate_text_right = true;
    ]]><![CDATA["use strict";
var details, searchbtn, unzoombtn, matchedtxt, svg, searching, frames;
function init(evt) {
    details = document.getElementById("details").firstChild;
    searchbtn = document.getElementById("search");
    unzoombtn = document.getElementById("unzoom");
    matchedtxt = document.getElementById("matched");
    svg = document.getElementsByTagName("svg")[0];
    frames = document.getElementById("frames");
    total_samples = parseInt(frames.attributes.total_samples.value);
    searching = 0;

    // Use GET parameters to restore a flamegraph's state.
    var restore_state = function() {
        var params = get_params();
        if (params.x && params.y)
            zoom(find_group(document.querySelector('[*|x="' + params.x + '"][y="' + params.y + '"]')));
        if (params.s)
            search(params.s);
    };

    if (fluiddrawing) {
        // Make width dynamic so the SVG fits its parent's width.
        svg.removeAttribute("width");
        // Edge requires us to have a viewBox that gets updated with size changes.
        var isEdge = /Edge\/\d./i.test(navigator.userAgent);
        if (!isEdge) {
          svg.removeAttribute("viewBox");
        }
        var update_for_width_change = function() {
            if (isEdge) {
                svg.attributes.viewBox.value = "0 0 " + svg.width.baseVal.value + " " + svg.height.baseVal.value;
            }

            // Keep consistent padding on left and right of frames container.
            frames.attributes.width.value = svg.width.baseVal.value - xpad * 2;

            // Text truncation needs to be adjusted for the current width.
            var el = frames.children;
            for(var i = 0; i < el.length; i++) {
                update_text(el[i]);
            }

            // Keep search elements at a fixed distance from right edge.
            var svgWidth = svg.width.baseVal.value;
            searchbtn.attributes.x.value = svgWidth - xpad - 100;
            matchedtxt.attributes.x.value = svgWidth - xpad - 100;
        };
        window.addEventListener('resize', function() {
            update_for_width_change();
        });
        // This needs to be done asynchronously for Safari to work.
        setTimeout(function() {
            unzoom();
            update_for_width_change();
            restore_state();
        }, 0);
    } else {
        restore_state();
    }
}
// event listeners
window.addEventListener("click", function(e) {
    var target = find_group(e.target);
    if (target) {
        if (target.nodeName == "a") {
            if (e.ctrlKey === false) return;
            e.preventDefault();
        }
        if (target.classList.contains("parent")) unzoom();
        zoom(target);

        // set parameters for zoom state
        var el = target.querySelector("rect");
        if (el && el.attributes && el.attributes.y && el.attributes["fg:x"]) {
            var params = get_params()
            params.x = el.attributes["fg:x"].value;
            params.y = el.attributes.y.value;
            history.replaceState(null, null, parse_params(params));
        }
    }
    else if (e.target.id == "unzoom") {
        unzoom();

        // remove zoom state
        var params = get_params();
        if (params.x) delete params.x;
        if (params.y) delete params.y;
        history.replaceState(null, null, parse_params(params));
    }
    else if (e.target.id == "search") search_prompt();
}, false)
// mouse-over for info
// show
window.addEventListener("mouseover", function(e) {
    var target = find_group(e.target);
    if (target) details.nodeValue = nametype + " " + g_to_text(target);
}, false)
// clear
window.addEventListener("mouseout", function(e) {
    var target = find_group(e.target);
    if (target) details.nodeValue = ' ';
}, false)
// ctrl-F for search
window.addEventListener("keydown",function (e) {
    if (e.keyCode === 114 || (e.ctrlKey && e.keyCode === 70)) {
        e.preventDefault();
        search_prompt();
    }
}, false)
// functions
function get_params() {
    var params = {};
    var paramsarr = window.location.search.substr(1).split('&');
    for (var i = 0; i < paramsarr.length; ++i) {
        var tmp = paramsarr[i].split("=");
        if (!tmp[0] || !tmp[1]) continue;
        params[tmp[0]]  = decodeURIComponent(tmp[1]);
    }
    return params;
}
function parse_params(params) {
    var uri = "?";
    for (var key in params) {
        uri += key + '=' + encodeURIComponent(params[key]) + '&';
    }
    if (uri.slice(-1) == "&")
        uri = uri.substring(0, uri.length - 1);
    if (uri == '?')
        uri = window.location.href.split('?')[0];
    return uri;
}
function find_child(node, selector) {
    var children = node.querySelectorAll(selector);
    if (children.length) return children[0];
    return;
}
function find_group(node) {
    var parent = node.parentElement;
    if (!parent) return;
    if (parent.id == "frames") return node;
    return find_group(parent);
}
function orig_save(e, attr, val) {
    if (e.attributes["fg:orig_" + attr] != undefined) return;
    if (e.attributes[attr] == undefined) return;
    if (val == undefined) val = e.attributes[attr].value;
    e.setAttribute("fg:orig_" + attr, val);
}
function orig_load(e, attr) {
    if (e.attributes["fg:orig_"+attr] == undefined) return;
    e.attributes[attr].value = e.attributes["fg:orig_" + attr].value;
    e.removeAttribute("fg:orig_" + attr);
}
function g_to_text(e) {
    var text = find_child(e, "title").firstChild.nodeValue;
    return (text)
}
function g_to_func(e) {
    var func = g_to_text(e);
    // if there's any manipulation we want to do to the function
    // name before it's searched, do it here before returning.
    return (func);
}
function update_text(e) {
    var r = find_child(e, "rect");
    var t = find_child(e, "text");
    var w = parseFloat(r.attributes.width.value) * frames.attributes.width.value / 100 - 3;
    var txt = find_child(e, "title").textContent.replace(/\([^(]*\)$/,"");
    t.attributes.x.value = format_percent((parseFloat(r.attributes.x.value) + (100 * 3 / frames.attributes.width.value)));
    // Smaller than this size won't fit anything
    if (w < 2 * fontsize * fontwidth) {
        t.textContent = "";
        return;
    }
    t.textContent = txt;
    // Fit in full text width
    if (/^ *\$/.test(txt) || t.getComputedTextLength() < w)
        return;
    if (truncate_text_right) {
        // Truncate the right side of the text.
        for (var x = txt.length - 2; x > 0; x--) {
            if (t.getSubStringLength(0, x + 2) <= w) {
                t.textContent = txt.substring(0, x) + "..";
                return;
            }
        }
    } else {
        // Truncate the left side of the text.
        for (var x = 2; x < txt.length; x++) {
            if (t.getSubStringLength(x - 2, txt.length) <= w) {
                t.textContent = ".." + txt.substring(x, txt.length);
                return;
            }
        }
    }
    t.textContent = "";
}
// zoom
function zoom_reset(e) {
    if (e.tagName == "rect") {
        e.attributes.x.value = format_percent(100 * parseInt(e.attributes["fg:x"].value) / total_samples);
        e.attributes.width.value = format_percent(100 * parseInt(e.attributes["fg:w"].value) / total_samples);
    }
    if (e.childNodes == undefined) return;
    for(var i = 0, c = e.childNodes; i < c.length; i++) {
        zoom_reset(c[i]);
    }
}
function zoom_child(e, x, zoomed_width_samples) {
    if (e.tagName == "text") {
        var parent_x = parseFloat(find_child(e.parentNode, "rect[x]").attributes.x.value);
        e.attributes.x.value = format_percent(parent_x + (100 * 3 / frames.attributes.width.value));
    } else if (e.tagName == "rect") {
        e.attributes.x.value = format_percent(100 * (parseInt(e.attributes["fg:x"].value) - x) / zoomed_width_samples);
        e.attributes.width.value = format_percent(100 * parseInt(e.attributes["fg:w"].value) / zoomed_width_samples);
    }
    if (e.childNodes == undefined) return;
    for(var i = 0, c = e.childNodes; i < c.length; i++) {
        zoom_child(c[i], x, zoomed_width_samples);
    }
}
function zoom_parent(e) {
    if (e.attributes) {
        if (e.attributes.x != undefined) {
            e.attributes.x.value = "0.0%";
        }
        if (e.attributes.width != undefined) {
            e.attributes.width.value = "100.0%";
        }
    }
    if (e.childNodes == undefined) return;
    for(var i = 0, c = e.childNodes; i < c.length; i++) {
        zoom_parent(c[i]);
    }
}
function zoom(node) {
    var attr = find_child(node, "rect").attributes;
    var width = parseInt(attr["fg:w"].value);
    var xmin = parseInt(attr["fg:x"].value);
    var xmax = xmin + width;
    var ymin = parseFloat(attr.y.value);
    unzoombtn.classList.remove("hide");
    var el = frames.children;
    for (var i = 0; i < el.length; i++) {
        var e = el[i];
        var a = find_child(e, "rect").attributes;
        var ex = parseInt(a["fg:x"].value);
        var ew = parseInt(a["fg:w"].value);
        // Is it an ancestor
        if (!inverted) {
            var upstack = parseFloat(a.y.value) > ymin;
        } else {
            var upstack = parseFloat(a.y.value) < ymin;
        }
        if (upstack) {
            // Direct ancestor
            if (ex <= xmin && (ex+ew) >= xmax) {
                e.classList.add("parent");
                zoom_parent(e);
                update_text(e);
            }
            // not in current path
            else
                e.classList.add("hide");
        }
        // Children maybe
        else {
            // no common path
            if (ex < xmin || ex >= xmax) {
                e.classList.add("hide");
            }
            else {
                zoom_child(e, xmin, width);
                update_text(e);
            }
        }
    }
}
function unzoom() {
    unzoombtn.classList.add("hide");
    var el = frames.children;
    for(var i = 0; i < el.length; i++) {
        el[i].classList.remove("parent");
        el[i].classList.remove("hide");
        zoom_reset(el[i]);
        update_text(el[i]);
    }
}
// search
function reset_search() {
    var el = document.querySelectorAll("#frames rect");
    for (var i = 0; i < el.length; i++) {
        orig_load(el[i], "fill")
    }
    var params = get_params();
    delete params.s;
    history.replaceState(null, null, parse_params(params));
}
function search_prompt() {
    if (!searching) {
        var term = prompt("Enter a search term (regexp " +
            "allowed, eg: ^ext4_)", "");
        if (term != null) {
            search(term)
        }
    } else {
        reset_search();
        searching = 0;
        searchbtn.classList.remove("show");
        searchbtn.firstChild.nodeValue = "Search"
        matchedtxt.classList.add("hide");
        matchedtxt.firstChild.nodeValue = ""
    }
}
function search(term) {
    var re = new RegExp(term);
    var el = frames.children;
    var matches = new Object();
    var maxwidth = 0;
    for (var i = 0; i < el.length; i++) {
        var e = el[i];
        // Skip over frames which are either not visible, or below the zoomed-to frame
        if (e.classList.contains("hide") || e.classList.contains("parent")) {
            continue;
        }
        var func = g_to_func(e);
        var rect = find_child(e, "rect");
        if (func == null || rect == null)
            continue;
        // Save max width. Only works as we have a root frame
        var w = parseInt(rect.attributes["fg:w"].value);
        if (w > maxwidth)
            maxwidth = w;
        if (func.match(re)) {
            // highlight
            var x = parseInt(rect.attributes["fg:x"].value);
            orig_save(rect, "fill");
            rect.attributes.fill.value = searchcolor;
            // remember matches
            if (matches[x] == undefined) {
                matches[x] = w;
            } else {
                if (w > matches[x]) {
                    // overwrite with parent
                    matches[x] = w;
                }
            }
            searching = 1;
        }
    }
    if (!searching)
        return;
    var params = get_params();
    params.s = term;
    history.replaceState(null, null, parse_params(params));

    searchbtn.classList.add("show");
    searchbtn.firstChild.nodeValue = "Reset Search";
    // calculate percent matched, excluding vertical overlap
    var count = 0;
    var lastx = -1;
    var lastw = 0;
    var keys = Array();
    for (k in matches) {
        if (matches.hasOwnProperty(k))
            keys.push(k);
    }
    // sort the matched frames by their x location
    // ascending, then width descending
    keys.sort(function(a, b){
        return a - b;
    });
    // Step through frames saving only the biggest bottom-up frames
    // thanks to the sort order. This relies on the tree property
    // where children are always smaller than their parents.
    for (var k in keys) {
        var x = parseInt(keys[k]);
        var w = matches[keys[k]];
        if (x >= lastx + lastw) {
            count += w;
            lastx = x;
            lastw = w;
        }
    }
    // display matched percent
    matchedtxt.classList.remove("hide");
    var pct = 100 * count / maxwidth;
    if (pct != 100) pct = pct.toFixed(1);
    matchedtxt.firstChild.nodeValue = "Matched: " + pct + "%";
}
function format_percent(n) {
    return n.toFixed(4) + "%";
}
]]></script><rect x="0" y="0" width="100%" height="646" fill="url(#background)"/><text id="title" x="50.0000%" y="24.00">39m139</text><text id="details" x="10" y="629.00"> </text><text id="unzoom" class="hide" x="10" y="24.00">Reset Zoom</text><text id="search" x="1330" y="24.00">Search</text><text id="matched" x="1330" y="629.00"> </text><svg id="frames" x="10" width="1420" total_samples="202"><g><title>internal/poll.runtime_pollReset (1 samples, 0.50%)</title><rect x="0.0000%" y="389" width="0.4950%" height="15" fill="rgb(92,239,92)" fg:x="0" fg:w="1"/><text x="0.2083%" y="399.50"></text></g><g><title>_copy_to_iter (1 samples, 0.50%)</title><rect x="9.4059%" y="261" width="0.4950%" height="15" fill="rgb(240,140,0)" fg:x="19" fg:w="1"/><text x="9.6143%" y="271.50"></text></g><g><title>copy_user_generic_string (1 samples, 0.50%)</title><rect x="9.4059%" y="245" width="0.4950%" height="15" fill="rgb(229,129,0)" fg:x="19" fg:w="1"/><text x="9.6143%" y="255.50"></text></g><g><title>blkcg_rstat_flush (29 samples, 14.36%)</title><rect x="11.8812%" y="181" width="14.3564%" height="15" fill="rgb(214,114,0)" fg:x="24" fg:w="29"/><text x="12.0895%" y="191.50">blkcg_rstat_flush</text></g><g><title>blkcg_iostat_update (2 samples, 0.99%)</title><rect x="25.2475%" y="165" width="0.9901%" height="15" fill="rgb(214,114,0)" fg:x="51" fg:w="2"/><text x="25.4559%" y="175.50"></text></g><g><title>__mem_cgroup_flush_stats (114 samples, 56.44%)</title><rect x="9.9010%" y="229" width="56.4356%" height="15" fill="rgb(232,132,0)" fg:x="20" fg:w="114"/><text x="10.1093%" y="239.50">__mem_cgroup_flush_stats</text></g><g><title>cgroup_rstat_flush_irqsafe (114 samples, 56.44%)</title><rect x="9.9010%" y="213" width="56.4356%" height="15" fill="rgb(223,123,0)" fg:x="20" fg:w="114"/><text x="10.1093%" y="223.50">cgroup_rstat_flush_irqsafe</text></g><g><title>cgroup_rstat_flush_locked (114 samples, 56.44%)</title><rect x="9.9010%" y="197" width="56.4356%" height="15" fill="rgb(223,123,0)" fg:x="20" fg:w="114"/><text x="10.1093%" y="207.50">cgroup_rstat_flush_locked</text></g><g><title>mem_cgroup_css_rstat_flush (81 samples, 40.10%)</title><rect x="26.2376%" y="181" width="40.0990%" height="15" fill="rgb(230,130,0)" fg:x="53" fg:w="81"/><text x="26.4460%" y="191.50">mem_cgroup_css_rstat_flush</text></g><g><title>_find_next_bit (3 samples, 1.49%)</title><rect x="64.8515%" y="165" width="1.4851%" height="15" fill="rgb(231,131,0)" fg:x="131" fg:w="3"/><text x="65.0598%" y="175.50">_..</text></g><g><title>memory_stat_format.constprop.0 (115 samples, 56.93%)</title><rect x="9.9010%" y="245" width="56.9307%" height="15" fill="rgb(230,130,0)" fg:x="20" fg:w="115"/><text x="10.1093%" y="255.50">memory_stat_format.constprop.0</text></g><g><title>_raw_spin_unlock_irqrestore (1 samples, 0.50%)</title><rect x="66.3366%" y="229" width="0.4950%" height="15" fill="rgb(237,137,0)" fg:x="134" fg:w="1"/><text x="66.5450%" y="239.50"></text></g><g><title>asm_sysvec_apic_timer_interrupt (1 samples, 0.50%)</title><rect x="66.3366%" y="213" width="0.4950%" height="15" fill="rgb(231,131,0)" fg:x="134" fg:w="1"/><text x="66.5450%" y="223.50"></text></g><g><title>sysvec_apic_timer_interrupt (1 samples, 0.50%)</title><rect x="66.3366%" y="197" width="0.4950%" height="15" fill="rgb(237,137,0)" fg:x="134" fg:w="1"/><text x="66.5450%" y="207.50"></text></g><g><title>__sysvec_apic_timer_interrupt (1 samples, 0.50%)</title><rect x="66.3366%" y="181" width="0.4950%" height="15" fill="rgb(227,127,0)" fg:x="134" fg:w="1"/><text x="66.5450%" y="191.50"></text></g><g><title>hrtimer_interrupt (1 samples, 0.50%)</title><rect x="66.3366%" y="165" width="0.4950%" height="15" fill="rgb(232,132,0)" fg:x="134" fg:w="1"/><text x="66.5450%" y="175.50"></text></g><g><title>__hrtimer_run_queues (1 samples, 0.50%)</title><rect x="66.3366%" y="149" width="0.4950%" height="15" fill="rgb(226,126,0)" fg:x="134" fg:w="1"/><text x="66.5450%" y="159.50"></text></g><g><title>enqueue_hrtimer (1 samples, 0.50%)</title><rect x="66.3366%" y="133" width="0.4950%" height="15" fill="rgb(248,148,0)" fg:x="134" fg:w="1"/><text x="66.5450%" y="143.50"></text></g><g><title>timerqueue_add (1 samples, 0.50%)</title><rect x="66.3366%" y="117" width="0.4950%" height="15" fill="rgb(231,131,0)" fg:x="134" fg:w="1"/><text x="66.5450%" y="127.50"></text></g><g><title>ksys_read (117 samples, 57.92%)</title><rect x="9.4059%" y="309" width="57.9208%" height="15" fill="rgb(231,131,0)" fg:x="19" fg:w="117"/><text x="9.6143%" y="319.50">ksys_read</text></g><g><title>vfs_read (117 samples, 57.92%)</title><rect x="9.4059%" y="293" width="57.9208%" height="15" fill="rgb(226,126,0)" fg:x="19" fg:w="117"/><text x="9.6143%" y="303.50">vfs_read</text></g><g><title>seq_read_iter (117 samples, 57.92%)</title><rect x="9.4059%" y="277" width="57.9208%" height="15" fill="rgb(236,136,0)" fg:x="19" fg:w="117"/><text x="9.6143%" y="287.50">seq_read_iter</text></g><g><title>memory_stat_show (116 samples, 57.43%)</title><rect x="9.9010%" y="261" width="57.4257%" height="15" fill="rgb(230,130,0)" fg:x="20" fg:w="116"/><text x="10.1093%" y="271.50">memory_stat_show</text></g><g><title>seq_puts (1 samples, 0.50%)</title><rect x="66.8317%" y="245" width="0.4950%" height="15" fill="rgb(236,136,0)" fg:x="135" fg:w="1"/><text x="67.0400%" y="255.50"></text></g><g><title>strlen (1 samples, 0.50%)</title><rect x="66.8317%" y="229" width="0.4950%" height="15" fill="rgb(229,129,0)" fg:x="135" fg:w="1"/><text x="67.0400%" y="239.50"></text></g><g><title>fpregs_restore_userregs (12 samples, 5.94%)</title><rect x="67.3267%" y="277" width="5.9406%" height="15" fill="rgb(242,142,0)" fg:x="136" fg:w="12"/><text x="67.5351%" y="287.50">fpregs_res..</text></g><g><title>restore_fpregs_from_fpstate (12 samples, 5.94%)</title><rect x="67.3267%" y="261" width="5.9406%" height="15" fill="rgb(236,136,0)" fg:x="136" fg:w="12"/><text x="67.5351%" y="271.50">restore_fp..</text></g><g><title>os.(*File).Read (154 samples, 76.24%)</title><rect x="0.0000%" y="421" width="76.2376%" height="15" fill="rgb(92,239,92)" fg:x="0" fg:w="154"/><text x="0.2083%" y="431.50">os.(*File).Read</text></g><g><title>internal/poll.(*FD).Read (154 samples, 76.24%)</title><rect x="0.0000%" y="405" width="76.2376%" height="15" fill="rgb(92,239,92)" fg:x="0" fg:w="154"/><text x="0.2083%" y="415.50">internal/poll.(*FD).Read</text></g><g><title>syscall.read (153 samples, 75.74%)</title><rect x="0.4950%" y="389" width="75.7426%" height="15" fill="rgb(93,240,93)" fg:x="1" fg:w="153"/><text x="0.7034%" y="399.50">syscall.read</text></g><g><title>syscall.Syscall (153 samples, 75.74%)</title><rect x="0.4950%" y="373" width="75.7426%" height="15" fill="rgb(93,240,93)" fg:x="1" fg:w="153"/><text x="0.7034%" y="383.50">syscall.Syscall</text></g><g><title>runtime/internal/syscall.Syscall6 (153 samples, 75.74%)</title><rect x="0.4950%" y="357" width="75.7426%" height="15" fill="rgb(90,237,90)" fg:x="1" fg:w="153"/><text x="0.7034%" y="367.50">runtime/internal/syscall.Syscall6</text></g><g><title>entry_SYSCALL_64_after_hwframe (135 samples, 66.83%)</title><rect x="9.4059%" y="341" width="66.8317%" height="15" fill="rgb(245,145,0)" fg:x="19" fg:w="135"/><text x="9.6143%" y="351.50">entry_SYSCALL_64_after_hwframe</text></g><g><title>do_syscall_64 (135 samples, 66.83%)</title><rect x="9.4059%" y="325" width="66.8317%" height="15" fill="rgb(243,143,0)" fg:x="19" fg:w="135"/><text x="9.6143%" y="335.50">do_syscall_64</text></g><g><title>syscall_exit_to_user_mode (18 samples, 8.91%)</title><rect x="67.3267%" y="309" width="8.9109%" height="15" fill="rgb(237,137,0)" fg:x="136" fg:w="18"/><text x="67.5351%" y="319.50">syscall_exit_to_..</text></g><g><title>exit_to_user_mode_prepare (18 samples, 8.91%)</title><rect x="67.3267%" y="293" width="8.9109%" height="15" fill="rgb(230,130,0)" fg:x="136" fg:w="18"/><text x="67.5351%" y="303.50">exit_to_user_mod..</text></g><g><title>schedule (6 samples, 2.97%)</title><rect x="73.2673%" y="277" width="2.9703%" height="15" fill="rgb(236,136,0)" fg:x="148" fg:w="6"/><text x="73.4757%" y="287.50">sche..</text></g><g><title>__schedule (6 samples, 2.97%)</title><rect x="73.2673%" y="261" width="2.9703%" height="15" fill="rgb(227,127,0)" fg:x="148" fg:w="6"/><text x="73.4757%" y="271.50">__sc..</text></g><g><title>finish_task_switch.isra.0 (6 samples, 2.97%)</title><rect x="73.2673%" y="245" width="2.9703%" height="15" fill="rgb(240,140,0)" fg:x="148" fg:w="6"/><text x="73.4757%" y="255.50">fini..</text></g><g><title>asm_common_interrupt (6 samples, 2.97%)</title><rect x="73.2673%" y="229" width="2.9703%" height="15" fill="rgb(231,131,0)" fg:x="148" fg:w="6"/><text x="73.4757%" y="239.50">asm_..</text></g><g><title>common_interrupt (6 samples, 2.97%)</title><rect x="73.2673%" y="213" width="2.9703%" height="15" fill="rgb(232,132,0)" fg:x="148" fg:w="6"/><text x="73.4757%" y="223.50">comm..</text></g><g><title>__common_interrupt (6 samples, 2.97%)</title><rect x="73.2673%" y="197" width="2.9703%" height="15" fill="rgb(230,130,0)" fg:x="148" fg:w="6"/><text x="73.4757%" y="207.50">__co..</text></g><g><title>handle_edge_irq (6 samples, 2.97%)</title><rect x="73.2673%" y="181" width="2.9703%" height="15" fill="rgb(231,131,0)" fg:x="148" fg:w="6"/><text x="73.4757%" y="191.50">hand..</text></g><g><title>runtime.systemstack.abi0 (1 samples, 0.50%)</title><rect x="76.2376%" y="373" width="0.4950%" height="15" fill="rgb(90,237,90)" fg:x="154" fg:w="1"/><text x="76.4460%" y="383.50"></text></g><g><title>runtime.SetFinalizer.func2 (1 samples, 0.50%)</title><rect x="76.2376%" y="357" width="0.4950%" height="15" fill="rgb(90,237,90)" fg:x="154" fg:w="1"/><text x="76.4460%" y="367.50"></text></g><g><title>runtime.addfinalizer (1 samples, 0.50%)</title><rect x="76.2376%" y="341" width="0.4950%" height="15" fill="rgb(90,237,90)" fg:x="154" fg:w="1"/><text x="76.4460%" y="351.50"></text></g><g><title>runtime.addspecial (1 samples, 0.50%)</title><rect x="76.2376%" y="325" width="0.4950%" height="15" fill="rgb(90,237,90)" fg:x="154" fg:w="1"/><text x="76.4460%" y="335.50"></text></g><g><title>runtime.lock2 (1 samples, 0.50%)</title><rect x="76.2376%" y="309" width="0.4950%" height="15" fill="rgb(90,237,90)" fg:x="154" fg:w="1"/><text x="76.4460%" y="319.50"></text></g><g><title>os.newFile (2 samples, 0.99%)</title><rect x="76.2376%" y="389" width="0.9901%" height="15" fill="rgb(92,239,92)" fg:x="154" fg:w="2"/><text x="76.4460%" y="399.50"></text></g><g><title>syscall.SetNonblock (1 samples, 0.50%)</title><rect x="76.7327%" y="373" width="0.4950%" height="15" fill="rgb(93,240,93)" fg:x="155" fg:w="1"/><text x="76.9410%" y="383.50"></text></g><g><title>syscall.fcntl (1 samples, 0.50%)</title><rect x="76.7327%" y="357" width="0.4950%" height="15" fill="rgb(93,240,93)" fg:x="155" fg:w="1"/><text x="76.9410%" y="367.50"></text></g><g><title>syscall.Syscall (1 samples, 0.50%)</title><rect x="76.7327%" y="341" width="0.4950%" height="15" fill="rgb(93,240,93)" fg:x="155" fg:w="1"/><text x="76.9410%" y="351.50"></text></g><g><title>runtime.entersyscall (1 samples, 0.50%)</title><rect x="76.7327%" y="325" width="0.4950%" height="15" fill="rgb(90,237,90)" fg:x="155" fg:w="1"/><text x="76.9410%" y="335.50"></text></g><g><title>runtime.reentersyscall (1 samples, 0.50%)</title><rect x="76.7327%" y="309" width="0.4950%" height="15" fill="rgb(90,237,90)" fg:x="155" fg:w="1"/><text x="76.9410%" y="319.50"></text></g><g><title>runtime.casgstatus (1 samples, 0.50%)</title><rect x="76.7327%" y="293" width="0.4950%" height="15" fill="rgb(90,237,90)" fg:x="155" fg:w="1"/><text x="76.9410%" y="303.50"></text></g><g><title>[unknown] (1 samples, 0.50%)</title><rect x="77.2277%" y="229" width="0.4950%" height="15" fill="rgb(243,112,112)" fg:x="156" fg:w="1"/><text x="77.4361%" y="239.50"></text></g><g><title>kprobe_ftrace_handler (1 samples, 0.50%)</title><rect x="77.2277%" y="213" width="0.4950%" height="15" fill="rgb(230,130,0)" fg:x="156" fg:w="1"/><text x="77.4361%" y="223.50"></text></g><g><title>aggr_pre_handler (1 samples, 0.50%)</title><rect x="77.2277%" y="197" width="0.4950%" height="15" fill="rgb(227,127,0)" fg:x="156" fg:w="1"/><text x="77.4361%" y="207.50"></text></g><g><title>kprobe_perf_func (1 samples, 0.50%)</title><rect x="77.2277%" y="181" width="0.4950%" height="15" fill="rgb(230,130,0)" fg:x="156" fg:w="1"/><text x="77.4361%" y="191.50"></text></g><g><title>trace_call_bpf (1 samples, 0.50%)</title><rect x="77.2277%" y="165" width="0.4950%" height="15" fill="rgb(234,134,0)" fg:x="156" fg:w="1"/><text x="77.4361%" y="175.50"></text></g><g><title>migrate_enable (1 samples, 0.50%)</title><rect x="77.2277%" y="149" width="0.4950%" height="15" fill="rgb(219,119,0)" fg:x="156" fg:w="1"/><text x="77.4361%" y="159.50"></text></g><g><title>preempt_count_add (1 samples, 0.50%)</title><rect x="77.2277%" y="133" width="0.4950%" height="15" fill="rgb(240,140,0)" fg:x="156" fg:w="1"/><text x="77.4361%" y="143.50"></text></g><g><title>__x64_sys_openat (2 samples, 0.99%)</title><rect x="77.2277%" y="309" width="0.9901%" height="15" fill="rgb(233,133,0)" fg:x="156" fg:w="2"/><text x="77.4361%" y="319.50"></text></g><g><title>do_sys_openat2 (2 samples, 0.99%)</title><rect x="77.2277%" y="293" width="0.9901%" height="15" fill="rgb(243,143,0)" fg:x="156" fg:w="2"/><text x="77.4361%" y="303.50"></text></g><g><title>do_filp_open (2 samples, 0.99%)</title><rect x="77.2277%" y="277" width="0.9901%" height="15" fill="rgb(243,143,0)" fg:x="156" fg:w="2"/><text x="77.4361%" y="287.50"></text></g><g><title>path_openat (2 samples, 0.99%)</title><rect x="77.2277%" y="261" width="0.9901%" height="15" fill="rgb(230,130,0)" fg:x="156" fg:w="2"/><text x="77.4361%" y="271.50"></text></g><g><title>may_open (2 samples, 0.99%)</title><rect x="77.2277%" y="245" width="0.9901%" height="15" fill="rgb(220,120,0)" fg:x="156" fg:w="2"/><text x="77.4361%" y="255.50"></text></g><g><title>inode_permission (1 samples, 0.50%)</title><rect x="77.7228%" y="229" width="0.4950%" height="15" fill="rgb(240,140,0)" fg:x="157" fg:w="1"/><text x="77.9311%" y="239.50"></text></g><g><title>kernfs_iop_permission (1 samples, 0.50%)</title><rect x="77.7228%" y="213" width="0.4950%" height="15" fill="rgb(230,130,0)" fg:x="157" fg:w="1"/><text x="77.9311%" y="223.50"></text></g><g><title>os.OpenFile (5 samples, 2.48%)</title><rect x="76.2376%" y="421" width="2.4752%" height="15" fill="rgb(92,239,92)" fg:x="154" fg:w="5"/><text x="76.4460%" y="431.50">os...</text></g><g><title>os.openFileNolog (5 samples, 2.48%)</title><rect x="76.2376%" y="405" width="2.4752%" height="15" fill="rgb(92,239,92)" fg:x="154" fg:w="5"/><text x="76.4460%" y="415.50">os...</text></g><g><title>syscall.openat (3 samples, 1.49%)</title><rect x="77.2277%" y="389" width="1.4851%" height="15" fill="rgb(93,240,93)" fg:x="156" fg:w="3"/><text x="77.4361%" y="399.50">s..</text></g><g><title>syscall.Syscall6 (3 samples, 1.49%)</title><rect x="77.2277%" y="373" width="1.4851%" height="15" fill="rgb(93,240,93)" fg:x="156" fg:w="3"/><text x="77.4361%" y="383.50">s..</text></g><g><title>runtime/internal/syscall.Syscall6 (3 samples, 1.49%)</title><rect x="77.2277%" y="357" width="1.4851%" height="15" fill="rgb(90,237,90)" fg:x="156" fg:w="3"/><text x="77.4361%" y="367.50">r..</text></g><g><title>entry_SYSCALL_64_after_hwframe (3 samples, 1.49%)</title><rect x="77.2277%" y="341" width="1.4851%" height="15" fill="rgb(245,145,0)" fg:x="156" fg:w="3"/><text x="77.4361%" y="351.50">e..</text></g><g><title>do_syscall_64 (3 samples, 1.49%)</title><rect x="77.2277%" y="325" width="1.4851%" height="15" fill="rgb(243,143,0)" fg:x="156" fg:w="3"/><text x="77.4361%" y="335.50">d..</text></g><g><title>syscall_exit_to_user_mode (1 samples, 0.50%)</title><rect x="78.2178%" y="309" width="0.4950%" height="15" fill="rgb(237,137,0)" fg:x="158" fg:w="1"/><text x="78.4262%" y="319.50"></text></g><g><title>syscall_exit_work (1 samples, 0.50%)</title><rect x="78.2178%" y="293" width="0.4950%" height="15" fill="rgb(237,137,0)" fg:x="158" fg:w="1"/><text x="78.4262%" y="303.50"></text></g><g><title>audit_reset_context.part.0.constprop.0 (1 samples, 0.50%)</title><rect x="78.2178%" y="277" width="0.4950%" height="15" fill="rgb(225,125,0)" fg:x="158" fg:w="1"/><text x="78.4262%" y="287.50"></text></g><g><title>kmem_cache_free (1 samples, 0.50%)</title><rect x="78.2178%" y="261" width="0.4950%" height="15" fill="rgb(220,120,0)" fg:x="158" fg:w="1"/><text x="78.4262%" y="271.50"></text></g><g><title>internal/poll.runtime_pollClose (2 samples, 0.99%)</title><rect x="78.7129%" y="325" width="0.9901%" height="15" fill="rgb(92,239,92)" fg:x="159" fg:w="2"/><text x="78.9212%" y="335.50"></text></g><g><title>runtime/internal/syscall.Syscall6 (2 samples, 0.99%)</title><rect x="78.7129%" y="309" width="0.9901%" height="15" fill="rgb(90,237,90)" fg:x="159" fg:w="2"/><text x="78.9212%" y="319.50"></text></g><g><title>entry_SYSCALL_64_after_hwframe (2 samples, 0.99%)</title><rect x="78.7129%" y="293" width="0.9901%" height="15" fill="rgb(245,145,0)" fg:x="159" fg:w="2"/><text x="78.9212%" y="303.50"></text></g><g><title>do_syscall_64 (2 samples, 0.99%)</title><rect x="78.7129%" y="277" width="0.9901%" height="15" fill="rgb(243,143,0)" fg:x="159" fg:w="2"/><text x="78.9212%" y="287.50"></text></g><g><title>__x64_sys_epoll_ctl (2 samples, 0.99%)</title><rect x="78.7129%" y="261" width="0.9901%" height="15" fill="rgb(233,133,0)" fg:x="159" fg:w="2"/><text x="78.9212%" y="271.50"></text></g><g><title>do_epoll_ctl (2 samples, 0.99%)</title><rect x="78.7129%" y="245" width="0.9901%" height="15" fill="rgb(243,143,0)" fg:x="159" fg:w="2"/><text x="78.9212%" y="255.50"></text></g><g><title>__fget_light (2 samples, 0.99%)</title><rect x="78.7129%" y="229" width="0.9901%" height="15" fill="rgb(228,128,0)" fg:x="159" fg:w="2"/><text x="78.9212%" y="239.50"></text></g><g><title>ima_file_free (2 samples, 0.99%)</title><rect x="79.7030%" y="181" width="0.9901%" height="15" fill="rgb(228,128,0)" fg:x="161" fg:w="2"/><text x="79.9113%" y="191.50"></text></g><g><title>[unknown] (2 samples, 0.99%)</title><rect x="79.7030%" y="165" width="0.9901%" height="15" fill="rgb(243,112,112)" fg:x="161" fg:w="2"/><text x="79.9113%" y="175.50"></text></g><g><title>kprobe_ftrace_handler (2 samples, 0.99%)</title><rect x="79.7030%" y="149" width="0.9901%" height="15" fill="rgb(230,130,0)" fg:x="161" fg:w="2"/><text x="79.9113%" y="159.50"></text></g><g><title>kprobe_perf_func (2 samples, 0.99%)</title><rect x="79.7030%" y="133" width="0.9901%" height="15" fill="rgb(230,130,0)" fg:x="161" fg:w="2"/><text x="79.9113%" y="143.50"></text></g><g><title>process_fetch_insn (2 samples, 0.99%)</title><rect x="79.7030%" y="117" width="0.9901%" height="15" fill="rgb(242,142,0)" fg:x="161" fg:w="2"/><text x="79.9113%" y="127.50"></text></g><g><title>os.ReadFile.func1 (5 samples, 2.48%)</title><rect x="78.7129%" y="421" width="2.4752%" height="15" fill="rgb(92,239,92)" fg:x="159" fg:w="5"/><text x="78.9212%" y="431.50">os...</text></g><g><title>os.(*File).Close (5 samples, 2.48%)</title><rect x="78.7129%" y="405" width="2.4752%" height="15" fill="rgb(92,239,92)" fg:x="159" fg:w="5"/><text x="78.9212%" y="415.50">os...</text></g><g><title>os.(*file).close (5 samples, 2.48%)</title><rect x="78.7129%" y="389" width="2.4752%" height="15" fill="rgb(92,239,92)" fg:x="159" fg:w="5"/><text x="78.9212%" y="399.50">os...</text></g><g><title>internal/poll.(*FD).Close (5 samples, 2.48%)</title><rect x="78.7129%" y="373" width="2.4752%" height="15" fill="rgb(92,239,92)" fg:x="159" fg:w="5"/><text x="78.9212%" y="383.50">int..</text></g><g><title>internal/poll.(*FD).decref (5 samples, 2.48%)</title><rect x="78.7129%" y="357" width="2.4752%" height="15" fill="rgb(92,239,92)" fg:x="159" fg:w="5"/><text x="78.9212%" y="367.50">int..</text></g><g><title>internal/poll.(*FD).destroy (5 samples, 2.48%)</title><rect x="78.7129%" y="341" width="2.4752%" height="15" fill="rgb(92,239,92)" fg:x="159" fg:w="5"/><text x="78.9212%" y="351.50">int..</text></g><g><title>syscall.Close (3 samples, 1.49%)</title><rect x="79.7030%" y="325" width="1.4851%" height="15" fill="rgb(93,240,93)" fg:x="161" fg:w="3"/><text x="79.9113%" y="335.50">s..</text></g><g><title>syscall.Syscall (3 samples, 1.49%)</title><rect x="79.7030%" y="309" width="1.4851%" height="15" fill="rgb(93,240,93)" fg:x="161" fg:w="3"/><text x="79.9113%" y="319.50">s..</text></g><g><title>runtime/internal/syscall.Syscall6 (3 samples, 1.49%)</title><rect x="79.7030%" y="293" width="1.4851%" height="15" fill="rgb(90,237,90)" fg:x="161" fg:w="3"/><text x="79.9113%" y="303.50">r..</text></g><g><title>entry_SYSCALL_64_after_hwframe (3 samples, 1.49%)</title><rect x="79.7030%" y="277" width="1.4851%" height="15" fill="rgb(245,145,0)" fg:x="161" fg:w="3"/><text x="79.9113%" y="287.50">e..</text></g><g><title>do_syscall_64 (3 samples, 1.49%)</title><rect x="79.7030%" y="261" width="1.4851%" height="15" fill="rgb(243,143,0)" fg:x="161" fg:w="3"/><text x="79.9113%" y="271.50">d..</text></g><g><title>syscall_exit_to_user_mode (3 samples, 1.49%)</title><rect x="79.7030%" y="245" width="1.4851%" height="15" fill="rgb(237,137,0)" fg:x="161" fg:w="3"/><text x="79.9113%" y="255.50">s..</text></g><g><title>exit_to_user_mode_prepare (3 samples, 1.49%)</title><rect x="79.7030%" y="229" width="1.4851%" height="15" fill="rgb(230,130,0)" fg:x="161" fg:w="3"/><text x="79.9113%" y="239.50">e..</text></g><g><title>task_work_run (3 samples, 1.49%)</title><rect x="79.7030%" y="213" width="1.4851%" height="15" fill="rgb(222,122,0)" fg:x="161" fg:w="3"/><text x="79.9113%" y="223.50">t..</text></g><g><title>__fput (3 samples, 1.49%)</title><rect x="79.7030%" y="197" width="1.4851%" height="15" fill="rgb(228,128,0)" fg:x="161" fg:w="3"/><text x="79.9113%" y="207.50">_..</text></g><g><title>kernfs_fop_release (1 samples, 0.50%)</title><rect x="80.6931%" y="181" width="0.4950%" height="15" fill="rgb(230,130,0)" fg:x="163" fg:w="1"/><text x="80.9014%" y="191.50"></text></g><g><title>kernfs_unlink_open_file (1 samples, 0.50%)</title><rect x="80.6931%" y="165" width="0.4950%" height="15" fill="rgb(230,130,0)" fg:x="163" fg:w="1"/><text x="80.9014%" y="175.50"></text></g><g><title>kvfree_call_rcu (1 samples, 0.50%)</title><rect x="80.6931%" y="149" width="0.4950%" height="15" fill="rgb(222,122,0)" fg:x="163" fg:w="1"/><text x="80.9014%" y="159.50"></text></g><g><title>_raw_spin_lock (1 samples, 0.50%)</title><rect x="80.6931%" y="133" width="0.4950%" height="15" fill="rgb(237,137,0)" fg:x="163" fg:w="1"/><text x="80.9014%" y="143.50"></text></g><g><title>runtime.growslice (6 samples, 2.97%)</title><rect x="81.1881%" y="421" width="2.9703%" height="15" fill="rgb(90,237,90)" fg:x="164" fg:w="6"/><text x="81.3965%" y="431.50">runt..</text></g><g><title>runtime.mallocgc (6 samples, 2.97%)</title><rect x="81.1881%" y="405" width="2.9703%" height="15" fill="rgb(90,237,90)" fg:x="164" fg:w="6"/><text x="81.3965%" y="415.50">runt..</text></g><g><title>runtime.gcStart (6 samples, 2.97%)</title><rect x="81.1881%" y="389" width="2.9703%" height="15" fill="rgb(90,237,90)" fg:x="164" fg:w="6"/><text x="81.3965%" y="399.50">runt..</text></g><g><title>runtime.gcBgMarkStartWorkers (6 samples, 2.97%)</title><rect x="81.1881%" y="373" width="2.9703%" height="15" fill="rgb(90,237,90)" fg:x="164" fg:w="6"/><text x="81.3965%" y="383.50">runt..</text></g><g><title>runtime.notetsleepg (6 samples, 2.97%)</title><rect x="81.1881%" y="357" width="2.9703%" height="15" fill="rgb(90,237,90)" fg:x="164" fg:w="6"/><text x="81.3965%" y="367.50">runt..</text></g><g><title>runtime.notetsleep_internal (6 samples, 2.97%)</title><rect x="81.1881%" y="341" width="2.9703%" height="15" fill="rgb(90,237,90)" fg:x="164" fg:w="6"/><text x="81.3965%" y="351.50">runt..</text></g><g><title>runtime.futex.abi0 (6 samples, 2.97%)</title><rect x="81.1881%" y="325" width="2.9703%" height="15" fill="rgb(90,237,90)" fg:x="164" fg:w="6"/><text x="81.3965%" y="335.50">runt..</text></g><g><title>entry_SYSCALL_64_after_hwframe (6 samples, 2.97%)</title><rect x="81.1881%" y="309" width="2.9703%" height="15" fill="rgb(245,145,0)" fg:x="164" fg:w="6"/><text x="81.3965%" y="319.50">entr..</text></g><g><title>do_syscall_64 (6 samples, 2.97%)</title><rect x="81.1881%" y="293" width="2.9703%" height="15" fill="rgb(243,143,0)" fg:x="164" fg:w="6"/><text x="81.3965%" y="303.50">do_s..</text></g><g><title>syscall_exit_to_user_mode (6 samples, 2.97%)</title><rect x="81.1881%" y="277" width="2.9703%" height="15" fill="rgb(237,137,0)" fg:x="164" fg:w="6"/><text x="81.3965%" y="287.50">sysc..</text></g><g><title>syscall_exit_work (6 samples, 2.97%)</title><rect x="81.1881%" y="261" width="2.9703%" height="15" fill="rgb(237,137,0)" fg:x="164" fg:w="6"/><text x="81.3965%" y="271.50">sysc..</text></g><g><title>__audit_syscall_exit (6 samples, 2.97%)</title><rect x="81.1881%" y="245" width="2.9703%" height="15" fill="rgb(232,132,0)" fg:x="164" fg:w="6"/><text x="81.3965%" y="255.50">__au..</text></g><g><title>audit_filter_syscall (6 samples, 2.97%)</title><rect x="81.1881%" y="229" width="2.9703%" height="15" fill="rgb(225,125,0)" fg:x="164" fg:w="6"/><text x="81.3965%" y="239.50">audi..</text></g><g><title>exc_page_fault (1 samples, 0.50%)</title><rect x="85.1485%" y="389" width="0.4950%" height="15" fill="rgb(235,135,0)" fg:x="172" fg:w="1"/><text x="85.3568%" y="399.50"></text></g><g><title>do_user_addr_fault (1 samples, 0.50%)</title><rect x="85.1485%" y="373" width="0.4950%" height="15" fill="rgb(243,143,0)" fg:x="172" fg:w="1"/><text x="85.3568%" y="383.50"></text></g><g><title>handle_mm_fault (1 samples, 0.50%)</title><rect x="85.1485%" y="357" width="0.4950%" height="15" fill="rgb(231,131,0)" fg:x="172" fg:w="1"/><text x="85.3568%" y="367.50"></text></g><g><title>__handle_mm_fault (1 samples, 0.50%)</title><rect x="85.1485%" y="341" width="0.4950%" height="15" fill="rgb(226,126,0)" fg:x="172" fg:w="1"/><text x="85.3568%" y="351.50"></text></g><g><title>__cgroup_throttle_swaprate (1 samples, 0.50%)</title><rect x="85.1485%" y="325" width="0.4950%" height="15" fill="rgb(230,130,0)" fg:x="172" fg:w="1"/><text x="85.3568%" y="335.50"></text></g><g><title>blk_cgroup_congested (1 samples, 0.50%)</title><rect x="85.1485%" y="309" width="0.4950%" height="15" fill="rgb(214,114,0)" fg:x="172" fg:w="1"/><text x="85.3568%" y="319.50"></text></g><g><title>main.runBenchmarks.func1 (174 samples, 86.14%)</title><rect x="0.0000%" y="485" width="86.1386%" height="15" fill="rgb(71,219,71)" fg:x="0" fg:w="174"/><text x="0.2083%" y="495.50">main.runBenchmarks.func1</text></g><g><title>main.manualBenchmark (174 samples, 86.14%)</title><rect x="0.0000%" y="469" width="86.1386%" height="15" fill="rgb(71,219,71)" fg:x="0" fg:w="174"/><text x="0.2083%" y="479.50">main.manualBenchmark</text></g><g><title>main.getStatsManual (174 samples, 86.14%)</title><rect x="0.0000%" y="453" width="86.1386%" height="15" fill="rgb(71,219,71)" fg:x="0" fg:w="174"/><text x="0.2083%" y="463.50">main.getStatsManual</text></g><g><title>os.ReadFile (174 samples, 86.14%)</title><rect x="0.0000%" y="437" width="86.1386%" height="15" fill="rgb(92,239,92)" fg:x="0" fg:w="174"/><text x="0.2083%" y="447.50">os.ReadFile</text></g><g><title>runtime.memclrNoHeapPointers (4 samples, 1.98%)</title><rect x="84.1584%" y="421" width="1.9802%" height="15" fill="rgb(90,237,90)" fg:x="170" fg:w="4"/><text x="84.3667%" y="431.50">ru..</text></g><g><title>asm_exc_page_fault (2 samples, 0.99%)</title><rect x="85.1485%" y="405" width="0.9901%" height="15" fill="rgb(231,131,0)" fg:x="172" fg:w="2"/><text x="85.3568%" y="415.50"></text></g><g><title>irqentry_exit_to_user_mode (1 samples, 0.50%)</title><rect x="85.6436%" y="389" width="0.4950%" height="15" fill="rgb(233,133,0)" fg:x="173" fg:w="1"/><text x="85.8519%" y="399.50"></text></g><g><title>exit_to_user_mode_prepare (1 samples, 0.50%)</title><rect x="85.6436%" y="373" width="0.4950%" height="15" fill="rgb(230,130,0)" fg:x="173" fg:w="1"/><text x="85.8519%" y="383.50"></text></g><g><title>fpregs_assert_state_consistent (1 samples, 0.50%)</title><rect x="85.6436%" y="357" width="0.4950%" height="15" fill="rgb(242,142,0)" fg:x="173" fg:w="1"/><text x="85.8519%" y="367.50"></text></g><g><title>security_file_permission (1 samples, 0.50%)</title><rect x="86.1386%" y="277" width="0.4950%" height="15" fill="rgb(238,138,0)" fg:x="174" fg:w="1"/><text x="86.3469%" y="287.50"></text></g><g><title>bpf_lsm_file_permission (1 samples, 0.50%)</title><rect x="86.1386%" y="261" width="0.4950%" height="15" fill="rgb(228,128,0)" fg:x="174" fg:w="1"/><text x="86.3469%" y="271.50"></text></g><g><title>_raw_spin_lock_irqsave (9 samples, 4.46%)</title><rect x="86.6337%" y="213" width="4.4554%" height="15" fill="rgb(237,137,0)" fg:x="175" fg:w="9"/><text x="86.8420%" y="223.50">_raw_sp..</text></g><g><title>preempt_count_add (1 samples, 0.50%)</title><rect x="90.5941%" y="197" width="0.4950%" height="15" fill="rgb(240,140,0)" fg:x="183" fg:w="1"/><text x="90.8024%" y="207.50"></text></g><g><title>cgroup_rstat_flush_locked (10 samples, 4.95%)</title><rect x="86.6337%" y="229" width="4.9505%" height="15" fill="rgb(223,123,0)" fg:x="175" fg:w="10"/><text x="86.8420%" y="239.50">cgroup_r..</text></g><g><title>mem_cgroup_css_rstat_flush (1 samples, 0.50%)</title><rect x="91.0891%" y="213" width="0.4950%" height="15" fill="rgb(230,130,0)" fg:x="184" fg:w="1"/><text x="91.2974%" y="223.50"></text></g><g><title>os.(*File).Read (13 samples, 6.44%)</title><rect x="86.1386%" y="421" width="6.4356%" height="15" fill="rgb(92,239,92)" fg:x="174" fg:w="13"/><text x="86.3469%" y="431.50">os.(*File)...</text></g><g><title>internal/poll.(*FD).Read (13 samples, 6.44%)</title><rect x="86.1386%" y="405" width="6.4356%" height="15" fill="rgb(92,239,92)" fg:x="174" fg:w="13"/><text x="86.3469%" y="415.50">internal/po..</text></g><g><title>syscall.read (13 samples, 6.44%)</title><rect x="86.1386%" y="389" width="6.4356%" height="15" fill="rgb(93,240,93)" fg:x="174" fg:w="13"/><text x="86.3469%" y="399.50">syscall.read</text></g><g><title>syscall.Syscall (13 samples, 6.44%)</title><rect x="86.1386%" y="373" width="6.4356%" height="15" fill="rgb(93,240,93)" fg:x="174" fg:w="13"/><text x="86.3469%" y="383.50">syscall.Sys..</text></g><g><title>runtime/internal/syscall.Syscall6 (13 samples, 6.44%)</title><rect x="86.1386%" y="357" width="6.4356%" height="15" fill="rgb(90,237,90)" fg:x="174" fg:w="13"/><text x="86.3469%" y="367.50">runtime/int..</text></g><g><title>entry_SYSCALL_64_after_hwframe (13 samples, 6.44%)</title><rect x="86.1386%" y="341" width="6.4356%" height="15" fill="rgb(245,145,0)" fg:x="174" fg:w="13"/><text x="86.3469%" y="351.50">entry_SYSCA..</text></g><g><title>do_syscall_64 (13 samples, 6.44%)</title><rect x="86.1386%" y="325" width="6.4356%" height="15" fill="rgb(243,143,0)" fg:x="174" fg:w="13"/><text x="86.3469%" y="335.50">do_syscall_..</text></g><g><title>ksys_read (13 samples, 6.44%)</title><rect x="86.1386%" y="309" width="6.4356%" height="15" fill="rgb(231,131,0)" fg:x="174" fg:w="13"/><text x="86.3469%" y="319.50">ksys_read</text></g><g><title>vfs_read (13 samples, 6.44%)</title><rect x="86.1386%" y="293" width="6.4356%" height="15" fill="rgb(226,126,0)" fg:x="174" fg:w="13"/><text x="86.3469%" y="303.50">vfs_read</text></g><g><title>seq_read_iter (12 samples, 5.94%)</title><rect x="86.6337%" y="277" width="5.9406%" height="15" fill="rgb(236,136,0)" fg:x="175" fg:w="12"/><text x="86.8420%" y="287.50">seq_read_i..</text></g><g><title>cpu_stat_show (12 samples, 5.94%)</title><rect x="86.6337%" y="261" width="5.9406%" height="15" fill="rgb(223,123,0)" fg:x="175" fg:w="12"/><text x="86.8420%" y="271.50">cpu_stat_s..</text></g><g><title>cgroup_base_stat_cputime_show (12 samples, 5.94%)</title><rect x="86.6337%" y="245" width="5.9406%" height="15" fill="rgb(223,123,0)" fg:x="175" fg:w="12"/><text x="86.8420%" y="255.50">cgroup_bas..</text></g><g><title>queued_spin_lock_slowpath (2 samples, 0.99%)</title><rect x="91.5842%" y="229" width="0.9901%" height="15" fill="rgb(233,133,0)" fg:x="185" fg:w="2"/><text x="91.7925%" y="239.50"></text></g><g><title>internal/testlog.Open (1 samples, 0.50%)</title><rect x="92.5743%" y="405" width="0.4950%" height="15" fill="rgb(92,239,92)" fg:x="187" fg:w="1"/><text x="92.7826%" y="415.50"></text></g><g><title>internal/poll.(*FD).Init (1 samples, 0.50%)</title><rect x="93.0693%" y="373" width="0.4950%" height="15" fill="rgb(92,239,92)" fg:x="188" fg:w="1"/><text x="93.2776%" y="383.50"></text></g><g><title>internal/poll.(*pollDesc).init (1 samples, 0.50%)</title><rect x="93.0693%" y="357" width="0.4950%" height="15" fill="rgb(92,239,92)" fg:x="188" fg:w="1"/><text x="93.2776%" y="367.50"></text></g><g><title>internal/poll.runtime_pollOpen (1 samples, 0.50%)</title><rect x="93.0693%" y="341" width="0.4950%" height="15" fill="rgb(92,239,92)" fg:x="188" fg:w="1"/><text x="93.2776%" y="351.50"></text></g><g><title>runtime/internal/syscall.Syscall6 (1 samples, 0.50%)</title><rect x="93.0693%" y="325" width="0.4950%" height="15" fill="rgb(90,237,90)" fg:x="188" fg:w="1"/><text x="93.2776%" y="335.50"></text></g><g><title>entry_SYSCALL_64_after_hwframe (1 samples, 0.50%)</title><rect x="93.0693%" y="309" width="0.4950%" height="15" fill="rgb(245,145,0)" fg:x="188" fg:w="1"/><text x="93.2776%" y="319.50"></text></g><g><title>do_syscall_64 (1 samples, 0.50%)</title><rect x="93.0693%" y="293" width="0.4950%" height="15" fill="rgb(243,143,0)" fg:x="188" fg:w="1"/><text x="93.2776%" y="303.50"></text></g><g><title>__x64_sys_epoll_ctl (1 samples, 0.50%)</title><rect x="93.0693%" y="277" width="0.4950%" height="15" fill="rgb(233,133,0)" fg:x="188" fg:w="1"/><text x="93.2776%" y="287.50"></text></g><g><title>do_epoll_ctl (1 samples, 0.50%)</title><rect x="93.0693%" y="261" width="0.4950%" height="15" fill="rgb(243,143,0)" fg:x="188" fg:w="1"/><text x="93.2776%" y="271.50"></text></g><g><title>_raw_spin_lock (1 samples, 0.50%)</title><rect x="93.0693%" y="245" width="0.4950%" height="15" fill="rgb(237,137,0)" fg:x="188" fg:w="1"/><text x="93.2776%" y="255.50"></text></g><g><title>do_syscall_64 (1 samples, 0.50%)</title><rect x="93.5644%" y="293" width="0.4950%" height="15" fill="rgb(243,143,0)" fg:x="189" fg:w="1"/><text x="93.7727%" y="303.50"></text></g><g><title>syscall_exit_to_user_mode (1 samples, 0.50%)</title><rect x="93.5644%" y="277" width="0.4950%" height="15" fill="rgb(237,137,0)" fg:x="189" fg:w="1"/><text x="93.7727%" y="287.50"></text></g><g><title>syscall_exit_work (1 samples, 0.50%)</title><rect x="93.5644%" y="261" width="0.4950%" height="15" fill="rgb(237,137,0)" fg:x="189" fg:w="1"/><text x="93.7727%" y="271.50"></text></g><g><title>os.newFile (3 samples, 1.49%)</title><rect x="93.0693%" y="389" width="1.4851%" height="15" fill="rgb(92,239,92)" fg:x="188" fg:w="3"/><text x="93.2776%" y="399.50">o..</text></g><g><title>syscall.SetNonblock (2 samples, 0.99%)</title><rect x="93.5644%" y="373" width="0.9901%" height="15" fill="rgb(93,240,93)" fg:x="189" fg:w="2"/><text x="93.7727%" y="383.50"></text></g><g><title>syscall.fcntl (2 samples, 0.99%)</title><rect x="93.5644%" y="357" width="0.9901%" height="15" fill="rgb(93,240,93)" fg:x="189" fg:w="2"/><text x="93.7727%" y="367.50"></text></g><g><title>syscall.Syscall (2 samples, 0.99%)</title><rect x="93.5644%" y="341" width="0.9901%" height="15" fill="rgb(93,240,93)" fg:x="189" fg:w="2"/><text x="93.7727%" y="351.50"></text></g><g><title>runtime/internal/syscall.Syscall6 (2 samples, 0.99%)</title><rect x="93.5644%" y="325" width="0.9901%" height="15" fill="rgb(90,237,90)" fg:x="189" fg:w="2"/><text x="93.7727%" y="335.50"></text></g><g><title>entry_SYSCALL_64_after_hwframe (2 samples, 0.99%)</title><rect x="93.5644%" y="309" width="0.9901%" height="15" fill="rgb(245,145,0)" fg:x="189" fg:w="2"/><text x="93.7727%" y="319.50"></text></g><g><title>zen_untrain_ret (1 samples, 0.50%)</title><rect x="94.0594%" y="293" width="0.4950%" height="15" fill="rgb(245,145,0)" fg:x="190" fg:w="1"/><text x="94.2677%" y="303.50"></text></g><g><title>inode_permission (1 samples, 0.50%)</title><rect x="94.5545%" y="229" width="0.4950%" height="15" fill="rgb(240,140,0)" fg:x="191" fg:w="1"/><text x="94.7628%" y="239.50"></text></g><g><title>kernfs_iop_permission (1 samples, 0.50%)</title><rect x="94.5545%" y="213" width="0.4950%" height="15" fill="rgb(230,130,0)" fg:x="191" fg:w="1"/><text x="94.7628%" y="223.50"></text></g><g><title>up_read (1 samples, 0.50%)</title><rect x="94.5545%" y="197" width="0.4950%" height="15" fill="rgb(226,126,0)" fg:x="191" fg:w="1"/><text x="94.7628%" y="207.50"></text></g><g><title>asm_common_interrupt (1 samples, 0.50%)</title><rect x="94.5545%" y="181" width="0.4950%" height="15" fill="rgb(231,131,0)" fg:x="191" fg:w="1"/><text x="94.7628%" y="191.50"></text></g><g><title>common_interrupt (1 samples, 0.50%)</title><rect x="94.5545%" y="165" width="0.4950%" height="15" fill="rgb(232,132,0)" fg:x="191" fg:w="1"/><text x="94.7628%" y="175.50"></text></g><g><title>__common_interrupt (1 samples, 0.50%)</title><rect x="94.5545%" y="149" width="0.4950%" height="15" fill="rgb(230,130,0)" fg:x="191" fg:w="1"/><text x="94.7628%" y="159.50"></text></g><g><title>handle_edge_irq (1 samples, 0.50%)</title><rect x="94.5545%" y="133" width="0.4950%" height="15" fill="rgb(231,131,0)" fg:x="191" fg:w="1"/><text x="94.7628%" y="143.50"></text></g><g><title>handle_irq_event (1 samples, 0.50%)</title><rect x="94.5545%" y="117" width="0.4950%" height="15" fill="rgb(231,131,0)" fg:x="191" fg:w="1"/><text x="94.7628%" y="127.50"></text></g><g><title>__handle_irq_event_percpu (1 samples, 0.50%)</title><rect x="94.5545%" y="101" width="0.4950%" height="15" fill="rgb(226,126,0)" fg:x="191" fg:w="1"/><text x="94.7628%" y="111.50"></text></g><g><title>irq_int_handler (1 samples, 0.50%)</title><rect x="94.5545%" y="85" width="0.4950%" height="15" fill="rgb(233,133,0)" fg:x="191" fg:w="1"/><text x="94.7628%" y="95.50"></text></g><g><title>atomic_notifier_call_chain (1 samples, 0.50%)</title><rect x="94.5545%" y="69" width="0.4950%" height="15" fill="rgb(227,127,0)" fg:x="191" fg:w="1"/><text x="94.7628%" y="79.50"></text></g><g><title>notifier_call_chain (1 samples, 0.50%)</title><rect x="94.5545%" y="53" width="0.4950%" height="15" fill="rgb(246,146,0)" fg:x="191" fg:w="1"/><text x="94.7628%" y="63.50"></text></g><g><title>mlx5_eq_comp_int (1 samples, 0.50%)</title><rect x="94.5545%" y="37" width="0.4950%" height="15" fill="rgb(221,121,0)" fg:x="191" fg:w="1"/><text x="94.7628%" y="47.50"></text></g><g><title>link_path_walk.part.0.constprop.0 (2 samples, 0.99%)</title><rect x="94.5545%" y="245" width="0.9901%" height="15" fill="rgb(226,126,0)" fg:x="191" fg:w="2"/><text x="94.7628%" y="255.50"></text></g><g><title>walk_component (1 samples, 0.50%)</title><rect x="95.0495%" y="229" width="0.4950%" height="15" fill="rgb(221,121,0)" fg:x="192" fg:w="1"/><text x="95.2578%" y="239.50"></text></g><g><title>lookup_fast (1 samples, 0.50%)</title><rect x="95.0495%" y="213" width="0.4950%" height="15" fill="rgb(232,132,0)" fg:x="192" fg:w="1"/><text x="95.2578%" y="223.50"></text></g><g><title>kernfs_dop_revalidate (1 samples, 0.50%)</title><rect x="95.0495%" y="197" width="0.4950%" height="15" fill="rgb(230,130,0)" fg:x="192" fg:w="1"/><text x="95.2578%" y="207.50"></text></g><g><title>perf_tp_event (1 samples, 0.50%)</title><rect x="95.5446%" y="165" width="0.4950%" height="15" fill="rgb(242,142,0)" fg:x="193" fg:w="1"/><text x="95.7529%" y="175.50"></text></g><g><title>filter_match_preds (1 samples, 0.50%)</title><rect x="95.5446%" y="149" width="0.4950%" height="15" fill="rgb(241,141,0)" fg:x="193" fg:w="1"/><text x="95.7529%" y="159.50"></text></g><g><title>__x64_sys_openat (6 samples, 2.97%)</title><rect x="94.5545%" y="309" width="2.9703%" height="15" fill="rgb(233,133,0)" fg:x="191" fg:w="6"/><text x="94.7628%" y="319.50">__x6..</text></g><g><title>do_sys_openat2 (6 samples, 2.97%)</title><rect x="94.5545%" y="293" width="2.9703%" height="15" fill="rgb(243,143,0)" fg:x="191" fg:w="6"/><text x="94.7628%" y="303.50">do_s..</text></g><g><title>do_filp_open (6 samples, 2.97%)</title><rect x="94.5545%" y="277" width="2.9703%" height="15" fill="rgb(243,143,0)" fg:x="191" fg:w="6"/><text x="94.7628%" y="287.50">do_f..</text></g><g><title>path_openat (6 samples, 2.97%)</title><rect x="94.5545%" y="261" width="2.9703%" height="15" fill="rgb(230,130,0)" fg:x="191" fg:w="6"/><text x="94.7628%" y="271.50">path..</text></g><g><title>may_open (4 samples, 1.98%)</title><rect x="95.5446%" y="245" width="1.9802%" height="15" fill="rgb(220,120,0)" fg:x="193" fg:w="4"/><text x="95.7529%" y="255.50">ma..</text></g><g><title>[unknown] (4 samples, 1.98%)</title><rect x="95.5446%" y="229" width="1.9802%" height="15" fill="rgb(243,112,112)" fg:x="193" fg:w="4"/><text x="95.7529%" y="239.50">[u..</text></g><g><title>kprobe_ftrace_handler (4 samples, 1.98%)</title><rect x="95.5446%" y="213" width="1.9802%" height="15" fill="rgb(230,130,0)" fg:x="193" fg:w="4"/><text x="95.7529%" y="223.50">kp..</text></g><g><title>aggr_pre_handler (4 samples, 1.98%)</title><rect x="95.5446%" y="197" width="1.9802%" height="15" fill="rgb(227,127,0)" fg:x="193" fg:w="4"/><text x="95.7529%" y="207.50">ag..</text></g><g><title>kprobe_perf_func (4 samples, 1.98%)</title><rect x="95.5446%" y="181" width="1.9802%" height="15" fill="rgb(230,130,0)" fg:x="193" fg:w="4"/><text x="95.7529%" y="191.50">kp..</text></g><g><title>process_fetch_insn (3 samples, 1.49%)</title><rect x="96.0396%" y="165" width="1.4851%" height="15" fill="rgb(242,142,0)" fg:x="194" fg:w="3"/><text x="96.2479%" y="175.50">p..</text></g><g><title>copy_from_kernel_nofault (2 samples, 0.99%)</title><rect x="96.5347%" y="149" width="0.9901%" height="15" fill="rgb(229,129,0)" fg:x="195" fg:w="2"/><text x="96.7430%" y="159.50"></text></g><g><title>os.OpenFile (11 samples, 5.45%)</title><rect x="92.5743%" y="421" width="5.4455%" height="15" fill="rgb(92,239,92)" fg:x="187" fg:w="11"/><text x="92.7826%" y="431.50">os.OpenFi..</text></g><g><title>os.openFileNolog (10 samples, 4.95%)</title><rect x="93.0693%" y="405" width="4.9505%" height="15" fill="rgb(92,239,92)" fg:x="188" fg:w="10"/><text x="93.2776%" y="415.50">os.openF..</text></g><g><title>syscall.openat (7 samples, 3.47%)</title><rect x="94.5545%" y="389" width="3.4653%" height="15" fill="rgb(93,240,93)" fg:x="191" fg:w="7"/><text x="94.7628%" y="399.50">sysca..</text></g><g><title>syscall.Syscall6 (7 samples, 3.47%)</title><rect x="94.5545%" y="373" width="3.4653%" height="15" fill="rgb(93,240,93)" fg:x="191" fg:w="7"/><text x="94.7628%" y="383.50">sysca..</text></g><g><title>runtime/internal/syscall.Syscall6 (7 samples, 3.47%)</title><rect x="94.5545%" y="357" width="3.4653%" height="15" fill="rgb(90,237,90)" fg:x="191" fg:w="7"/><text x="94.7628%" y="367.50">runti..</text></g><g><title>entry_SYSCALL_64_after_hwframe (7 samples, 3.47%)</title><rect x="94.5545%" y="341" width="3.4653%" height="15" fill="rgb(245,145,0)" fg:x="191" fg:w="7"/><text x="94.7628%" y="351.50">entry..</text></g><g><title>do_syscall_64 (7 samples, 3.47%)</title><rect x="94.5545%" y="325" width="3.4653%" height="15" fill="rgb(243,143,0)" fg:x="191" fg:w="7"/><text x="94.7628%" y="335.50">do_sy..</text></g><g><title>syscall_exit_to_user_mode (1 samples, 0.50%)</title><rect x="97.5248%" y="309" width="0.4950%" height="15" fill="rgb(237,137,0)" fg:x="197" fg:w="1"/><text x="97.7331%" y="319.50"></text></g><g><title>syscall_exit_work (1 samples, 0.50%)</title><rect x="97.5248%" y="293" width="0.4950%" height="15" fill="rgb(237,137,0)" fg:x="197" fg:w="1"/><text x="97.7331%" y="303.50"></text></g><g><title>__audit_syscall_exit (1 samples, 0.50%)</title><rect x="97.5248%" y="277" width="0.4950%" height="15" fill="rgb(232,132,0)" fg:x="197" fg:w="1"/><text x="97.7331%" y="287.50"></text></g><g><title>audit_filter_syscall (1 samples, 0.50%)</title><rect x="97.5248%" y="261" width="0.4950%" height="15" fill="rgb(225,125,0)" fg:x="197" fg:w="1"/><text x="97.7331%" y="271.50"></text></g><g><title>audit_filter_rules.constprop.0 (1 samples, 0.50%)</title><rect x="97.5248%" y="245" width="0.4950%" height="15" fill="rgb(225,125,0)" fg:x="197" fg:w="1"/><text x="97.7331%" y="255.50"></text></g><g><title>internal/poll.runtime_pollClose (1 samples, 0.50%)</title><rect x="98.0198%" y="325" width="0.4950%" height="15" fill="rgb(92,239,92)" fg:x="198" fg:w="1"/><text x="98.2281%" y="335.50"></text></g><g><title>runtime/internal/syscall.Syscall6 (1 samples, 0.50%)</title><rect x="98.0198%" y="309" width="0.4950%" height="15" fill="rgb(90,237,90)" fg:x="198" fg:w="1"/><text x="98.2281%" y="319.50"></text></g><g><title>entry_SYSCALL_64_after_hwframe (1 samples, 0.50%)</title><rect x="98.0198%" y="293" width="0.4950%" height="15" fill="rgb(245,145,0)" fg:x="198" fg:w="1"/><text x="98.2281%" y="303.50"></text></g><g><title>do_syscall_64 (1 samples, 0.50%)</title><rect x="98.0198%" y="277" width="0.4950%" height="15" fill="rgb(243,143,0)" fg:x="198" fg:w="1"/><text x="98.2281%" y="287.50"></text></g><g><title>__x64_sys_epoll_ctl (1 samples, 0.50%)</title><rect x="98.0198%" y="261" width="0.4950%" height="15" fill="rgb(233,133,0)" fg:x="198" fg:w="1"/><text x="98.2281%" y="271.50"></text></g><g><title>do_epoll_ctl (1 samples, 0.50%)</title><rect x="98.0198%" y="245" width="0.4950%" height="15" fill="rgb(243,143,0)" fg:x="198" fg:w="1"/><text x="98.2281%" y="255.50"></text></g><g><title>ep_remove (1 samples, 0.50%)</title><rect x="98.0198%" y="229" width="0.4950%" height="15" fill="rgb(239,139,0)" fg:x="198" fg:w="1"/><text x="98.2281%" y="239.50"></text></g><g><title>call_rcu (1 samples, 0.50%)</title><rect x="98.0198%" y="213" width="0.4950%" height="15" fill="rgb(221,121,0)" fg:x="198" fg:w="1"/><text x="98.2281%" y="223.50"></text></g><g><title>ima_file_free (1 samples, 0.50%)</title><rect x="98.5149%" y="181" width="0.4950%" height="15" fill="rgb(228,128,0)" fg:x="199" fg:w="1"/><text x="98.7232%" y="191.50"></text></g><g><title>[unknown] (1 samples, 0.50%)</title><rect x="98.5149%" y="165" width="0.4950%" height="15" fill="rgb(243,112,112)" fg:x="199" fg:w="1"/><text x="98.7232%" y="175.50"></text></g><g><title>kprobe_ftrace_handler (1 samples, 0.50%)</title><rect x="98.5149%" y="149" width="0.4950%" height="15" fill="rgb(230,130,0)" fg:x="199" fg:w="1"/><text x="98.7232%" y="159.50"></text></g><g><title>kprobe_perf_func (1 samples, 0.50%)</title><rect x="98.5149%" y="133" width="0.4950%" height="15" fill="rgb(230,130,0)" fg:x="199" fg:w="1"/><text x="98.7232%" y="143.50"></text></g><g><title>process_fetch_insn (1 samples, 0.50%)</title><rect x="98.5149%" y="117" width="0.4950%" height="15" fill="rgb(242,142,0)" fg:x="199" fg:w="1"/><text x="98.7232%" y="127.50"></text></g><g><title>kernfs_fop_release (1 samples, 0.50%)</title><rect x="99.0099%" y="181" width="0.4950%" height="15" fill="rgb(230,130,0)" fg:x="200" fg:w="1"/><text x="99.2182%" y="191.50"></text></g><g><title>seq_release (1 samples, 0.50%)</title><rect x="99.0099%" y="165" width="0.4950%" height="15" fill="rgb(236,136,0)" fg:x="200" fg:w="1"/><text x="99.2182%" y="175.50"></text></g><g><title>__kmem_cache_free (1 samples, 0.50%)</title><rect x="99.0099%" y="149" width="0.4950%" height="15" fill="rgb(223,123,0)" fg:x="200" fg:w="1"/><text x="99.2182%" y="159.50"></text></g><g><title>mod_objcg_state (1 samples, 0.50%)</title><rect x="99.0099%" y="133" width="0.4950%" height="15" fill="rgb(229,129,0)" fg:x="200" fg:w="1"/><text x="99.2182%" y="143.50"></text></g><g><title>__mod_memcg_lruvec_state (1 samples, 0.50%)</title><rect x="99.0099%" y="117" width="0.4950%" height="15" fill="rgb(232,132,0)" fg:x="200" fg:w="1"/><text x="99.2182%" y="127.50"></text></g><g><title>all (202 samples, 100%)</title><rect x="0.0000%" y="597" width="100.0000%" height="15" fill="rgb(255,130,130)" fg:x="0" fg:w="202"/><text x="0.2083%" y="607.50"></text></g><g><title>derp (202 samples, 100.00%)</title><rect x="0.0000%" y="581" width="100.0000%" height="15" fill="rgb(247,119,119)" fg:x="0" fg:w="202"/><text x="0.2083%" y="591.50">derp</text></g><g><title>runtime.goexit.abi0 (202 samples, 100.00%)</title><rect x="0.0000%" y="565" width="100.0000%" height="15" fill="rgb(90,237,90)" fg:x="0" fg:w="202"/><text x="0.2083%" y="575.50">runtime.goexit.abi0</text></g><g><title>runtime.main (202 samples, 100.00%)</title><rect x="0.0000%" y="549" width="100.0000%" height="15" fill="rgb(90,237,90)" fg:x="0" fg:w="202"/><text x="0.2083%" y="559.50">runtime.main</text></g><g><title>main.main (202 samples, 100.00%)</title><rect x="0.0000%" y="533" width="100.0000%" height="15" fill="rgb(71,219,71)" fg:x="0" fg:w="202"/><text x="0.2083%" y="543.50">main.main</text></g><g><title>main.runBenchmarks (202 samples, 100.00%)</title><rect x="0.0000%" y="517" width="100.0000%" height="15" fill="rgb(71,219,71)" fg:x="0" fg:w="202"/><text x="0.2083%" y="527.50">main.runBenchmarks</text></g><g><title>main.benchmarkFunc (202 samples, 100.00%)</title><rect x="0.0000%" y="501" width="100.0000%" height="15" fill="rgb(71,219,71)" fg:x="0" fg:w="202"/><text x="0.2083%" y="511.50">main.benchmarkFunc</text></g><g><title>main.runBenchmarks.func2 (28 samples, 13.86%)</title><rect x="86.1386%" y="485" width="13.8614%" height="15" fill="rgb(71,219,71)" fg:x="174" fg:w="28"/><text x="86.3469%" y="495.50">main.runBenchmarks.func2</text></g><g><title>main.manualBenchmark (28 samples, 13.86%)</title><rect x="86.1386%" y="469" width="13.8614%" height="15" fill="rgb(71,219,71)" fg:x="174" fg:w="28"/><text x="86.3469%" y="479.50">main.manualBenchmark</text></g><g><title>main.getStatsManual (28 samples, 13.86%)</title><rect x="86.1386%" y="453" width="13.8614%" height="15" fill="rgb(71,219,71)" fg:x="174" fg:w="28"/><text x="86.3469%" y="463.50">main.getStatsManual</text></g><g><title>os.ReadFile (28 samples, 13.86%)</title><rect x="86.1386%" y="437" width="13.8614%" height="15" fill="rgb(92,239,92)" fg:x="174" fg:w="28"/><text x="86.3469%" y="447.50">os.ReadFile</text></g><g><title>os.ReadFile.func1 (4 samples, 1.98%)</title><rect x="98.0198%" y="421" width="1.9802%" height="15" fill="rgb(92,239,92)" fg:x="198" fg:w="4"/><text x="98.2281%" y="431.50">os..</text></g><g><title>os.(*File).Close (4 samples, 1.98%)</title><rect x="98.0198%" y="405" width="1.9802%" height="15" fill="rgb(92,239,92)" fg:x="198" fg:w="4"/><text x="98.2281%" y="415.50">os..</text></g><g><title>os.(*file).close (4 samples, 1.98%)</title><rect x="98.0198%" y="389" width="1.9802%" height="15" fill="rgb(92,239,92)" fg:x="198" fg:w="4"/><text x="98.2281%" y="399.50">os..</text></g><g><title>internal/poll.(*FD).Close (4 samples, 1.98%)</title><rect x="98.0198%" y="373" width="1.9802%" height="15" fill="rgb(92,239,92)" fg:x="198" fg:w="4"/><text x="98.2281%" y="383.50">in..</text></g><g><title>internal/poll.(*FD).decref (4 samples, 1.98%)</title><rect x="98.0198%" y="357" width="1.9802%" height="15" fill="rgb(92,239,92)" fg:x="198" fg:w="4"/><text x="98.2281%" y="367.50">in..</text></g><g><title>internal/poll.(*FD).destroy (4 samples, 1.98%)</title><rect x="98.0198%" y="341" width="1.9802%" height="15" fill="rgb(92,239,92)" fg:x="198" fg:w="4"/><text x="98.2281%" y="351.50">in..</text></g><g><title>syscall.Close (3 samples, 1.49%)</title><rect x="98.5149%" y="325" width="1.4851%" height="15" fill="rgb(93,240,93)" fg:x="199" fg:w="3"/><text x="98.7232%" y="335.50">s..</text></g><g><title>syscall.Syscall (3 samples, 1.49%)</title><rect x="98.5149%" y="309" width="1.4851%" height="15" fill="rgb(93,240,93)" fg:x="199" fg:w="3"/><text x="98.7232%" y="319.50">s..</text></g><g><title>runtime/internal/syscall.Syscall6 (3 samples, 1.49%)</title><rect x="98.5149%" y="293" width="1.4851%" height="15" fill="rgb(90,237,90)" fg:x="199" fg:w="3"/><text x="98.7232%" y="303.50">r..</text></g><g><title>entry_SYSCALL_64_after_hwframe (3 samples, 1.49%)</title><rect x="98.5149%" y="277" width="1.4851%" height="15" fill="rgb(245,145,0)" fg:x="199" fg:w="3"/><text x="98.7232%" y="287.50">e..</text></g><g><title>do_syscall_64 (3 samples, 1.49%)</title><rect x="98.5149%" y="261" width="1.4851%" height="15" fill="rgb(243,143,0)" fg:x="199" fg:w="3"/><text x="98.7232%" y="271.50">d..</text></g><g><title>syscall_exit_to_user_mode (3 samples, 1.49%)</title><rect x="98.5149%" y="245" width="1.4851%" height="15" fill="rgb(237,137,0)" fg:x="199" fg:w="3"/><text x="98.7232%" y="255.50">s..</text></g><g><title>exit_to_user_mode_prepare (3 samples, 1.49%)</title><rect x="98.5149%" y="229" width="1.4851%" height="15" fill="rgb(230,130,0)" fg:x="199" fg:w="3"/><text x="98.7232%" y="239.50">e..</text></g><g><title>task_work_run (3 samples, 1.49%)</title><rect x="98.5149%" y="213" width="1.4851%" height="15" fill="rgb(222,122,0)" fg:x="199" fg:w="3"/><text x="98.7232%" y="223.50">t..</text></g><g><title>__fput (3 samples, 1.49%)</title><rect x="98.5149%" y="197" width="1.4851%" height="15" fill="rgb(228,128,0)" fg:x="199" fg:w="3"/><text x="98.7232%" y="207.50">_..</text></g><g><title>locks_remove_file (1 samples, 0.50%)</title><rect x="99.5050%" y="181" width="0.4950%" height="15" fill="rgb(232,132,0)" fg:x="201" fg:w="1"/><text x="99.7133%" y="191.50"></text></g></svg></svg>

Possible fix: bobrik/linux@50b6278