Collecting performance data, either in test or production, is only half the story. Why? First, all data is wrong ... by definition. Data is the result of some kind of measurement process and, like cutting a piece of lumber to a measured length, there's always some deviation around the expected length (the pencil line) and the actual cut length (made by the saw blade). The important question is, how much deviation should be tolerated? The building industry has standards that define a tolerance for blueprint lengths, viz., ± 1/8" (or ± 3mm). What is the tolerance for performance engineering measurements?
Second, the only sensible way to address to this question is to compare measurements with their expected values. Expected values come from a "blueprint" called a performance model: a calculable abstraction that characterizes the system being measured. This notion should come as no