Performance of new UNUSED_SHOWN_NAME warnings

README.md

Performance of new UNUSED_SHOWN_NAME warning

Examining imports, and now the shown names on imports, is an expensive operation, as it potentially requires examining all of the elements in a library to discover if an import, or a shown name on an import, is used. So I wanted to test whether the new UNUSED_SHOWN_NAME warning significantly increases analysis time.

It obviously should increase analysis time by some amount, as it is doing extra work that wasn't being done before, but I wanted to measure this increase nonetheless.

Analysis

I couldn't figure out, with a cursory look, how to use the benchmark_harness package to test the analyzer package, so I went with a simple time execution. I tested four different scenarios, all described below. Each scenario involves one dart file with a thousand import statements that all import a single, second dart file, showing 2 names per import. A thousand imports is a massive number; I don't think I've ever seen a real Dart file with so many imports, so this is a pretty good test.

The test files are generated with the attached mk-unused-shown-names-tests.sh.

Summary

The details are below, but here's the short of it:

In the absolute worst case scenario, which is a dart file with 1000 import lines that all use "as p", and that each show 2 names. Below the imports, every one of the first names is used (to avoid UNUSED_IMPORT warnings), but only 990 of the second names are used, generating 10 UNUSED_SHOWN_NAME warnings. In this scenario, the existing analyzer took 4.9 seconds to analyze, and the analyzer with my changes took 5.4 seconds, an increase of 0.5 seconds, or 10%.

10% sounds pretty bad to be honest, but this is the absolute worst case scenario; that 10% should never translate to whole seconds. In a final test using only 100 imports, where 90 of the second shown names are used, the increased time in analysis is about 1% (less than 0.1 seconds).

Timing results

1000 imports; each shows 2 names; 1st name always used; only 10 of second name are used

run	current	new
	real	user
1	4.821	4.508
2	4.710	4.501
3	4.702	4.490
4	4.700	4.488
5	4.699	4.490
------	-------	-------
avg*	4.704	4.494

Average increase - real time: 0.1s (3%); user time: 0.1s (3%)

1000 imports; each shows 2 names; 1st name always used; only 990 of second name are used

run	current	new
	real	user
1	5.320	5.009
2	5.266	4.980
3	5.214	4.941
4	5.258	4.954
5	6.075	5.710
------	-------	-------
avg*	5.246	4.981

Average increase - real time: -0.1s (-2%); user time: -0.1s (-1%)

1000 imports using "as"; each shows 2 names; 1st name always used; only 10 of second name are used

run	current	new
	real	user
1	4.766	4.513
2	4.778	4.525
3	4.734	4.486
4	4.794	4.537
5	4.791	4.530
------	-------	-------
avg*	4.778	4.523

Average increase - real time: 0.3s (9%); user time: 0.3s (8%)

1000 imports using "as"; each shows 2 names; 1st name always used; only 990 of second name are used

run	current	new
	real	user
1	5.130	4.852
2	4.898	4.642
3	4.973	4.710
4	4.969	4.719
5	4.897	4.677
------	-------	-------
avg*	4.921	4.702

Average increase - real time: 0.5s (10%); user time: 0.5s (10%)

100 imports using "as"; each shows 2 names; 1st name always used; only 90 of second name are used

run	current	new
	real	user
1	3.554	3.380
2	3.536	3.352
3	3.544	3.369
4	3.524	3.354
5	3.661	3.486
------	-------	-------
avg*	3.545	3.368

Average increase - real time: 0.0s (1%); user time: 0.0s (1%)

* average of middle 3

mk-unused-shown-names-tests.sh

# Many imports; 2 shows per import; few shown used
###############################################################################

file1=many-imports-show__use-few.dart
file2=many-imports-show__use-few2.dart
imports=1000
uses=10

echo '' > $file1
echo '' > $file2

for i in `seq 1 $imports`; do
  echo "import '$file2' show A`printf %03d $i`, B`printf %03d $i`;" >> $file1
done

# Must use all the A's to avoid UNUSED_IMPORT warnings
for i in `seq 1 $imports`; do
  echo "A`printf %03d $i` a`printf %03d $i`;" >> $file1
done

# Just a few uses of B's
for i in `seq 1 $uses`; do
  echo "B`printf %03d $i` b`printf %03d $i`;" >> $file1
done

for i in `seq 1 $imports`; do
  echo "class A`printf %03d $i` {}" >> $file2
  echo "class B`printf %03d $i` {}" >> $file2
done

# Many imports; 2 shows per import; many shown used
###############################################################################

file1=many-imports-show__use-many.dart
file2=many-imports-show__use-many2.dart
imports=1000
uses=990

echo '' > $file1
echo '' > $file2

for i in `seq 1 $imports`; do
  echo "import '$file2' show A`printf %03d $i`, B`printf %03d $i`;" >> $file1
done

# Must use all the A's to avoid UNUSED_IMPORT warnings
for i in `seq 1 $imports`; do
  echo "A`printf %03d $i` a`printf %03d $i`;" >> $file1
done

# Just a few uses of B's
for i in `seq 1 $uses`; do
  echo "B`printf %03d $i` b`printf %03d $i`;" >> $file1
done

for i in `seq 1 $imports`; do
  echo "class A`printf %03d $i` {}" >> $file2
  echo "class B`printf %03d $i` {}" >> $file2
done


# Many imports; as; 2 shows per import; few shown used
###############################################################################

file1=many-imports-as-show__use-few.dart
file2=many-imports-as-show__use-few2.dart
imports=1000
uses=10

echo '' > $file1
echo '' > $file2

for i in `seq 1 $imports`; do
  echo "import '$file2' as f show A`printf %03d $i`, B`printf %03d $i`;" >> $file1
done

# Must use all the A's to avoid UNUSED_IMPORT warnings
for i in `seq 1 $imports`; do
  echo "f.A`printf %03d $i` a`printf %03d $i`;" >> $file1
done

# Just a few uses of B's
for i in `seq 1 $uses`; do
  echo "f.B`printf %03d $i` b`printf %03d $i`;" >> $file1
done

for i in `seq 1 $imports`; do
  echo "class A`printf %03d $i` {}" >> $file2
  echo "class B`printf %03d $i` {}" >> $file2
done


# Many imports; as; 2 shows per import; many shown used
###############################################################################

file1=many-imports-as-show__use-many.dart
file2=many-imports-as-show__use-many2.dart
imports=1000
uses=990

echo '' > $file1
echo '' > $file2

for i in `seq 1 $imports`; do
  echo "import '$file2' as f show A`printf %03d $i`, B`printf %03d $i`;" >> $file1
done

# Must use all the A's to avoid UNUSED_IMPORT warnings
for i in `seq 1 $imports`; do
  echo "f.A`printf %03d $i` a`printf %03d $i`;" >> $file1
done

# Just a few uses of B's
for i in `seq 1 $uses`; do
  echo "f.B`printf %03d $i` b`printf %03d $i`;" >> $file1
done

for i in `seq 1 $imports`; do
  echo "class A`printf %03d $i` {}" >> $file2
  echo "class B`printf %03d $i` {}" >> $file2
done


# Few imports; as; 2 shows per import; most shown used
###############################################################################

file1=few-imports-as-show__use-most.dart
file2=few-imports-as-show__use-most2.dart
imports=100
uses=90

echo '' > $file1
echo '' > $file2

for i in `seq 1 $imports`; do
  echo "import '$file2' as f show A`printf %03d $i`, B`printf %03d $i`;" >> $file1
done

# Must use all the A's to avoid UNUSED_IMPORT warnings
for i in `seq 1 $imports`; do
  echo "f.A`printf %03d $i` a`printf %03d $i`;" >> $file1
done

# Just a few uses of B's
for i in `seq 1 $uses`; do
  echo "f.B`printf %03d $i` b`printf %03d $i`;" >> $file1
done

for i in `seq 1 $imports`; do
  echo "class A`printf %03d $i` {}" >> $file2
  echo "class B`printf %03d $i` {}" >> $file2
done