Lua vararg iteration benchmark

0. Abstract

This small benchmark test compares the performance of using `ipairs{...}` vs.
`select('#',<n>)` to iterate over the arguments inside a variadic function.

1. The code.lua

local unpack = table.unpack or
function(tab)
	return table.remove(tab,1), unpack(tab)
end

function try(fnc,n,...)
	name = name or ""
	ts = os.clock()
	for i=1,n do
		fnc(...)
	end
	te = os.clock()
	print(te-ts)
end

for i=1,8 do
	values = {}
	n = 2^i
	print(("Number of elements: %i"):format(n))
	for i=1,n*2 do
		values[#values+1]=i
	end
	n = 2^22 / n
	print(("Number of iterations: %i"):format(n))

	print("---- STARTING TEST ----")

	io.write "numeric for: "
	try(function(...)
		args = {...}
		for i=1,#args do
			a = args[i]
		end
	end, n, unpack(values))

	io.write "ipairs{...}: "
	try(function(...)
		args = {...}
		for key, value in ipairs(args) do
			a = value
		end
	end, n, unpack(values))

	io.write "select     : "
	try(function(...)
		for i=1,select("#",...) do
			a = select(i,...)
		end
	end, n, unpack(values))

	---[[
	io.write "combination: "
	try(function(...)
		n_args = select("#",...)
		if n_args<=16 then
			for i=1,n_args do
				a = select(i,...)
			end
		else
			args = {...}
			for key, value in ipairs(args) do
				a = value
			end
		end
	end, n, unpack(values))
	--]]
	print()
end

2.1 The results

Number of elements: 1
Number of iterations: 4194304
---- STARTING TEST ----
numeric for: 1.314748
ipairs{...}: 1.786948
select     : 0.825248
combination: 1.115971

Number of elements: 2
Number of iterations: 2097152
---- STARTING TEST ----
numeric for: 0.838786
ipairs{...}: 1.09825
select     : 0.578854
combination: 0.722172

Number of elements: 4
Number of iterations: 1048576
---- STARTING TEST ----
numeric for: 0.521183
ipairs{...}: 0.66698
select     : 0.459284
combination: 0.535527

Number of elements: 8
Number of iterations: 524288
---- STARTING TEST ----
numeric for: 0.370473
ipairs{...}: 0.469861
select     : 0.384568
combination: 0.443036

Number of elements: 16
Number of iterations: 262144
---- STARTING TEST ----
numeric for: 0.307234
ipairs{...}: 0.398395
select     : 0.43302
combination: 0.447391

Number of elements: 32
Number of iterations: 131072
---- STARTING TEST ----
numeric for: 0.26259
ipairs{...}: 0.352319
select     : 0.476176
combination: 0.342218

Number of elements: 64
Number of iterations: 65536
---- STARTING TEST ----
numeric for: 0.222863
ipairs{...}: 0.306345
select     : 0.621975
combination: 0.327539

Number of elements: 128
Number of iterations: 32768
---- STARTING TEST ----
numeric for: 0.208248
ipairs{...}: 0.286315
select     : 0.920346
combination: 0.309997

Number of elements: 256
Number of iterations: 16384
---- STARTING TEST ----
numeric for: 0.230745
ipairs{...}: 0.295762
select     : 1.526707
combination: 0.30213

2.2 Aditional tests

Number of elements: 16
Number of iterations: 262144
---- STARTING TEST ----
numeric for: 0.53651
ipairs{...}: 0.658203
select     : 1.006718
combination: 0.738774

Number of elements: 32
Number of iterations: 131072
---- STARTING TEST ----
numeric for: 0.532007
ipairs{...}: 0.649775
select     : 1.298958
combination: 0.691573

Number of elements: 64
Number of iterations: 65536
---- STARTING TEST ----
numeric for: 0.506338
ipairs{...}: 0.607839
select     : 1.91865
combination: 0.656192

Number of elements: 128
Number of iterations: 32768
---- STARTING TEST ----
numeric for: 0.507063
ipairs{...}: 0.60743
select     : 3.321204
combination: 0.64475

Number of elements: 256
Number of iterations: 16384
---- STARTING TEST ----
numeric for: 0.504592
ipairs{...}: 0.625383
select     : 7.188905
combination: 0.643067

Number of elements: 512
Number of iterations: 8192
---- STARTING TEST ----
numeric for: 0.488121
ipairs{...}: 0.596162
select     : 13.666135
combination: 0.609433

Number of elements: 1024
Number of iterations: 4096
---- STARTING TEST ----
numeric for: 0.487915
ipairs{...}: 0.57412900000001
select     : 25.984348
combination: 0.589311

2.3 LuaJIT results

Number of elements: 32
Number of iterations: 131072
---- STARTING TEST ----
numeric for: 0.067259
ipairs{...}: 0.061422
select     : 0.022591
combination: 0.071471

Number of elements: 64
Number of iterations: 65536
---- STARTING TEST ----
numeric for: 0.046058
ipairs{...}: 0.046445
select     : 0.014507
combination: 0.054622

Number of elements: 128
Number of iterations: 32768
---- STARTING TEST ----
numeric for: 0.041115
ipairs{...}: 0.03876
select     : 0.011633
combination: 0.0432

Number of elements: 256
Number of iterations: 16384
---- STARTING TEST ----
numeric for: 0.029038
ipairs{...}: 0.031937
select     : 0.010491
combination: 0.034294

Number of elements: 512
Number of iterations: 8192
---- STARTING TEST ----
numeric for: 0.030625
ipairs{...}: 0.030857
select     : 0.01198
combination: 0.038555

Number of elements: 1024
Number of iterations: 4096
---- STARTING TEST ----
numeric for: 0.0268
ipairs{...}: 0.030364
select     : 0.018461
combination: 0.033491

Number of elements: 2048
Number of iterations: 2048
---- STARTING TEST ----
numeric for: 0.023028
ipairs{...}: 0.028488
select     : 0.013719
combination: 0.029133

Number of elements: 4096
Number of iterations: 1024
---- STARTING TEST ----
numeric for: 0.021597
ipairs{...}: 0.024167
select     : 0.011663
combination: 0.029756

3. Conclusion

# Conclusion

The tests clearly show that with up to around 20 arguments, using `select()`
seems to be faster, while above that it grows increasingly slower.

In practice this means that most variadic functions can perform better if
implemented that way. With functions that expect more arguments than around 16
to 20 though it is more effective to use the traditional method of building a
table and iterating it with `ipairs`.

## Update

After adding a version with a numeric for loop, the results didn't change all
that much. Select is still a bit faster for small numbers of arguments, but the
tipping point has moved to around 8 arguments. The difference between the mixed
implementation and the pire numeric for implementation is also way bigger than
with the ipairs implementation. What this means is that one should probably go
for that when expecting lots of arguments, though the mixed one is still faster
for very few arguments and still safe for large numbers of arguments, when the
pure select loop would start taking a lot longer for each doubling of the
argument count. (see 2.5)

## LuaJIT

The results when using LuaJIT are somewhat surprising, though not completely
unexpected. The first iteration is not all that representative, as is usually
the case with JIT compilation. Once the interpreter knows what's going on though
the optimization kicks in and there's a massive speed gain. What's clear
throughout all the iterations though,  is that select is a lot faster, almost
x2 the speed ot the numeric for loop and more than x2 as fast as ipairs{...}.

Thanks for your benchmark. I randomly found it while developing an addon for World of Warcraft and was curious about the results in their Lua runtime.

This benchmark shows that

• ipairs is never a good option when iterating over an array
• select is almost never a good option (only for arrays with less than 8 elements)
• for loop is always the best option when the array has more than 8 elements

The best mixed function is

function(...)
    n_args = select("#", ...)
    if n_args < 8 then
        for i = 1, n_args do
            _ = select(i, ...)
        end
    else
        args = { ... }
        for i = 1, #args do
            _ = args[i]
        end
    end
end

Below results are in milliseconds.

Number of elements: 2
Number of iterations: 131072
---- STARTING TEST ----
numeric for     :   70.318300001323
ipairs{...}     :   89.153599999845
select          :   43.624799996614
combination     :   42.834199994802

Number of elements: 4
Number of iterations: 65536
---- STARTING TEST ----
numeric for     :   65.433800004423
ipairs{...}     :   81.087600000203
select          :   54.121400006115
combination     :   50.093099996448

Number of elements: 8
Number of iterations: 32768
---- STARTING TEST ----
numeric for     :   45.637099996209
ipairs{...}     :   72.589699998498
select          :   59.218299999833
combination     :   44.484600000083

Number of elements: 16
Number of iterations: 16384
---- STARTING TEST ----
numeric for     :   49.789599999785
ipairs{...}     :   58.955499999225
select          :   67.243799999356
combination     :   46.134999997914

Number of elements: 32
Number of iterations: 8192
---- STARTING TEST ----
numeric for     :   37.584800004959
ipairs{...}     :   62.997199997306
select          :   85.755700007081
combination     :   38.319199994206

Number of elements: 64
Number of iterations: 4096
---- STARTING TEST ----
numeric for     :   40.573600001633
ipairs{...}     :   52.718400001526
select          :  110.66619999707
combination     :   33.006200000644

Number of elements: 128
Number of iterations: 2048
---- STARTING TEST ----
numeric for     :   27.979100003839
ipairs{...}     :   55.9571999982
select          :  170.39100000262
combination     :   28.91629999876

Number of elements: 256
Number of iterations: 1024
---- STARTING TEST ----
numeric for     :   26.68069999665
ipairs{...}     :   60.546299993992
select          :  291.39919999987
combination     :   28.142300002277