Unit tests are quick to run, quick to write in bulk, target a small area of code, are not prone to timing issues and other intermittent failures and have excellent reporting tools.
However, actual usage is often prevented by side effects which require resources such as processes, ports and ets tables to exist or will crash the test.
Separating code into pure and impure functions can help and is typically beneficial. It can also harm readability, and even then not enable full coverage using unit testing. For example:
receive_message(Msg) ->
case check_message(Msg) of
ok ->
exometer:update([recv_good_message], 1),
% put the message into an ets table
queue_message(Msg);
{error, _} ->
exometer:update([skipped_bad_message], 1)
end.
check_message(Msg) ->
% pure function that does some validation....The code is perfectly readable, but how can this function be tested? There is not much purity to be extracted through pure functions.
What I propose is to wrap side effect causing functions in funs, and passing them to the function that will call them.
receive_message(Update_stat_fn, Queue_fn, Msg) ->
case check_message(Msg) of
ok ->
Update_stat_fn([recv_good_message], 1),
% put the message into an ets table
Queue_fn(Msg);
{error, _} ->
Update_stat_fn([skipped_bad_message], 1)
end.This is still easy to read, even though we needed to add two more arguments. This can be wrapped in a module interface much like cowboy_req, which is an API over a record and has side effects such as reading from a network socket.
-module(message_receiver).
-export([init/0]).
-export([receive_message/2]).
-record(message_receiver, {update_stats_fn, queue_fn}).
init() ->
#message_receiver{
update_stats_fn = fun exometer:update/2,
queue_fn = fun queue_message/1 }.
receive_message(Msg, #message_receiver{ update_stats_fn = Update_stat_fn, queue_fn = Queue_fn }) ->
case check_message(Msg) of
ok ->
Update_stat_fn([recv_good_message], 1),
% put the message into an ets table
Queue_fn(Msg);
{error, _} ->
Update_stat_fn([skipped_bad_message], 1)
end.We can now write a whole battery of tests:
-ifdef(TEST).
-include_lib("eunit/include/eunit.hrl").
receive_message_good_message_updates_stat_test() ->
Result_ref = make_ref(),
Msg_receiver = #message_receiver{
update_stats_fn =
fun([recv_good_message],1) ->
put(Result_ref, {[recv_good_message],1})
end,
queue_fn = fun(_) -> ok end
},
receive_message(binary:copy(<<"0">>, 10), Msg_receiver),
?assertEqual(
{[recv_good_message],1},
get(Result_ref)
).
receive_message_good_message_gets_queued_test() ->
Good_message = binary:copy(<<"0">>, 10),
Result_ref = make_ref(),
Msg_receiver = #message_receiver{
update_stats_fn = fun(_,_) -> ok end,
queue_fn = fun(X_msg) -> put(Result_ref, X_msg) end
},
receive_message(Good_message, Msg_receiver),
?assertEqual(
Good_message,
get(Result_ref)
).
receive_message_bad_message_updates_stat_test() ->
Result_ref = make_ref(),
Msg_receiver = #message_receiver{
update_stats_fn =
fun([skipped_bad_message],1) ->
put(Result_ref, {[skipped_bad_message],1})
end
},
receive_message(binary:copy(<<"0">>, 11), Msg_receiver),
?assertEqual(
{[skipped_bad_message],1},
get(Result_ref)
).
-endif.Each test follows the Arrange-Act-Assert pattern. Setup the funs (arrange), execute the the code under test (act), assert the results.
Each test is targeted at testing one specific behaviour. This is to minimise cascading failures, where changing one feature breaks seemingly unrelated tests.
When we come to this code it is obvious how we are supposed to add side effects at any level without breaking existing tests: Add a fun to the state record and call it where we need to.
Some tests may break because the code is now calling a fun that is undefined in the record, it is simple to stub the new fun in the existing tests.
The side effects themselves can be tested. For example, we can test that stats are being incremented.
This is difficult to do in an integration test because the test must sleep or retry to check the stat is updated on an external interface and it cannot be run in parallel.
We didn't have to include meck or any other testing library as a dependency, everything needed to test is part of erlangs basic syntax.
This is important for library developers where adding a dependency forces the library consumers to accept the dependency as well. For example, adding Folsom as a dependency will make rebar download meck as well.
Replacing direct function calls with funs introduces indirection, which should be minimised. It should clear to the reader exactly what is being called by each fun. The points below can help retain clarity despite using funs.
The side effecting funs should never contain logic, function references should always be preferred. They should look like the following:
#message_receiver{
update_stats_fn = fun exometer:update/2, % exported function in other module
queue_fn = fun queue_message/1 % local function
}.Debugging code inside funs because are not named, lets compare a function_clause errors, first a plain fun:
1> Fn = fun(0) -> ok end.
#Fun<erl_eval.6.80484245>
2> Fn(1).
** exception error: no function clause matching
erl_eval:'-inside-an-interpreted-fun-'(1)Not that great, we don't get the function name because funs are not named. Named funs do not give the name in the error message either. Now the function_clause errors for a normal function and a function reference:
3> binary:copy(0).
** exception error: bad argument
in function binary:copy/1
called as binary:copy(0)
4> Copy_fn = fun binary:copy/1.
#Fun<binary.copy.1>
5> Copy_fn(0).
** exception error: bad argument
in function binary:copy/1
called as binary:copy(0)The error message has lost no detail, it is exactly the same so there is no impact on debugging. The function that is referenced is also obvious from the printed value of the fun.
Using function references also means the function signature for the fun is the same as the API function which makes it more familiar than a customised wrapper.
Erlang function tracing can also be used against the referenced function.
There is an exception to the above rule. Partial application of arguments to a function to create a new, lower arity function. For example to send a message to another process we need the pid and and a fun, if we use the method described here. The alternative is to curry the pid into the fun so only the fun needs to directly kept in state:
init(Receiver_pid) ->
Send_fn =
fun(Message) ->
Receiver_pid ! Message
end.This can be used with ets table names and network or file ports as well.
This can be used as long as the applied value doesn't change for the lift of the fun. If the pid will change then a new fun would need to be created with the new pid. In that case currying is probably not worth while.
Using funs instead of calling all functions directly is an abstraction, making what is happening somewhat less obvious. Record field names label the fun in a way that is enforced by the compiler.
Assign the funs to the record in an init function, not in the record definition. Otherwise tests might unexpectedly attempt to invoke side effects if you had forgot to assign a stub fun. Also, it is handy to see all the side effects in one place.
To reduce indirection, assign the funs in the module where they are used. This means you will have an easy to find lookup of funs to their implementation.
I hate meck. Tests that use meck have been slow to run and difficult to maintain, they are a drag to productivity. Take an example from the project page:
my_test() ->
meck:new(my_library_module),
meck:expect(my_library_module, fib, fun(8) -> 21 end),
?assertEqual(21, code_under_test:run(fib, 8)), % Uses my_library_module
?assert(meck:validate(my_library_module)),
meck:unload(my_library_module).You are required to know that the code_under_test module uses my_library_module, this is much different to object mocking where you pass in the a mocked object which has methods called on it. The mocking that meck applies also takes effect globally, this thwarts any attempt to run unit tests in parallel.
In practice unit tests using meck always require a try/catch or meck:unload will not be called on a failure and cause the rest of the tests to fail. It is important to run all tests at least for a module, to see patterns of failures and narrow down the cause.
For testing purists, meck violates the assert last principle.
I fully respect the authors of meck and all those who have contributed to the project. It is the concept of mocking in erlang that I do not agree with.
I heartily encourage the use of integration tests, in conjunction with unit tests. When designing a test suite I aim to make it look like the Test Pyramid. Many unit tests and proportionally less integration tests.
To reiterate the points above, using funs to wrap side effects gives us:
- A consistent way to develop and test new code.
- Test invocations of side effects and their result handling.
- Tests written in this manner execute fast because there is no IO involved.
- 3rd party dependencies are not required.
- This technique adds some indirection, this can be offset by following the patterns under Maintaining understandabilty.