Using IEX.pry within a test-case

Overview

I wanted to examine the output from an EEx template, I didn't want to polute the application code, and I wanted to build up some assertions around the template. These things start simple and often get complicated really quickly, so I wanted to get a start made early.

I also wanted to verify early on that I would be able to debug test code.

Erlang/Elixir doesn't have the kind of IDE support that something like Java or Python would provide at this stage.

At least within the Intellij plugin. That is both a blessing and a curse. A blessing because it reduces the amount of magic, and reduces the number of abstractions and a curse because you are forced to learn how the system works. Perhaps that isn't a curse.

Implementation

My test case code test/deckset_parser_test.exs looked like this:

test "verify template compiles" do
    output=EEx.eval_file "priv/template.eex", [content: "xxxx"]
    require IEx
    IEx.pry
    output =~ "xxx"
end

First attempt, invoked from the 'mix test' task

So lets see if I can break into it, just be running the test driver, mix test, oh and starting iex for good measure.

% mix test -S iex 
test/deckset_parser_test.exs:6: warning: use of operator == has no effect
.Cannot pry #PID<0.860.0> at test/deckset_parser_test.exs:12. Is an IEx shell running?
..

Finished in 0.1 seconds (0.04s on load, 0.07s on tests)
3 tests, 0 failures

Randomized with seed 109228

Second attempt, invoking from within an 'iex' session

That strategy didn't seem to work, I guess I need to be running an iex session first:

% iex -S mix 
Erlang/OTP 18 [erts-7.3] [source] [64-bit] [smp:8:8] [async-threads:10] [hipe] [kernel-poll:false] [dtrace]

lib/deckset_parser.ex:21: warning: variable callback is unused
lib/deckset_parser.ex:3: warning: unused import Earmark
Compiled lib/deckset_parser.ex
Interactive Elixir (1.2.4) - press Ctrl+C to exit (type h() ENTER for help)

I examine the loaded files:

iex(2)> Code.loaded_files
["/binarytemple/deckset_parser/deps/earmark/mix.exs",
 "/binarytemple/deckset_parser/mix.exs", "/usr/local/bin/mix"]

Ok, the test case hasn't been loaded, or anything else for that matter.

I try compiling the test case. The compilation fails:

Test code can't be compiled unless ExUnit is running

iex(3)> c("./test/deckset_parser_test.exs")
== Compilation error on file test/deckset_parser_test.exs ==
** (RuntimeError) cannot use ExUnit.Case without starting the ExUnit application, please call ExUnit.start() or explicitly start the :ex_unit app
    expanding macro: ExUnit.Case.__using__/1
    test/deckset_parser_test.exs:2: DecksetParserTest (module)
    (elixir) expanding macro: Kernel.use/1
    test/deckset_parser_test.exs:2: DecksetParserTest (module)

** (exit) shutdown: 1
    (elixir) lib/kernel/parallel_compiler.ex:202: Kernel.ParallelCompiler.handle_failure/5
    (elixir) lib/kernel/parallel_compiler.ex:185: Kernel.ParallelCompiler.wait_for_messages/8
    (elixir) lib/kernel/parallel_compiler.ex:55: Kernel.ParallelCompiler.spawn_compilers/3
       (iex) lib/iex/helpers.ex:168: IEx.Helpers.c/2

So that error indicated that the test case cannot be compiled until ExUnit (the Elixir test framework) is running.

As Elixir runs on an Erlang VM, that makes sense. ExUnit must be running some essential process to coordinate the asynchronous test cases.

Accordingly, I start ExUnit:

iex(3)> ExUnit.start
[#Function<0.92649421/1 in ExUnit.start/1>]

And I compile the test module again:

iex(4)> c("./test/deckset_parser_test.exs")
test/deckset_parser_test.exs:6: warning: use of operator == has no effect
[DecksetParserTest]

Success!

Running Individual test cases is hard

I'm going to cut the next bit short. I examine the compiled module. The function names are strange and I'm not sure how to invoke them.

Eventually I do manage to invoke them and a crash occurs.

iex(5)> DecksetParserTest.<tab-key>
__ex_unit__/1
__ex_unit__/2
test that Earmark is working/1
test that Earmark nicely converts the co
test verify template compiles/1
iex(5)> DecksetParserTest."test verify template compiles/1"
** (UndefinedFunctionError) undefined function DecksetParserTest."test verify template compiles/1"/0
    DecksetParserTest."test verify template compiles/1"()

Questions:

There are missing function arguments, what could they be? I need more knowledge of the macro system in order to delve deeper...
There is most likely some sort of test context, perhaps it is encapsulated in the ExUnit process. Can I set a default?

Reverting to a simpler tactic

Lets try a simpler tactic.

I modify the environment, changing it the MIX_ENV variable to 'test'

% MIX_ENV=test iex -S mix

Start ExUnit:

iex(1)> ExUnit.start
[#Function<0.92649421/1 in ExUnit.start/1>]

And run ExUnit, but nothing happens... 0 tests, 0 failures. IEX must not have loaded the code files.

iex(2)> ExUnit.run


Finished in 1.7 seconds (1.7s on load, 0.00s on tests)
0 tests, 0 failures

Randomized with seed 553952
%{failures: 0, skipped: 0, total: 0}

I quickly verify this to be the case:

iex(6)> Code.loaded_files
["/binarytemple/deckset_parser/deps/earmark/mix.exs",
 "/binarytemple/deckset_parser/mix.exs", "/usr/local/bin/mix"]

Load the test code modele:

iex(11)> Code.load_file("deckset_parser_test.exs","test")  |> List.first |> Tuple.to_list |> List.first
test/deckset_parser_test.exs:1: warning: redefining module DecksetParserTest
test/deckset_parser_test.exs:6: warning: use of operator == has no effect
DecksetParserTest

And success

And run ExUnit. Success! The code breaks at IEx.pry.

iex(12)> ExUnit.run
Request to pry #PID<0.221.0> at test/deckset_parser_test.exs:12

        output=EEx.eval_file "priv/template.eex", [content: "xxxx"]
        require IEx
        IEx.pry
      end


Allow? [Yn] y

pry(2)> output
"     <html>\n       <head>\n       <meta http-equiv=\"Content-Type\" content=\"text/html; charset=UTF-8\" />\n       <title>converted markdown</title>\n       </head>\n       <body>\n       xxxx\n       </body>\n\n"

Having examined the variable I allow the session to time out. The default timeout is 60 seconds.

Questions:

How can I redefine the timeout?
How can I obtain a listing of the all the variables, and their values, in the current context?

Answers:

Redefine the timeout using ExUnit.start(timeout: x)
I don't know... TODO

** (EXIT from #PID<0.221.0>) killed


  1) test verify template compiles (DecksetParserTest)
     test/deckset_parser_test.exs:9
     ** (ExUnit.TimeoutError) test timed out after 60000ms. You can change the timeout:

       1. per test by setting "@tag timeout: x"
       2. per case by setting "@moduletag timeout: x"
       3. globally via "ExUnit.start(timeout: x)" configuration
       4. or set it to infinity per run by calling "mix test --trace"
          (useful when using IEx.pry)

     Timeouts are given as integers in milliseconds.

     stacktrace:
       (iex) lib/iex/evaluator.ex:25: IEx.Evaluator.loop/3
       (iex) lib/iex/evaluator.ex:18: IEx.Evaluator.init/3
       (iex) lib/iex.ex:439: IEx.pry/3
       test/deckset_parser_test.exs:12
       (ex_unit) lib/ex_unit/runner.ex:293: ExUnit.Runner.exec_test/1
       (stdlib) timer.erl:166: :timer.tc/1
       (ex_unit) lib/ex_unit/runner.ex:242: anonymous fn/3 in ExUnit.Runner.spawn_test/3

A warning is issued as IEX reloads my dotfile (I named my helper module R):

iex(1)> /Users/bryanhunt/.iex.exs:3: warning: redefining module R
..

Finished in 60.0 seconds
3 tests, 1 failure

Randomized with seed 369325
%{failures: 1, skipped: 0, total: 3}

And the new shell sees to have forgoten about the previously compiled and loaded modules:

iex(1)> ExUnit.run


Finished in 0.00 seconds
0 tests, 0 failures

Randomized with seed 311915
%{failures: 0, skipped: 0, total: 0}

I guess the next step is to define an extra function in my dotfile to handle all this code reloading and test execution.

To be continued;

binarytemple/iex.pry.on.a.test.case.markdown