when_unset does not appear to behave as documented or expected. This change appears to have been introduced in Ansible devel 62b39d3d. I expect $bippy to fai. I don't expect $foo and $bar to fail. Unseen is 'fake-inventory' that is just "localhost".

playbook.yml

---
- name: webserver plays
  hosts: all
  user: nathan
  sudo: no
  gather_facts: false

  vars:
    foo: "onetwothree"
    bar: "fourfivesix"
    err_variable_not_present: |
        Required variables not present. Make sure you have ~/.wa-dev-ansible-vars defined
        Use playbooks/group_vars/default as a starting point

  pre_tasks:
    - name: "ensure variables"
      fail: >
        msg="{{ err_variable_not_present }}"
      when_unset: $item
      with_items:
        - $foo
        - $bar
        - $bippy

result-62b39d3d

# ansible devel 62b39d3d

$ ansible-playbook -vvv -i fake-inventory playbooks/tmp.yml

PLAY [webserver plays] ******************************************************** 

TASK: [ensure variables] ****************************************************** 
<localhost> ESTABLISH CONNECTION FOR USER: nathan
failed: [localhost] => (item=onetwothree) => {"failed": true, "item": "onetwothree"}
msg: Required variables not present. Make sure you have ~/.wa-dev-ansible-vars defined
Use playbooks/group_vars/default as a starting point

<localhost> ESTABLISH CONNECTION FOR USER: nathan
failed: [localhost] => (item=fourfivesix) => {"failed": true, "item": "fourfivesix"}
msg: Required variables not present. Make sure you have ~/.wa-dev-ansible-vars defined
Use playbooks/group_vars/default as a starting point

<localhost> ESTABLISH CONNECTION FOR USER: nathan
failed: [localhost] => (item=$bippy) => {"failed": true, "item": "$bippy"}
msg: Required variables not present. Make sure you have ~/.wa-dev-ansible-vars defined
Use playbooks/group_vars/default as a starting point


FATAL: all hosts have already failed -- aborting

PLAY RECAP ******************************************************************** 
           to retry, use: --limit @/var/tmp/ansible/tmp.retry

localhost                  : ok=0    changed=0    unreachable=0    failed=1 

result-ca512c7

# ansible devel ca512c7

$ ansible-playbook -vvv -i fake-inventory playbooks/tmp.yml

PLAY [webserver plays] ******************************************************** 

TASK: [ensure variables] ****************************************************** 
skipping: [localhost] => (item=onetwothree)
skipping: [localhost] => (item=fourfivesix)
<localhost> ESTABLISH CONNECTION FOR USER: nathan
failed: [localhost] => (item=$bippy) => {"failed": true, "item": "$bippy"}
msg: Required variables not present. Make sure you have ~/.wa-dev-ansible-vars defined
Use playbooks/group_vars/default as a starting point


FATAL: all hosts have already failed -- aborting

PLAY RECAP ******************************************************************** 
           to retry, use: --limit @/var/tmp/ansible/tmp.retry

localhost                  : ok=0    changed=0    unreachable=0    failed=1   

git bisect on 'devel' branch shows this behavior changed here: ansible/ansible@62b39d3