Standardization of custom logic in typing tools

We report here all the custom logic we implemented in previous typing tools, and propose a standardization that fits the new web app we are building.

What we are currently doing with the "options" tab

In the current implementation, the options sheet controls node-level mutations of the CART tree before the XLSForm is generated. Each row in the sheet says: when you reach the tree node whose variable is src_question, replace it with the following structure. We are currently supporting 3 types of options:

option	handler	effect on the tree
`calculate`	`apply_calculate_option`	Insert a new visible question `dst_question` before the CART node, then turn the CART node into a hidden `calculate` whose formula references `${dst_question}`.
`split`	`apply_split_option`	Insert two visible questions (`dst_question_a`, `dst_question_b`) before the CART node, turn the CART node into a hidden `calculate` whose formula combines both.
`hide`	`apply_hide_option`	Append an XPath `relevant` expression to an existing question, and prune that question's name from descendants' relevance lists.

The sheet is always structured as two columns:

| option   | config                        |
|----------|-------------------------------|
| calculate| <YAML payload, multi-line>    |
| split    | <YAML payload, multi-line>    |
| hide     | <YAML payload, multi-line>    |

Example cell for a calculate option:

src_question: <CART variable>
dst_question_a: <user-facing question A>
dst_question_b: <user-facing question B>
calculation: |
  if(${a} != '' and ${b} != '',
     if(${a} = '<x>' and ${b} = '<y>', <true_value>, <false_value>),
     '')
default: <optional fallback>

Inventory of options used in past typing tools

We inspected 7 workbooks / form configs. Total options per TT are as follows:

tt	calculate	split	hide
IDN_2022DHS7 (s1.1)	3	0	0
IND_MEG_2020DHS8 (op1)	1	2	0
IND_MEG_2020DHS8 (op2)	7	2	2
Kenya_2022DHS8 (S1.0 v2)	4	0	0
NGA_NN_2022PW1 (p2.0)	6	0	0
NGA_NN_2024DHS8 (S1.0)	3	1	0
SEN_2019DHS8 (S6.1)	2	2	0
Total	26	7	2

Observed patterns

Even though every calculate row is a free-form XLSForm formula, the expressions fall into a small number of recurring patterns:

pattern	example	files
Count of selected in a `select_multiple`, then categorized via thresholds	`if(count-selected(${x}) <= 2, '0', if(count-selected(${x}) <= 5, '1', '2'))`	NGA_NN_2024 (`house_index`, `wealth_sum_index_cat1`), NGA_NN_2022PW1 (`asset_index`, `med_index`), IDN (`vehic_index`, `media_sum_2plus`), SEN (`wealth_sum`)
Categorical -> grouped category	`if(${x}='no_education','no_education', if(${x}='incomplete_primary','incomplete_primary','complete_primary_or_higher'))`	Kenya (`ed_lev3`, `toilet_improved`), IDN (`insurance_cov_cat1`), IND op2 (`floor_slum`, `latrine`, `hh_cook_*`, `head_male`), NGA_NN_2022PW1 (`deliver_hos_prefer`)
Boolean recode (categorical -> yes/no or 0/1)	`if(${x}='muslim','yes','no')`	Kenya (`muslim`, `med_cost`), NGA_NN_2022PW1 (`muslim`, `permit`, `early_1stcohab_19`)
Numeric -> boolean threshold	`if(${x}='yes', 9, 7)` (treats integer thresholds as yes/no)	SEN (`highestyearsedinhh_yrs`), NGA_NN_2024 (`highestyearsedinhh_7plus`)
Set-membership (multi-valued select_multiple, but care about whether one value was the only selection)	`if(selected(${x},'through_air...') and count-selected(${x})=1, 0, 1)`	IND op2 (`know_tb_misinfo`)

What goes wrong when we write the options sheet

Inversion of src_question / dst_question
Plain wrong expressions, syntax errors, wrong question names
In the expressions, we reference questions by names. Easy to refer to a question that no long exists or was renamed.
Output domain is implicit. An expression return '0'/'1'/'2' is only correct if the CART node's split values are exactly those. No validation today to an error silently corrupts the flow.
if(${x} != '', ..., default) in every formulat, to guard against empty inputs.
Recodes written as calculates. A pure choice relabel (Kenya muslim: muslim -> yes, anything else -> no) is currently a 6-line XLSForm formula plus an extra duplicated question
Undocumented operations. latrine_cat returns 0 for the choice 'different_type_of_toilet' and 1 for everything else - read the formula 3 times to be sure. A UI with a "which output for which input" table makes this obvious.
hide option: we rewrite descendant relevance using string operations (because a question that can be hidden cannot be required anymore). Very brittle.

Web app strategy

In the new Typing Tool App, a node can contain 1 or multiple questions. A node's aggregation strategy declares how the node's output value is computed from the answers to the questions inside it. The available aggregation strategies depend on how many questions the node has and their types.

The 4 supported strategues (+ an escape hatch) are listed below.

Direct - output is the question's answer

This is the default for any 1-question node. No configuration needed: the node's output equals the question's answer and the answer's value already matches a valid CART value.

No UI configuration needed.

Count and categorize

Available in 1-question nodes where the question is a select_multiple.

UI:

Toggle: "use count" or "categorize count"
If "categorize count": ordered list of categories. Each category is "<= n" + an output value; the last category is implicit "else" with a single output value.
A separate field for the output when the question has no answer yet (default value)

Output: string or integer; should match CART variable's domain

Validation: Each output value is checked against the CART node's split values; output values the CART can never take should be flagged.

Example use cases:

NGA_NN_DHS8 "house_index", "wealth_sum_index_cat1"
IDN_DHS7 "vehic_index", "media_sum_2plus"
NGA_NN_PW1 "asset_index", "med_index"
SEN "wealth_sum"
IND "assets"

Recode

Available in 1-question nodes where the question is a "select_one". Supports n -> m categorical regroups, yes/no -> number, category -> boolean, category -> coarser category, etc.

UI:

A mapping table.

List of source choices on the left, output value on the right. Multiple source choices can map to the same output value; choosing the output is a dropdown when the CART variable is also categorical, or a free input when it's numeric.
A "default" row at the bottom for choices that are not explicitly mapped. Also used when the question is unanswered.

Output: same domain as the CART variable.

Validation: Output values should belong to the CART variable's domain at the relevant node; flag if it's not the case.

Example use cases:

Kenya "muslim", "med_cost", "ed_lev3", "toilet_improved"
IDN "insurance_cov_cat1"
IND "floor_slum", "head_male", "hh_cook_fuel_nosmoke", "hh_cook_inside", "latrine"
NGA_NN_PW1 "muslim", "permit", "early_1stcohab_19", "deliver_hos_prefer"
SEN "highestyearsedinhh_yrs"
NHA_NN_DHS8 "highestyearsedinhh_7plus"

Combine

Available for 2-question nodes (3+ nodes not supported yet)

UI:

For each of the 2 questions in the node, define what counts as a "true" answer: a one-of-[...] set. Single choices and sets are allowed for multiselect questions.
Pick to output for each cell of the table: both true, mixed, neither true. Plus a default value that will be used if any answer is missing.

Output: Any

Validation: Output values should belong to CART value domain at the relevant node.

Example use cases:

IND_MEG "hh_wash_toilet"
IND_MEG "married_10years"
SEN "hh_noimp_water"
SEN "med_cost_dist_moh"
NGA_NN_DHS8 "hh_noimp_housing"

Custom expression

Available for any node, but should be the exception rather than the rule. This is the escape hatch.

UI: Free-form XLSForm input field, hidden behind "advanced" disclosure. The user picks the input questions so we still know what the formula depends on.

Example use cases:

"know_tb_misinfo"

Implementation

Each node can have its own aggregation strategy, e.g:

type Aggregation =
  | { kind: 'direct' }
  | { kind: 'count' }
  | { kind: 'count_and_categorize', categories: ..., default: ...}
  | { kind: 'recode', mapping: ..., default: ...}
  | { kind: 'combine', truth_table: ..., default: ...}
  | { kind: 'custom_expression', inputs: string[], expression: string }

The actual XLSForm formula will be generated at form-build time.

Annex: Inventory of options across previous form configs

Complete row-by-row dump of every active option in the 7 inspected workbooks (35 rows total). Each entry shows the legacy option type, the CART variable (legacy src_question), the user-facing inputs (legacy dst_question / dst_question_a/b), the optional default, the raw XLSForm formula (or relevant expression for hide), and the strategy it maps to in the new model.

IDN_2022DHS7 (s1.1) - 3 options

calculate -- media_sum_2plus → count_and_categorize

inputs: media_sum_2plus_cat
default: 1

if(${media_sum_2plus_cat} != '',
  if(count-selected(${media_sum_2plus_cat}) >= 2, 2, 1),
  1)

calculate -- vehic_index → count_and_categorize

inputs: vehic_index_cat
default: 1

if(${vehic_index_cat} != '',
  if(count-selected(${vehic_index_cat}) >= 2, 3,
    if(count-selected(${vehic_index_cat}) >= 1, 2, 1)),
  1)

calculate -- insurance_cov_cat1 → recode

inputs: insurance_cov_cat1_cat

if(${insurance_cov_cat1_cat} = '', '',
  if(${insurance_cov_cat1_cat} = 'private' or
     ${insurance_cov_cat1_cat} = 'nonsubsidized' or
     ${insurance_cov_cat1_cat} = 'multiple', 2, 1))

IND_MEG_2020DHS8 op1 - 3 options

split -- hh_wash_toilet → combine

inputs: hh_wash_toilet_type, hh_wash_toilet_shared

if(${hh_wash_toilet_type} = 'yes' and ${hh_wash_toilet_shared} = 'no', '1',
  if(${hh_wash_toilet_type} = 'yes' and ${hh_wash_toilet_shared} = 'yes', '0',
    if(${hh_wash_toilet_type} = 'no', '0', '')))

split -- married_10years → combine

inputs: married_10years_time, married_10years_still_living

if(${married_10years_time} = 'yes' and ${married_10years_still_living} = 'yes', '1',
  if(${married_10years_time} = 'no' or ${married_10years_still_living} = 'no', '0', ''))

calculate -- assets → count_and_categorize (raw count, no buckets)

inputs: assets_select

count-selected(${assets_select})

IND_MEG_2020DHS8 op2 - 11 options

split -- hh_wash_toilet → combine

inputs: hh_wash_toilet_type_cat, hh_wash_toilet_shared

if(${hh_wash_toilet_type_cat} != '',
  if((${hh_wash_toilet_type_cat} = 'flush_piped_sewer' or
      ${hh_wash_toilet_type_cat} = 'flush_septic_tank') and
     ${hh_wash_toilet_shared} = 'no', 1, 0),
  0)

split -- married_10years → combine

inputs: married_10years_time, married_10years_still_living

if(${married_10years_time} != '',
  if(${married_10years_time} = 'yes' and ${married_10years_still_living} = 'yes', 1, 0),
  0)

calculate -- assets → count_and_categorize (raw count)

inputs: assets_select

count-selected(${assets_select})

calculate -- floor_slum → recode

inputs: floor_slum_cat

if(${floor_slum_cat} != '', if(${floor_slum_cat} = 'other', 0, 1), 0)

calculate -- head_male → recode

inputs: head_male_cat

if(${head_male_cat} != '', if(${head_male_cat} = 'male', 1, 0), 0)

calculate -- hh_cook_fuel_nosmoke → recode

inputs: hh_cook_fuel_nosmoke_cat

if(${hh_cook_fuel_nosmoke_cat} != '',
  if(${hh_cook_fuel_nosmoke_cat} = 'lpg_natural_gas_electricity', 1, 0),
  0)

calculate -- hh_cook_inside → recode

inputs: hh_cook_inside_cat

if(${hh_cook_inside_cat} != '',
  if(${hh_cook_inside_cat} = 'in_the_house', 1, 0),
  0)

calculate -- latrine → recode (output is inverted: the named value → 0)

inputs: latrine_cat

if(${latrine_cat} != '',
  if(${latrine_cat} = 'different_type_of_toilet', 0, 1),
  0)

calculate -- know_tb_misinfo → custom_expression

inputs: know_tb_misinfo_cat
intent: "is 'through_air…' the only selected item?"

if(selected(${know_tb_misinfo_cat}, 'through_air_when_coughing_sneezing') and
   count-selected(${know_tb_misinfo_cat}) = 1,
  0, 1)

hide -- hh_wash_toilet_shared → intra-node visibility

relevant:

${hh_wash_toilet_type_cat} = 'flush_piped_sewer' or
${hh_wash_toilet_type_cat} = 'flush_septic_tank'

hide -- married_10years_still_living → intra-node visibility

relevant:

${married_10years_time} = 'yes'

Kenya_2022DHS8 (S1.0 v2) - 4 options

calculate -- ed_lev3 → recode

inputs: ed_lev3_cat

if(${ed_lev3_cat} = 'no_education', 'no_education',
  if(${ed_lev3_cat} = 'incomplete_primary', 'incomplete_primary',
    'complete_primary_or_higher'))

calculate -- med_cost → recode

inputs: med_cost_cat

if(${med_cost_cat} = 'big_problem', 'yes', 'no')

calculate -- muslim → recode

inputs: muslim_cat

if(${muslim_cat} = 'muslim', 'yes', 'no')

calculate -- toilet_improved → recode

inputs: toilet_improved_cat

if(${toilet_improved_cat} = 'flush', 'yes',
  if(${toilet_improved_cat} = 'closed', 'yes', 'no'))

NGA_NN_2022PW1 (p2.0) - 6 options

calculate -- asset_index → count_and_categorize (raw count)

inputs: asset_index_cat
default: 0

if(${asset_index_cat} != '', count-selected(${asset_index_cat}), 0)

calculate -- med_index → count_and_categorize (raw count)

inputs: med_index_cat
default: 0

if(${med_index_cat} != '', count-selected(${med_index_cat}), 0)

calculate -- deliver_hos_prefer → recode

inputs: deliver_hos_prefer_cat

if(${deliver_hos_prefer_cat} = '', '',
  if((${deliver_hos_prefer_cat} = 'private_hospital' or
      ${deliver_hos_prefer_cat} = 'public_hospital'),
    1, 0))

calculate -- muslim → recode

inputs: muslim_cat

if(${muslim_cat} = '', '',
  if(${muslim_cat} = 'muslim', 1, 0))

calculate -- early_1stcohab_19 → recode

inputs: early_1stcohab_19_cat

if(${early_1stcohab_19_cat} = '', '',
  if(${early_1stcohab_19_cat} = 'younger_than_19', 1, 0))

calculate -- permit → recode

inputs: permit_cat

if(${permit_cat} = '', '',
  if(${permit_cat} = 'big_problem', 1, 0))

NGA_NN_2024DHS8 (S1.0) - 4 options

calculate -- highestyearsedinhh_7plus → recode

inputs: highestyearsedinhh_yesno
default: '0-6'

if(${highestyearsedinhh_yesno} = 'yes', '7+', '0-6')

calculate -- house_index → count_and_categorize

inputs: house_index_cat
default: '0'

if(${house_index_cat} != '',
  if(count-selected(${house_index_cat}) <= 2, '0',
    if(count-selected(${house_index_cat}) <= 5, '1', '2')),
  '0')

calculate -- wealth_sum_index_cat1 → count_and_categorize

inputs: wealth_sum_index_cat1_cat
default: '0-1'

if(${wealth_sum_index_cat1_cat} != '',
  if(count-selected(${wealth_sum_index_cat1_cat}) <= 1, '0-1', '2-7'),
  '0-1')

split -- hh_noimp_housing → combine

inputs: hh_noimp_housing_walls, hh_noimp_housing_floor

if(${hh_noimp_housing_walls} != '' and ${hh_noimp_housing_floor} != '',
  if(${hh_noimp_housing_walls} = 'unimproved_walls' and
     ${hh_noimp_housing_floor} = 'unimproved_floor',
    'Yes', 'No'),
  '')

SEN_2019DHS8 (S6.1) - 4 options

split -- hh_noimp_water → combine

inputs: hh_noimp_water_source, hh_noimp_water_dist

if(${hh_noimp_water_source} != '' and ${hh_noimp_water_dist} != '',
  if(${hh_noimp_water_source} = 'no' and ${hh_noimp_water_dist} = 'no', 0, 1),
  '')

split -- med_cost_dist_moh → combine

inputs: med_cost_moh, med_dist_moh

if(${med_cost_moh} != '',
  if(${med_cost_moh} = 'yes' and ${med_dist_moh} = 'yes', 1, 0),
  '')

calculate -- wealth_sum → count_and_categorize (raw count)

inputs: wealth
default: 0

if(${wealth} != '', count-selected(${wealth}), 0)

calculate -- highestyearsedinhh_yrs → recode

inputs: highestyearsedinhh_yesno
default: 7

if(${highestyearsedinhh_yesno} = 'yes', 9, 7)

Strategy distribution across the 35 examples

strategy	n
`count_and_categorize`	9
`recode`	16
`combine`	7
`custom_expression`	1

yannforget/tt-app-custom-logic.md

Select an option

No results found

Select an option

No results found

Standardization of custom logic in typing tools

What we are currently doing with the "options" tab

Inventory of options used in past typing tools

Observed patterns

What goes wrong when we write the options sheet

Web app strategy

Direct - output is the question's answer

Count and categorize

Recode

Combine

Custom expression

Implementation

Annex: Inventory of options across previous form configs

IDN_2022DHS7 (s1.1) - 3 options

IND_MEG_2020DHS8 op1 - 3 options

IND_MEG_2020DHS8 op2 - 11 options

Kenya_2022DHS8 (S1.0 v2) - 4 options

NGA_NN_2022PW1 (p2.0) - 6 options

NGA_NN_2024DHS8 (S1.0) - 4 options

SEN_2019DHS8 (S6.1) - 4 options

Strategy distribution across the 35 examples