yannforget

Typing tool deadends

What we mean by "deadends"

The typing tool is based on a CART tree trained on survey data. Each categorical split in that tree only knows about the values that were observed during training: at every node, some values go left, some go right, and some were never seen at all. A deadend is a form path that lands on one of those unseen values - the CART has no rule to apply, so it can neither pick a branch nor assign a segment. This happens when a respondent picks a choice

yannforget

Standardization of custom logic in typing tools

We report here all the custom logic we implemented in previous typing tools, and propose a standardization that fits the new web app we are building.

What we are currently doing with the "options" tab

In the current implementation, the options sheet controls node-level mutations of the CART tree before the XLSForm is generated. Each row in the sheet says: *when you reach the tree node whose variable is src_question, replace it

yannforget

de = DataElementCreate(
    name="New data element",
    aggregation_type=AggregationType.SUM,
    domain_type=DataElementDomain.AGGREGATE,
    value_type=ValueType.NUMBER,
    short_name="new de",
)

client.data_elements.post(de)

yannforget

Explicit task ordering

Current implementation and the problem with it

Task dependencies are currently inferred implicitly through data flow. When a task's output is passed as input to another task, a dependency is created:

@pipeline("example")
def my_pipeline():

yannforget

#' Regression analysis for vulnerability factors.
#'
#' This script performs logistic regression analysis on vulnerability factors
#' using survey-weighted data to identify significant predictors of health outcomes.

library(dplyr)
library(duckdb)
library(survey)
library(stringr)
library(parallel)

yannforget

read_parquet <- function(fp) {
  con <- dbConnect(duckdb::duckdb())
  query <- sprintf("CREATE TABLE data AS SELECT * FROM read_parquet('%s')", fp)
  dbExecute(con, query)
  df <- dbGetQuery(con, "SELECT * FROM data")
  dbDisconnect(con)
  return(df)
}

df <- read_parquet("Senegal_2019DHS8_1.0/data/input/SEN_2019DHS8.parquet")

yannforget

library("rpart")
library("dplyr")
library("tibble")
library("jsonlite")

to_json <- function(tree, fp) {

  # Convert rownames to columns as they contain important information: binary
  # node index for tree$frame and split variable for tree$splits
  #   - tree: rpart object

yannforget

from pathlib import Path

import requests


BASE_URL = "https://storage.googleapis.com/gcp-public-data-arco-era5/raw/date-variable-single_level"
DATA_DIR = Path("data")

VARIABLE = "total_precipitation"
YEAR = 2010

yannforget

import sys
from pathlib import Path
from time import sleep

import docker
import psycopg2
import requests


def run(dbdump_dir: Path, conf_dir: Path, version: str = "2.40"):

yannforget

FROM ubuntu:18.04

LABEL maintainer="yannforget@mailbox.org"

ENV DEBIAN_FRONTEND=noninteractive

RUN useradd -ms /bin/bash shedecides

# Install python 2.7, Pandas and NumPy
RUN apt-get -y update && \