sample YMLs

2-stations-with-badges.yml

title: bizcard-digitization-with-awesome-badges
public: false
description: A sample line to take an image of a business card, process it so worker can easily read it and then digitize it
department: Digitization

input_formats:
  - name: image_url
    required: true
    valid_type: url

stations:
  # Sample Station #1: Tournament
  - station:
      station_index: 1
      station_type: tournament
      # Create the task form for this station
      task_form:
        form_title: Research to find CEO of a company.
        instruction: Enter details of the CEO for the company below
        form_fields:
          - label: first_name
            field_type: short_answer
          - label: last_name
            field_type: short_answer
          - label: company
            field_type: short_answer
          - label: email
            field_type: short_answer
            valid_type: email
          - label: phone_number
            field_type: short_answer
          - label: country
            field_type: short_answer
      # Define the gold standards
      gold_standards:
        - name: easy
          input:
            image_url: http://link_to_clear_image_of_business_card
          expected_output:
            first_name:
              value: "John" # The answer must be John
            last_name:
              value: "Lennon"
            country:
              value: ["England", "UK", "United Kingdom", "Great Britain"] # can be any one of these
            phone number:
              value: /\+?1?[-.\s]?408[-.\s]?555[-.\s]?1212/ # this regex will accept 408-555-1212, +1 408-555-1212, 4085551212, 408 555 1212 ...
        - name: hard
          input:
            image_url: http://link_to_blurry_image_of_business_card
          expected_output:
            first_name:
              value: "Chuck"
            last_name:
              value: "Norris"
            company:
              value: <% x.titleize.squish ~= /.*Ultimate Dojo.*/ %>
            country:
              value: ["USA", "US", "United States"] # can be any one of these
            phone_number:
              value: /\+?1?[-.\s]?408[-.\s]?555[-.\s]?1212/ # this regex will accept 408-555-1212, +1 408-555-1212, 4085551212, 408 555 1212
            email:
              value: "[email protected]"
      jury_worker:
        num_workers: 2
        max_judges: 5
        reward: 3
      worker:
        worker_type: human
        num_workers: 2
        reward: 5
        badges:
          - name: bizcard-digitizer
            description: this badge qualifies the worker for bizcard digitization
            max_badges: 100
            check_manually: false # if true will mean that the requester reviews the answers and decides to give the worker the badge or not
            test:
              # form: you can define another form here, if no option is given the station's form is used by this badge
              settings: # these are not necessary for manual checking bu
                type: medium
                maximum_retries: 10
                pass_percentage: 100 # all answers should be correct
              gold_standards: ["easy", "hard"]# This means that you have to give the tests in this order

  # Sample Station #2: Improve Station
  - station:
      station_index: 2
      station_type: improve
      gold_standards:
        - name: improver-test
          # input: here, we do not need the input as all inputs will be specified directly in our designed form
          expected_output:
            first_name:
              value: "Elvis"
            year_of_birth:
              value: "1935"
            year_of_death:
              value: "1977"
            eulogy:
              value: This was a crazy dude
              tolerance: 10
      worker:
        num_workers: 1
        reward: 3
        badges:
          - name: bizcard-digitizer # the improver must also have the digitizer badge
          - name: bizcard-improver
            description: Allows you to check and correct previous business card results
            max_badges: 100
            check_manually: true # the requester reviews the answers and decides to give the worker the badge or not
            test:
              custom_form: # Here i want to use the tombfinder form just because I can
                form_title: Correct the information filled in by examining the picture of the tomb
                instruction: Look at the image of the tomb and correct the information filled in previously
                html: station2_badge_test.html
              settings: # not required since manual evaluation but can be specified
                type: strict
                maximum_retries: 5
                pass_percentage: 100
              gold_standards: 1 # It chooses a random gold standard from the station

# gold standards for the line, these can be used to compare the obtained output with the desired one
# these can be used to create test units and analyze the results
gold_standards:
  - name: good_image
    input:
      image_url: link_to_image
    expected_output:
      first_name:
        value: "Chuck"
      last_name:
        value: "Norris"
      country:
        value: "USA"
      phone_number:
        value: "408-555-1212"
      email:
        value: "[email protected]"

Question: Is there anyway we could simplify all the Gold Standard input/output syntax to not use field and name values.

expected_output:
- field: first_name
value: "Elvis"
- field: year_of_birth
value: "1935"
- field: year_of_death
value: "1977"

expected_output:
- first_name: Elvis
- year_of_birth: "1935"
- year_of_death: "1977"