Options for 3rd Party Integration into CrowdFlower

Option 1: CML iFrame

Include the following javascript snippet in the <head> of your custom interface:

<script src="https://crowdflower.github.io/construe/cml_iframe.js"></script>

Then there are two required calls for your interface to communicate with the CrowdFlower platform:

window._cf.init((payload) => {
	//Initialize your interface with the meta data from the CrowdFlower platform
})

The payload passed in the init method looks as follows:

{
	"mode": "default", // Can also be, “missed”, “test_questions”, and “show”
	"gold": null, // Will contain a JSON object with the saved truth data if a test ?
	"options": {}, // Options passed from the CML
	"missed": {}, // The JSON returned from your custom gold endpoint
	"data": {}, // The unit data for the given row being worked on
}

The other required call is:

window._cf.post("persist", payload)

Payload can be a javascript object or a string. It’s what will be persisted in the CrowdFlower platform. To embed your interface into a CrowdFlower job you use the following CML:

<cml:iframe name="annotation" src="https://yourinterface.com" matcher-url="https://yourmatchinglogic.com">
	{
		"some_option": true
	}
</cml:iframe>

The matching logic endpoint will receive a POST request with the following JSON body:

{
	"unit_data": {},
  	"unit_id": 1,
 	"judgment_data": {},
  	"options": {},
  	"gold_data": {
  		"truth":{},
  		"judgment": {}
  	}
}

You’ll want to compare the truth and judgment of the gold_data to determine if judgment is good enough. Once you’ve made the comparison the response must be a 200 status code with a JSON body containing an "accept" key which should either be true or false. You can also put other data in the JSON body for rendering what was missed to the end user, this will be passed to your interface in the init method with the "missed" key. If your server takes longer than 5 seconds, returns invalid JSON, or a non-200 response code the judgment is accepted and considered correct.

Option 2: External Validation

This is especially useful for surveys, but can be used for tasks with right and wrong answers. However, if you use this method for tasks you are completely responsible for the quality of the data and won't be able to leverage any of CrowdFlower quality control tools.

First create a cml:text or cml:checkbox in your job and add the ss-external validator, and provide a link to your interface:

<cml:text name="code" validates="required ss-external" validator-args-ss-external="https://yourvalidationlogic.com" />
<a href="http://yourinterface.com" target="_blank">Perform a task here</a>

When a contributor submits the task we will issue a POST to the url you provide in the "validator-args-ss-external" attribute with the following data:

{
	"ip": "192.168.0.1", // IP address of the worker
	"worker_id": 12345,
	"job_id": 12345,
	"unit_id": 12345,
	"judgment_data": {}, // the judgment data
	"unit_data": {}, // the unit data
}

If you respond with a 403 response code we will reject the judgment and not let it through. If you respond with any other response code or take longer than 5 seconds to respond we let the judgment through. This generally only make sense for jobs which have 1 unit per assignment. You could simply verify the IP address as one that recently did work on your platform, or provide the user a token when they complete work on your platform which will be passed through in judgment_data for validation.

If your job requires asynchronous verification of results the best approach is to price your job at $0.01 per assignment with 1 unit per assignment. Then inform users in the title and instructions that they will receive a bonus of X amount within X hours if their work is approved. You should also flag workers that provide bad judgments from doing work on your future jobs. The relevant API endpoints are:

#Bonus
curl -X POST --data-urlencode "amount={amount_in_cents}" https://api.crowdflower.com/v1/jobs/{job_id}/workers/{worker_id}/bonus.json?key={api_key}

#Flag
curl -X PUT --data-urlencode "persist=true&flag={reason_for_flagging_contributor}" https://api.crowdflower.com/v1/jobs/{job_id}/workers/{worker_id}.json?key={api_key}

vanpelt/documentation.md

Option 1: CML iFrame

Option 2: External Validation