Should an API allow for polling or instead push results? What do you prefer?

gistfile1.md

Question

If http://localhost/api/do_job kicks off a job on the backend that will take 45-120 seconds to complete, what should the API return?

POLL) Return 202 Accepted + a URL polling end point.

Eg:

202 Accepted
{ "job_id": 103859195, "poll": "http://localhost/poll/103859195", "complete": "0%" }

You send:

GET http://localhost/poll/103859195

And get back:

200 OK
{ "job_id": 103859195, "poll": "http://localhost/poll/103859195", "complete": "86%" }

You send:

GET http://localhost/poll/103859195

And get back:

200 OK
{ "job_id": 103859195, "data": .... }

PUSH) You send in a callback and the server hits the callback when it's done.

Send:

POST http://localhost/api/do_job
{ "callback": "http://myserver.com/hitme" }

Returns:

200 OK
{ "job_id": 103859195 }

A while later, the server goes:

POST http://myserver.com/hitme
{ "job_id": 103859195, "complete": "100%", "data": .... }

To all who may comment: I have my initial gut feeling but I want to hear your thoughts first.

If I were the consumer and I managed the consuming app entirely, I would 100% of the time prefer a push, webhook-style, callback. It takes less resources on my end because I don't have to constantly poll the server to check progress. I've built things that had to do this (poll Facebook profiles for changes) and this doesn't scale well because you're just guessing at processing time. If the processing time is widely variable, you'll likely perform a lot of unneeded operations only to find out that it's progressed by 5%. I changed that same app I mentioned previously to use a webhook-style callback and I'm able to support far more users without extra processing dynos (it's hosted on Heroku), and thus more cost. It also helps staying under any kind of API rate throttle that may or may not be part of the picture.

However, if it's necessary to update a user via a UI, knowing progress of the job is probably valuable, in which case a poll is probably more reasonable. I could also imagine that for one reason or another, adding a route to accept the push-style callback might not be acceptable, so that would probably dictate the polling-style is used.

@brandonhilkert Thanks for your thoughts on this. I can see how, at scale, the polling method would lead to a lot of traffic noise. The webhook method would require the consumer to have something that accepts the callback, which is my only real concern. Perhaps allowing both would be smart idea.