with this async generator queue, you could make sure the tasks given are run with at a maximum concurrency
Created
July 6, 2024 11:30
-
-
Save ycmjason/de703c549db23110080b8f82f40206d3 to your computer and use it in GitHub Desktop.
async queue with async generator
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
async function* queue(tasks, maxConcurrency = 5) { | |
const runningPromises = []; | |
const resultPromises = tasks.map(async (task) => { | |
while (runningPromises.length > maxConcurrency) | |
await Promise.race(runningPromises); | |
const promise = task(); | |
runningPromises.push(promise); | |
const result = await promise; | |
runningPromises.splice(runningPromises.indexOf(promise), 1); | |
return result; | |
}); | |
for (const p of resultPromises) { | |
yield await p; | |
} | |
} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
const tasks = Array.from({ length: 50 }, (_, i) => async () => { | |
console.log(`start task ${i}`); | |
await new Promise((res) => setTimeout(res, 5000 * Math.random())); | |
console.log(`end task ${i}`); | |
return i; | |
}); | |
for await (const result of queue(tasks)) console.log({ result }); | |
/* | |
Example result:start task 0 | |
start task 1 | |
start task 2 | |
start task 3 | |
start task 4 | |
start task 5 | |
end task 0 | |
start task 6 | |
{result: 0} | |
end task 3 | |
start task 7 | |
end task 2 | |
start task 8 | |
end task 6 | |
start task 9 | |
end task 5 | |
start task 10 | |
end task 4 | |
start task 11 | |
end task 1 | |
start task 12 | |
{result: 1} | |
{result: 2} | |
{result: 3} | |
{result: 4} | |
{result: 5} | |
{result: 6} | |
end task 9 | |
start task 13 | |
end task 7 | |
start task 14 | |
{result: 7} | |
end task 10 | |
start task 15 | |
end task 8 | |
start task 16 | |
{result: 8} | |
{result: 9} | |
{result: 10} | |
end task 13 | |
start task 17 | |
end task 15 | |
start task 18 | |
end task 12 | |
start task 19 | |
end task 14 | |
start task 20 | |
end task 11 | |
start task 21 | |
{result: 11} | |
{result: 12} | |
{result: 13} | |
{result: 14} | |
{result: 15} | |
end task 20 | |
start task 22 | |
end task 18 | |
start task 23 | |
end task 22 | |
start task 24 | |
end task 21 | |
start task 25 | |
end task 25 | |
start task 26 | |
end task 16 | |
start task 27 | |
{result: 16} | |
end task 17 | |
start task 28 | |
{result: 17} | |
{result: 18} | |
end task 27 | |
start task 29 | |
end task 24 | |
start task 30 | |
end task 30 | |
start task 31 | |
end task 19 | |
start task 32 | |
{result: 19} | |
{result: 20} | |
{result: 21} | |
{result: 22} | |
end task 28 | |
start task 33 | |
end task 23 | |
start task 34 | |
{result: 23} | |
{result: 24} | |
{result: 25} | |
end task 29 | |
start task 35 | |
end task 26 | |
start task 36 | |
{result: 26} | |
{result: 27} | |
{result: 28} | |
{result: 29} | |
{result: 30} | |
end task 32 | |
start task 37 | |
end task 33 | |
start task 38 | |
end task 34 | |
start task 39 | |
end task 36 | |
start task 40 | |
end task 35 | |
start task 41 | |
end task 31 | |
start task 42 | |
{result: 31} | |
{result: 32} | |
{result: 33} | |
{result: 34} | |
{result: 35} | |
{result: 36} | |
end task 38 | |
start task 43 | |
end task 43 | |
start task 44 | |
end task 40 | |
start task 45 | |
end task 37 | |
start task 46 | |
{result: 37} | |
{result: 38} | |
end task 39 | |
start task 47 | |
{result: 39} | |
{result: 40} | |
end task 42 | |
start task 48 | |
end task 46 | |
start task 49 | |
end task 41 | |
{result: 41} | |
{result: 42} | |
{result: 43} | |
end task 44 | |
{result: 44} | |
end task 49 | |
end task 45 | |
{result: 45} | |
{result: 46} | |
end task 47 | |
{result: 47} | |
end task 48 | |
{result: 48} | |
{result: 49} | |
*/ |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment