Skip to content

Instantly share code, notes, and snippets.

@art-divin
Created December 29, 2023 22:23
Show Gist options
  • Save art-divin/02ecfa24b6bbf3ba88352c4c60d1a356 to your computer and use it in GitHub Desktop.
Save art-divin/02ecfa24b6bbf3ba88352c4c60d1a356 to your computer and use it in GitHub Desktop.
Copy of apple/swift/utils/update_checkout Python script's concurrent clone function
def run_parallel(fn, pool_args, n_processes=0):
"""Function used to run a given closure in parallel.
NOTE: This function was originally located in the shell module of
swift_build_support and should eventually be replaced with a better
parallel implementation.
"""
if n_processes == 0:
n_processes = cpu_count() * 2
lk = Lock()
print("Running ``%s`` with up to %d processes." %
(fn.__name__, n_processes))
pool = Pool(processes=n_processes, initializer=child_init, initargs=(lk,))
results = pool.map_async(func=fn, iterable=pool_args).get(999999)
pool.close()
pool.join()
return results
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment