Skip to content

Instantly share code, notes, and snippets.

@faebser
Created March 9, 2019 17:42
Show Gist options
  • Save faebser/912ba4b38a3ab5755886458542cb2a3f to your computer and use it in GitHub Desktop.
Save faebser/912ba4b38a3ab5755886458542cb2a3f to your computer and use it in GitHub Desktop.
supervisor code
def process({:get_thumbs, folder_path, json_path}) do
# lets start a supervisor here
Task.Supervisor.start_link(name: CLI.Supervisor, strategy: :one_for_one, max_timeout: 35)
download_thumbs(json_object, 1, folder_fn)
# download_thumbs unpacks a list of links inside of json object
# and calls donwload_thumbs_files with it
end
def download_thumbs_files(links, result, path) do
# I know I should have used Task.Supervisor.async_nolink
# but I could not get it to work. So i used stream and Enum.into as a kind of await
#
Task.Supervisor.async_stream_nolink(CLI.Supervisor, four, fn url ->
file_name = String.split(url, ")")
|> Enum.reverse
|> hd
file_name = Path.join path, file_name
Logger.info "Downloading #{url} into #{file_name}"
case HTTPoison.get(url, [], [recv_timeout: 20000]) do
{:ok, %HTTPoison.Response{status_code: 200, body: body}} ->
case File.write(file_name, body) do
:ok -> Logger.info "finished writing to #{file_name}"
{:error, reason} -> Logger.error "Could not write #{file_name}: #{reason}"
end
{:ok, %HTTPoison.Response{status_code: 404}} ->
Logger.error "404: Not found #{url}"
{:error, %HTTPoison.Error{reason: reason}} ->
Logger.error "Downloading #{url} failed: #{reason}"
end
end, [timeout: 21000])
|> Enum.into([])
# recursive call
download_thumbs_files(rest, result, path)
end
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment