Download large files in rust with progress bar using reqwest, future_util and indicatif

download_file.rs

// you need this in your cargo.toml
// reqwest = { version = "0.11.3", features = ["stream"] }
// futures-util = "0.3.14"
// indicatif = "0.15.0"
use std::cmp::min;
use std::fs::File;
use std::io::Write;

use reqwest::Client;
use indicatif::{ProgressBar, ProgressStyle};
use futures_util::StreamExt;

pub async fn download_file(client: &Client, url: &str, path: &str) -> Result<(), String> {
    // Reqwest setup
    let res = client
        .get(url)
        .send()
        .await
        .or(Err(format!("Failed to GET from '{}'", &url)))?;
    let total_size = res
        .content_length()
        .ok_or(format!("Failed to get content length from '{}'", &url))?;
    
    // Indicatif setup
    let pb = ProgressBar::new(total_size);
    pb.set_style(ProgressStyle::default_bar()
        .template("{msg}\n{spinner:.green} [{elapsed_precise}] [{wide_bar:.cyan/blue}] {bytes}/{total_bytes} ({bytes_per_sec}, {eta})")
        .progress_chars("#>-"));
    pb.set_message(&format!("Downloading {}", url));

    // download chunks
    let mut file = File::create(path).or(Err(format!("Failed to create file '{}'", path)))?;
    let mut downloaded: u64 = 0;
    let mut stream = res.bytes_stream();

    while let Some(item) = stream.next().await {
        let chunk = item.or(Err(format!("Error while downloading file")))?;
        file.write_all(&chunk)
            .or(Err(format!("Error while writing to file")))?;
        let new = min(downloaded + (chunk.len() as u64), total_size);
        downloaded = new;
        pb.set_position(new);
    }

    pb.finish_with_message(&format!("Downloaded {} to {}", url, path));
    return Ok(());
}

thanks for this example, do you know if the progressbar is possible with POST request ? like https://bashupload.com/how_to_upload_progress_curl

@ozkanpakdil I'm pretty sure that if you change line 16 from .get to .post it will work.

can you please provide a full working example?
Not sure how to call this async function from a non-async main..

For future reference, I've pushed a full example here.

@giuliano-oliveira I could not find a way to create progress bar for upload :) https://stackoverflow.com/questions/70252995/how-to-monitor-reqwest-client-upload-progress feel free to help

@ozkanpakdil : have a look at the implementation of aim for how upload with a progress bar is handled.
I'm adding more features in the coming weeks/months.

@giuliano-oliveira @mihaigalos can this be used to download multiple files concurrently?

You might want to consider using write_all instead of write: https://rust-lang.github.io/rust-clippy/master/index.html#unused_io_amount

Yes, I agree. It is important to use write_all instead of write, otherwise it won't work

Makes perfect sense, thanks @orhun @lennartkloock

Does .get().send().await not immediately retrieve the whole response body to memory?

@Tails that just sends initial http headers and waits for the server http headers to come back, but doesn't block on the response (this is why e.g. .json on responses needs await)

@mihaigalos Thanks for your example with resuming if the file exists. But don't you have to seek also in the response stream the same amount (file_size)? Or can you please clarify by which mechanism the response stream will be forwarded to the right byte index, because I don't see anything that advances the stream:

https://github.com/mihaigalos/tutorials/blob/800d5acbc333fd4068622e9b3d870cb5b7d34e12/rust/download_with_progressbar/src/main.rs#L47

Hi @Boscop,
You're right, you do need to forward the byte index.

That needs to happen even before the download, however, because the backend needs to know from which byte it needs to start the stream.

Long story short, see this.

Hi @giuliano-oliveira, thank you for this solution. But May I suggest improving it? Before appending an existing file, we need to check its file size compared with the response file size. If we don't check, it will append every time when the file exists then the file size becomes larger and larger. Hope you understand my idea! Thanks.

@lehaidangdev I like the ideia (and I miss pull requests in gists :( ), could you fork, modify it and make another comment with the fork?
Then i can comment on your gist and we can "merge" it, what do you think? I could create a repo just for this example as well, and replace this gist with a link to the repo.

If we don't check, it will append every time when the file exists then the file size becomes larger and larger. Hope you understand my idea! Thanks.

Though I'm might be misunderstanding what you are saying, but if the file already exists in line 32 it would truncate the original file (which is bad as well in my opinion, maybe we should check if the file already exists, and if does, return an error)

I found this discussion by chance because I too wanted to download files easily using Tauri and Tauri Upload Plugin as now it also has a download function with a ProgressHandler which can download a file from a URL to a Disk

Just in Case anyone stumbles here with the same problem.