Asking ChatGPT how to use Typelevel libraries for some parallel fetching.

I'm writing purely functional code in Scala, using the ecosystem of Typelevel libraries. Specifically I'm using cats-effect, fs2, and http4s.

I have a list of URLs, how do make parallel GET requests to them and write their bytes to disk?

Here is one way you can do it using the cats-effect, fs2, and http4s libraries in Scala:

Import the necessary libraries:

import cats.effect.{Blocker, ContextShift, IO}
import fs2.{Stream, io, text}
import org.http4s.client.blaze.BlazeClientBuilder
import org.http4s.client.dsl.Http4sClientDsl
import org.http4s.Uri

Define a function that takes a URL and returns a Stream of IO actions that make a GET request to the URL, and writes the response body to a file. For example:

def downloadUrl(url: Uri)(implicit cs: ContextShift[IO]): Stream[IO, Unit] = {
  val dsl = new Http4sClientDsl[IO] {}
  import dsl._

  // Create a client for making HTTP requests
  val client = BlazeClientBuilder[IO](global).resource

  // Create a blocking execution context for the file I/O operations
  val blocker = Blocker.liftExecutionContext(global)

  // Define the stream that makes the GET request, and writes the response body to a file
  Stream.resource(client)
    .flatMap { c =>
      c.expect[String](GET(url))
        .map(responseBody => responseBody.getBytes)
        .through(io.file.writeAll(Paths.get(s"response_${url.path}.txt"), blocker))
    }
}

Define a list of URLs that you want to make parallel GET requests to. For example:

val urls = List(
  Uri.uri("http://www.example.com/path1"),
  Uri.uri("http://www.example.com/path2"),
  Uri.uri("http://www.example.com/path3")
)

Use the parJoin method from fs2 to run the streams returned by the downloadUrl function in parallel. For example:

Stream
  .emits(urls)
  .covary[IO]
  .map(downloadUrl)
  .parJoin(urls.length)
  .compile
  .drain
  .unsafeRunSync()

This code creates a stream from the list of URLs, then applies the downloadUrl function to each URL to create a stream of IO actions that make the GET request and write the response body to a file. The parJoin method is then used to run these streams in parallel, and the resulting stream is executed using the unsafeRunSync method.

Note that this is just one way to do it, and there are many other ways to achieve the same thing using these libraries. This is just an example to give you an idea of how you might approach the problem.

valencik/gpt-parallel-requests-typelevel.md