Skip to content

Instantly share code, notes, and snippets.

@adamw
Created June 14, 2018 15:37
Show Gist options
  • Save adamw/e60861ac52b077288f77739dcc9ba0ac to your computer and use it in GitHub Desktop.
Save adamw/e60861ac52b077288f77739dcc9ba0ac to your computer and use it in GitHub Desktop.
def handleMessage(msg: CrawlerMessage,
data: CrawlerData
): IO[Nothing, CrawlerData] = msg match {
case Start(url) =>
crawlUrl(data, url)
case CrawlResult(url, links) =>
val data2 = data.copy(inProgress = data.inProgress - url)
links.foldM(data2) {
case (d, link) =>
val d2 = d.copy(referenceCount = d.referenceCount.updated(
link.host, d.referenceCount.getOrElse(link.host, 0) + 1))
crawlUrl(d2, link)
}
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment