Async site scraping in ASP.NET 4.5

gistfile1.cs

public class SiteScrape : HttpTaskAsyncHandler
{
    public override async Task ProcessRequestAsync(HttpContext context)
    {
        using (var http = new HttpClient())
        {
            var downloadTasks = new List<Task<string>> {
                http.GetStringAsync("http://bing.com"),
                http.GetStringAsync("http://google.com"),
                http.GetStringAsync("http://oredev.org"),
                http.GetStringAsync("http://microsoft.com"),
                http.GetStringAsync("http://asp.net"),
                http.GetStringAsync("http://signalr.net")
            };
            while (downloadTasks.Count > 0)
            {
                var completed = await Task.WhenAny(downloadTasks);
                downloadTasks.Remove(completed);
                context.Response.Write(completed.Result);
                await context.Response.FlushAsync();
            }
        }
    }
}

Just AMAZING... :D

I never knew about HttpTaskAsyncHandler until now. The fact that it allows you to do IsReusable="true" is awesome. Can't wait to benchmark this thing.

Thanks for posting!

Throw some linq in there while you're at it:

while(downloadTasks.Any()) { ... }

:)

Would this work with Response.BufferOutput = false; if you are getting a Stream not a string?