HttpClient Ideas

README.md

This adds a async-only Http fluent interface to help us reduce the crazy amount of overloads and combinations in HtmlUtilities. In general, we have a combinatorial for each send/receive/sync/async and a litany of options (as optional parameters) underneath. We deal with sending: JSON, Protobuf, plaintext, HTML, Forms, and maybe more fun I haven't seen yet. We also receive and handle JSON, Protobuf, plaintext, HTML.

We send one type and get another, or we send nothing (GET) and get one of several several types. Sometimes it's synchronous, sometimes it's asynchronous (but not async, so that's fun). In short: what we have is a potato with umpteen spuds coming out. We also explicitly pass what's private (either via the method name or an optional parameter) rather than noticing we replaced something in SubstituteInternalUrl.

What's here is a more composable way of handling any combination needed in a (hopefully) easy-to-use, easy-to-do-the-right-thing, and extensible way.

1. We have an Http class (static) which you use as the root for any request, via Http.Request(url).
2. (If sending a body) You can use .Send____ such as .SendJson(myObject) or .SendForm(myFormCollection).
3. Set the response type via .Expect____ such as .ExpectJson<User>(), .ExpectProtobuf<List<MyClass>>(), .ExpectString(), etc.
4. Issue with the request with the HTTP verb desired, e.g. .DeleteAsync(), .GetAsync(), .PostAsync(), .PutAsync()

The response from the above is an HttpResponse<T>, where T is the expected type.

All of the above .Send and .Expect methods are extensions on the interface modifying the request builder before sending. We can add more where needed (or use and then graduate them to StackExchange.Network if this API moves there).

Method overview:

• Http.Request(string url) - Creates a new request builder
• Extensions: .Send____() - Sets the request body (if any) to send
• Extensions: .Expect____() - Sets the response processor/deserializer/handler
• Extension: .WithTimeout(TimeSpan timeout) - Sets a timeout explicitly (default will be 3 seconds for GET and 6 seconds for POST as per previous defaults in the final version)
• Extension: .DontLogErrors() - Disables the Exceptional error log call no matter what happens
• Extension: .AddHeaders() - Add arbitrary headers to the request
• Extensions: .___Async() - Sends the request out to get a result, all async

Http.cs

using System;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.Collections.Immutable;
using System.ComponentModel;
using System.Diagnostics;
using System.IO;
using System.IO.Compression;
using System.Linq;
using System.Net;
using System.Net.Http;
using System.Net.Http.Headers;
using System.Reflection;
using System.Runtime.CompilerServices;
using System.Runtime.Serialization;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
using System.Web.Mvc;
using Jil;
using ProtoBuf;
using StackExchange.Profiling;
using StackOverflow.Models;
using static StackOverflow.Helpers.Http;

namespace StackOverflow.Helpers
{
    /// <summary>
    /// HTTP communications helper for making HttpRequests.
    /// </summary>
    public static class Http
    {
        public static class Settings
        {
            public static string UserAgent { get; set; } = "Stack Exchange Core (https://stackexchange.com)";

            internal static void OnBeforeSend(object sender, IRequestBuilder builder) => BeforeSend?.Invoke(sender, builder);
            internal static void OnException(object sender, HttpExceptionArgs args) => Exception?.Invoke(sender, args);

            internal static event EventHandler<IRequestBuilder> BeforeSend = (sender, builder) =>
            {
                var uri = builder.Message.RequestUri.ToString();
                // Set referer
                if (Current.Context != null && Current.Request != null && Site.IsInNetwork(uri))
                {
                    try { builder.Message.Headers.Referrer = Current.OriginalUrl(); }
                    catch { /* just don't set the referrer in this case */ }
                }

                var newUri = HtmlUtilities.SubstituteInternalUrl(uri);
                // add the private API key if needed
                if (newUri != uri)
                {
                    var key = "key=" + SiteSettings.Global.Api.InternalKey;
                    builder.Message.RequestUri = new Uri(!newUri.Contains(key) ? newUri + (newUri.Contains("?") ? "&" : "?") + key : newUri);
                }
            };

            internal static event EventHandler<HttpExceptionArgs> Exception = (sender, args) =>
            {
                // If we're in prod, don't log timeout exceptions (legacy behavior)
                if (Current.Tier == DeploymentTier.Prod && args.Error.Message.Contains("has timed out"))
                {
                    return;
                }

                GlobalApplication.LogException(args.Error);
            };
        }

        private static HttpClient GetClient(HttpClientOptions options) => ClientPool.GetOrAdd(options, CreateHttpClient);

        private static readonly ConcurrentDictionary<HttpClientOptions, HttpClient> ClientPool = new ConcurrentDictionary<HttpClientOptions, HttpClient>();

        private static HttpClient CreateHttpClient(HttpClientOptions options)
        {
            var handler = new HttpClientHandler
            {
                UseCookies = false
            };
            if (handler.SupportsAutomaticDecompression)
            {
                handler.AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip;
            }

            var client = new HttpClient(handler)
            {
                Timeout = options.Timeout,
                DefaultRequestHeaders =
                {
                    AcceptEncoding =
                    {
                        new StringWithQualityHeaderValue("gzip"),
                        new StringWithQualityHeaderValue("deflate")
                    }
                }
            };
            client.DefaultRequestHeaders.Add("User-Agent", Settings.UserAgent);
            return client;
        }

        public static void ClearPool() => ClientPool.Clear();

        /// <summary>
        /// Gets a new request at the specified URL.
        /// </summary>
        /// <param name="uri">The URI we're making a request to (this client takes care of .internal itself).</param>
        /// <returns>A chaining builder for your request.</returns>
        public static IRequestBuilder Request(
            string uri,
            [CallerMemberName] string callerName = null,
            [CallerFilePath] string callerFile = null,
            [CallerLineNumber] int callerLine = 0) => new HttpBuilder(uri, callerName, callerFile, callerLine);

        private static readonly FieldInfo stackTraceString = typeof(Exception).GetField("_stackTraceString", BindingFlags.Instance | BindingFlags.NonPublic);

        internal static async Task<HttpCallResponse<T>> SendAsync<T>(IRequestBuilder<T> builder, HttpMethod method, CancellationToken cancellationToken = default(CancellationToken))
        {
            Settings.OnBeforeSend(builder, builder.Inner);

            var request = builder.Inner.Message;
            request.Method = method;

            Exception exception = null;
            HttpResponseMessage response = null;
            try
            {
                using (Current.ProfileHttp(request.Method.Method, request.RequestUri.ToString()))
                using (request)
                {
                    // Send the request
                    using (response = await GetClient(builder.GetClientOptions()).SendAsync(request, cancellationToken))
                    {
                        if (!response.IsSuccessStatusCode && !builder.Inner.IgnoredResponseStatuses.Contains(response.StatusCode))
                        {
                            exception = new HttpClientException($"Response code was {(int)response.StatusCode} ({response.StatusCode}) from {response.RequestMessage.RequestUri}: {response.ReasonPhrase}");
                            stackTraceString.SetValue(exception, new StackTrace(true).ToString());
                        }
                        else
                        {
                            var data = await builder.Handler(response);
                            return HttpCallResponse.Create(response, data);
                        }
                    }
                }
            }
            catch (Exception ex)
            {
                exception = ex;
            }

            var result = default(HttpCallResponse<T>);
            if (response == null)
            {
                result = HttpCallResponse.Create<T>(request, exception);
            }
            else
            {
                result = HttpCallResponse.Create<T>(response, exception);
            }

            // If we're told not to log at all, don't log
            if (builder.Inner.LogErrors)
            {
                var args = new HttpExceptionArgs(builder.Inner, exception);
                builder.Inner.OnBeforeExceptionLog(args);

                if (!args.AbortLogging)
                {
                    Settings.OnException(builder, args);
                }
            }

            return result;
        }
    }

    public class HttpExceptionArgs
    {
        public IRequestBuilder Builder { get; }
        public Exception Error { get; }
        public bool AbortLogging { get; set; }

        public HttpExceptionArgs(IRequestBuilder builder, Exception ex)
        {
            Builder = builder;
            Error = ex;
        }
    }

    /// <summary>
    /// Struct for <see cref="ConcurrentDictionary{HttpClientOptions, HttpClient}"/> keying.
    /// </summary>
    public struct HttpClientOptions
    {
        public TimeSpan Timeout { get; }

        public HttpClientOptions(TimeSpan timeout)
        {
            Timeout = timeout;
        }
    }

    /// <summary>
    /// A request construct for building request options before issuing.
    /// </summary>
    public interface IRequestBuilder
    {
        [EditorBrowsable(EditorBrowsableState.Never)]
        HttpRequestMessage Message { get; }
        [EditorBrowsable(EditorBrowsableState.Never)]
        bool LogErrors { get; set; }
        [EditorBrowsable(EditorBrowsableState.Never)]
        IEnumerable<HttpStatusCode> IgnoredResponseStatuses { get; set; }
        [EditorBrowsable(EditorBrowsableState.Never)]
        TimeSpan Timeout { get; set; }
        [EditorBrowsable(EditorBrowsableState.Never)]
        event EventHandler<HttpExceptionArgs> BeforeExceptionLog;
        [EditorBrowsable(EditorBrowsableState.Never)]
        void OnBeforeExceptionLog(HttpExceptionArgs args);
        [EditorBrowsable(EditorBrowsableState.Never)]
        IRequestBuilder<T> WithHandler<T>(Func<HttpResponseMessage, Task<T>> handler);
    }

    /// <summary>
    /// A typed request construct for building request options before issuing.
    /// </summary>
    /// <typeparam name="T">The type this request will return.</typeparam>
    public interface IRequestBuilder<T>
    {
        [EditorBrowsable(EditorBrowsableState.Never)]
        IRequestBuilder Inner { get; }
        [EditorBrowsable(EditorBrowsableState.Never)]
        Func<HttpResponseMessage, Task<T>> Handler { get; }
        [EditorBrowsable(EditorBrowsableState.Never)]
        HttpClientOptions GetClientOptions();
    }

    internal class HttpBuilder : IRequestBuilder
    {
        public HttpRequestMessage Message { get; }
        public bool LogErrors { get; set; } = true;
        public IEnumerable<HttpStatusCode> IgnoredResponseStatuses { get; set; } = Enumerable.Empty<HttpStatusCode>();
        public TimeSpan Timeout { get; set; } = TimeSpan.FromSeconds(3); // We've defaulted to 3 seconds since forever
        public event EventHandler<HttpExceptionArgs> BeforeExceptionLog;
        private readonly string _callerName, _callerFile;
        private readonly int _callerLine;

        public HttpBuilder(string uri, string callerName, string callerFile, int callerLine)
        {
            Message = new HttpRequestMessage
            {
                RequestUri = new Uri(uri, UriKind.RelativeOrAbsolute)
            };
            _callerName = callerName;
            _callerFile = callerFile;
            _callerLine = callerLine;
        }

        public void OnBeforeExceptionLog(HttpExceptionArgs args)
        {
            args.Error?.AddLoggedData("Caller.Name", _callerName)
                       .AddLoggedData("Caller.File", _callerFile)
                       .AddLoggedData("Caller.Line", _callerLine.ToString());
            BeforeExceptionLog?.Invoke(this, args);
        }

        public IRequestBuilder<T> WithHandler<T>(Func<HttpResponseMessage, Task<T>> handler) => new HttpBuilder<T>(this, handler);
    }

    internal class HttpBuilder<T> : IRequestBuilder<T>
    {
        public IRequestBuilder Inner { get; }
        public HttpClientOptions GetClientOptions() => new HttpClientOptions(Inner.Timeout);
        public Func<HttpResponseMessage, Task<T>> Handler { get; }

        public HttpBuilder(HttpBuilder builder, Func<HttpResponseMessage, Task<T>> handler)
        {
            Inner = builder;
            Handler = handler;
        }
    }

    public class HttpCallResponse
    {
        public bool Success { get; }
        public string RequestUri { get; }
        public HttpRequestMessage RawRequest { get; }
        public HttpResponseMessage RawResponse { get; }
        public Exception Error { get; }
        public HttpStatusCode? StatusCode { get; }

        protected HttpCallResponse(HttpRequestMessage request, Exception error)
        {
            Success = false;
            RequestUri = request.RequestUri.AbsoluteUri;
            Error = error;
        }

        protected HttpCallResponse(HttpResponseMessage response)
        {
            Success = response.IsSuccessStatusCode;
            StatusCode = response.StatusCode;
            RequestUri = response.RequestMessage.RequestUri.AbsoluteUri;
            RawResponse = response;
            RawRequest = response.RequestMessage;
        }

        protected HttpCallResponse(HttpResponseMessage response, Exception error) : this(response)
        {
            Success = false;
            Error = error;
        }

        public static HttpCallResponse<T> Create<T>(HttpRequestMessage request, Exception error = null)
        {
            error = (error ?? new HttpClientException("Failed to send request for " + request.RequestUri))
                // Add these regardless of source
                .AddLoggedData("Request URI", request.RequestUri);

            return new HttpCallResponse<T>(request, error);
        }

        public static HttpCallResponse<T> Create<T>(HttpResponseMessage response, Exception error)
        {
            // Add these regardless of source
            error.AddLoggedData("Response.Code", ((int)response.StatusCode).ToString())
                 .AddLoggedData("Response.Status", response.StatusCode.ToString())
                 .AddLoggedData("Response.ReasonPhrase", response.ReasonPhrase)
                 .AddLoggedData("Response.ContentType", response.Content.Headers.ContentType)
                 .AddLoggedData("Request.URI", response.RequestMessage.RequestUri);

            return new HttpCallResponse<T>(response, error);
        }

        public static HttpCallResponse<T> Create<T>(HttpResponseMessage response, T data)
        {
            return new HttpCallResponse<T>(response, data);
        }
    }

    public class HttpCallResponse<T> : HttpCallResponse
    {
        public T Data { get; }

        public HttpCallResponse(HttpResponseMessage response, T data) : base(response)
        {
            Data = data;
        }

        public HttpCallResponse(HttpRequestMessage request, Exception error) : base(request, error) { }

        public HttpCallResponse(HttpResponseMessage response, Exception error) : base(response, error) { }
    }

    /// <summary>
    /// Extensions for sending - named this way to not show in Intellisense
    /// </summary>
    public static class SendExtensionsForHttp
    {
        /// <summary>
        /// Sets the given <see cref="HttpContent"/> as the body for this request.
        /// </summary>
        /// <param name="builder">The builder we're working on.</param>
        /// <param name="content">The <see cref="HttpContent"/> to use.</param>
        /// <returns>The request builder for chaining.</returns>
        public static IRequestBuilder SendContent(this IRequestBuilder builder, HttpContent content)
        {
            builder.Message.Content = content;
            return builder;
        }

        /// <summary>
        /// Adds a <see cref="FormCollection"/> as the body for this request.
        /// </summary>
        /// <param name="builder">The builder we're working on.</param>
        /// <param name="form">The <see cref="FormCollection"/> to use.</param>
        /// <returns>The request builder for chaining.</returns>
        public static IRequestBuilder SendForm(this IRequestBuilder builder, FormCollection form) =>
            SendContent(builder, new FormUrlEncodedContent(form.AllKeys.ToDictionary(k => k, v => form[v])));

        /// <summary>
        /// Adds raw HTML content as the body for this request.
        /// </summary>
        /// <param name="builder">The builder we're working on.</param>
        /// <param name="html">The raw HTML string to use.</param>
        /// <returns>The request builder for chaining.</returns>
        public static IRequestBuilder SendHtml(this IRequestBuilder builder, string html) =>
            SendContent(builder, new StringContent(html, Encoding.UTF8, "text/html"));

        /// <summary>
        /// Adds JSON (Jil-serialized) content as the body for this request.
        /// </summary>
        /// <param name="builder">The builder we're working on.</param>
        /// <param name="obj">The object to serialize as JSON in the body.</param>
        /// <param name="jsonOptions">The Jil options to use when serializing.</param>
        /// <returns>The request builder for chaining.</returns>
        public static IRequestBuilder SendJson(this IRequestBuilder builder, object obj, Options jsonOptions = null) =>
            SendContent(builder, new StringContent(JSON.Serialize(obj, jsonOptions ?? Options.Default), Encoding.UTF8, "application/json"));

        /// <summary>
        /// Adds raw text content as the body for this request.
        /// </summary>
        /// <param name="builder">The builder we're working on.</param>
        /// <param name="text">The raw text string to use.</param>
        /// <returns>The request builder for chaining.</returns>
        public static IRequestBuilder SendPlaintext(this IRequestBuilder builder, string text) =>
            SendContent(builder, new StringContent(text, Encoding.UTF8, "application/x-www-form-urlencoded"));

        /// <summary>
        /// Adds protobuf-serialized content as the body for this request.
        /// </summary>
        /// <param name="builder">The builder we're working on.</param>
        /// <param name="obj">The object to serialize with protobuf in the body.</param>
        /// <returns>The request builder for chaining.</returns>
        public static IRequestBuilder SendProtobuf(this IRequestBuilder builder, object obj)
        {
            using (var output = new MemoryStream())
            using (var gzs = new GZipStream(output, CompressionMode.Compress))
            {
                Serializer.Serialize(gzs, obj);
                gzs.Close();
                var protoContent = new ByteArrayContent(output.ToArray());
                protoContent.Headers.Add("Content-Type", "application/x-www-form-urlencoded");
                protoContent.Headers.Add("Content-Encoding", "gzip");
                return SendContent(builder, protoContent);
            }
        }
    }

    /// <summary>
    /// Extensions for modifiers - named this way to not show in Intellisense
    /// </summary>
    public static class ModifierExtensionsForHttp
    {
        /// <summary>
        /// Sets a timeout for this request.
        /// </summary>
        /// <param name="builder">The builder we're working on.</param>
        /// <param name="timeout">The timeout to use on this request.</param>
        /// <returns>The request builder for chaining.</returns>
        /// <remarks>
        /// This isn't *really* per request since it's global on <see cref="HttpClient"/>,
        /// so in reality we grab a different client from the pool.
        /// </remarks>
        public static IRequestBuilder WithTimeout(this IRequestBuilder builder, TimeSpan timeout)
        {
            builder.Timeout = timeout;
            return builder;
        }

        /// <summary>
        /// Disables logging errors to the exceptional log on this request.
        /// </summary>
        /// <param name="builder">The builder we're working on.</param>
        /// <returns>The request builder for chaining.</returns>
        public static IRequestBuilder WithoutErrorLogging(this IRequestBuilder builder)
        {
            builder.LogErrors = false;
            return builder;
        }

        /// <summary>
        /// Doesn't log an error when the response's HTTP status code is any of the <paramref name="ignoredStatusCodes"/>.
        /// </summary>
        /// <param name="builder">The builder we're working on.</param>
        /// <param name="ignoredStatusCodes">HTTP status codes to ignore.</param>
        /// <returns>The request builder for chaining.</returns>
        public static IRequestBuilder WithoutLogging(this IRequestBuilder builder, IEnumerable<HttpStatusCode> ignoredStatusCodes)
        {
            builder.IgnoredResponseStatuses = ignoredStatusCodes;
            return builder;
        }

        private static readonly ConcurrentDictionary<HttpStatusCode, ImmutableHashSet<HttpStatusCode>> _ignoreCache = new ConcurrentDictionary<HttpStatusCode, ImmutableHashSet<HttpStatusCode>>();
        /// <summary>
        /// Doesn't log an error when the response's HTTP status code is <paramref name="ignoredStatusCode"/>.
        /// </summary>
        /// <param name="builder">The builder we're working on.</param>
        /// <param name="ignoredStatusCode">HTTP status code to ignore.</param>
        /// <returns>The request builder for chaining.</returns>
        public static IRequestBuilder WithoutLogging(this IRequestBuilder builder, HttpStatusCode ignoredStatusCode)
        {
            builder.IgnoredResponseStatuses = _ignoreCache.GetOrAdd(ignoredStatusCode, k => ImmutableHashSet.Create(k));
            return builder;
        }

        /// <summary>
        /// Adds an event handler for this request, for appending additional information to the logged exception for example.
        /// </summary>
        /// <param name="builder">The builder we're working on.</param>
        /// <param name="beforeLogHandler">The exception handler to run before logging</param>
        /// <returns>The request builder for chaining.</returns>
        public static IRequestBuilder OnException(this IRequestBuilder builder, EventHandler<HttpExceptionArgs> beforeLogHandler)
        {
            builder.BeforeExceptionLog += beforeLogHandler;
            return builder;
        }

        public static IRequestBuilder WithAcceptHeader(this IRequestBuilder builder, string accept) => builder.AddHeader("Accept", accept);

        /// <summary>
        /// Add a header to this request.
        /// </summary>
        /// <param name="builder">The builder we're working on.</param>
        /// <param name="name">The header name to add to this request.</param>
        /// <param name="value">The header value (for <paramref name="name"/>) to add to this request.</param>
        /// <returns>The request builder for chaining.</returns>
        public static IRequestBuilder AddHeader(this IRequestBuilder builder, string name, string value)
        {
            if (name.HasValue())
            {
                try
                {
                    builder.Message.Headers.Add(name, value);
                }
                catch (Exception e)
                {
                    var wrapper = new HttpClientException("Unable to set header: " + name + " to '" + value + "'", e);
                    Settings.OnException(builder, new HttpExceptionArgs(builder, wrapper));
                }
            }
            return builder;
        }

        /// <summary>
        /// Adds headers to this request.
        /// </summary>
        /// <param name="builder">The builder we're working on.</param>
        /// <param name="headers">The headers to add to this request.</param>
        /// <returns>The request builder for chaining.</returns>
        public static IRequestBuilder AddHeaders(this IRequestBuilder builder, IDictionary<string, string> headers)
        {
            if (headers == null) return builder;

            var pHeaders = builder.Message.Headers;
            foreach (var kv in headers)
            {
                try
                {
                    //pHeaders.Add(kv.Key, kv.Value);
                    switch (kv.Key)
                    {
                        //    certain headers must be accessed via the named property on the WebRequest
                        case "Accept": pHeaders.Accept.ParseAdd(kv.Value); break;
                        //    case "Connection": break;
                        //    case "proxy-connection": break;
                        //    case "Proxy-Connection": break;
                        //    case "Content-Length": break;
                        case "Content-Type": builder.Message.Content.Headers.ContentType = new MediaTypeHeaderValue(kv.Value); break;
                        //    case "Host": break;
                        //    case "If-Modified-Since": pHeaders.IfModifiedSince = DateTime.ParseExact(kv.Value, "R", CultureInfo.InvariantCulture); break;
                        //    case "Referer": pHeaders.Referrer = new Uri(kv.Value); break;
                        //    case "User-Agent": pHeaders.UserAgent.ParseAdd("Stack Exchange (Proxy)"); break;
                        default: pHeaders.Add(kv.Key, kv.Value); break;
                    }
                }
                catch (Exception e)
                {
                    var wrapper = new HttpClientException("Unable to set header: " + kv.Key + " to '" + kv.Value + "'", e);
                    Settings.OnException(builder, new HttpExceptionArgs(builder, wrapper));
                }
            }
            return builder;
        }
    }

    /// <summary>
    /// Extensions for result expectations - named this way to not show in Intellisense
    /// </summary>
    public static class ExpectExtensionsForHttp
    {
        /// <summary>
        /// Sets the response handler for this request to a <see cref="bool"/> (200-299 response code).
        /// </summary>
        /// <param name="builder">The builder we're working on.</param>
        /// <returns>A typed request builder for chaining.</returns>
        public static IRequestBuilder<bool> ExpectBool(this IRequestBuilder builder) =>
            builder.WithHandler(responseMessage => Task.FromResult(responseMessage.IsSuccessStatusCode));

        /// <summary>
        /// Holds handlers for ExpectJson(T) calls, so we don't re-create them in the common "default Options" case.
        ///
        /// Without this, we create a new Func for each ExpectJson call even
        /// </summary>
        private static class JsonHandler<T>
        {
            internal static readonly Func<HttpResponseMessage, Task<T>> Default = WithOptions(Options.Default);

            internal static Func<HttpResponseMessage, Task<T>> WithOptions(Options jsonOptions)
            {
                return async responseMessage =>
                {
                    using (var responseStream = await responseMessage.Content.ReadAsStreamAsync()) // Get the response here
                    using (var streamReader = new StreamReader(responseStream))                    // Stream reader
                    using (MiniProfiler.Current.Step("JSON Deserialize"))
                    {
                        return JSON.Deserialize<T>(streamReader, jsonOptions ?? Options.Default);
                    }
                };
            }
        }

        /// <summary>
        /// Sets the response handler for this request to a JSON deserializer.
        /// </summary>
        /// <typeparam name="T">The type to Jil-deserialize to.</typeparam>
        /// <param name="builder">The builder we're working on.</param>
        /// <param name="jsonOptions">The Jil options to use when serializing.</param>
        /// <returns>A typed request builder for chaining.</returns>
        public static IRequestBuilder<T> ExpectJson<T>(this IRequestBuilder builder, Options jsonOptions = null)
        {
            if (jsonOptions == null) return builder.WithHandler(JsonHandler<T>.Default);

            return builder.WithHandler(JsonHandler<T>.WithOptions(jsonOptions));
        }

        /// <summary>
        /// Sets the response handler for this request to a protobuf deserializer.
        /// </summary>
        /// <typeparam name="T">The type to protobuf-deserialize to.</typeparam>
        /// <param name="builder">The builder we're working on.</param>
        /// <returns>A typed request builder for chaining.</returns>
        public static IRequestBuilder<T> ExpectProtobuf<T>(this IRequestBuilder builder) =>
            builder.WithHandler(async responseMessage =>
            {
                using (var responseStream = await responseMessage.Content.ReadAsStreamAsync())
                using (MiniProfiler.Current.Step("Protobuf Deserialize"))
                {
                    return Serializer.Deserialize<T>(responseStream);
                }
            });

        /// <summary>
        /// Sets the response handler for this request to return the response as a <see cref="byte[]"/>.
        /// </summary>
        /// <typeparam name="T">The type to protobuf-deserialize to.</typeparam>
        /// <param name="builder">The builder we're working on.</param>
        /// <returns>A typed request builder for chaining.</returns>
        public static IRequestBuilder<byte[]> ExpectByteArray(this IRequestBuilder builder) =>
            builder.WithHandler(responseMessage => responseMessage.Content.ReadAsByteArrayAsync());

        /// <summary>
        /// Sets the response handler for this request to return the response as a <see cref="string"/>.
        /// </summary>
        /// <param name="builder">The builder we're working on.</param>
        /// <returns>A typed request builder for chaining.</returns>
        public static IRequestBuilder<string> ExpectString(this IRequestBuilder builder) =>
            builder.WithHandler(async responseMessage =>
            {
                using (var responseStream = await responseMessage.Content.ReadAsStreamAsync())
                using (var streamReader = new StreamReader(responseStream))
                {
                    return await streamReader.ReadToEndAsync();
                }
            });
    }

    /// <summary>
    /// Extensions for actual execution - named this way to not show in Intellisense
    /// </summary>
    public static class VerbExtensionsForHttp
    {
        /// <summary>
        /// Issue the request as a DELETE.
        /// </summary>
        /// <typeparam name="T">The return type.</typeparam>
        /// <param name="builder">The builder used for this request.</param>
        /// <param name="cancellationToken">The cancellation token for stopping the request.</param>
        /// <returns>A <see cref="HttpCallResponse{T}"/> to consume.</returns>
        public static Task<HttpCallResponse<T>> DeleteAsync<T>(this IRequestBuilder<T> builder, CancellationToken cancellationToken = default(CancellationToken)) =>
            SendAsync(builder, HttpMethod.Delete, cancellationToken);

        /// <summary>
        /// Issue the request as a GET.
        /// </summary>
        /// <typeparam name="T">The return type.</typeparam>
        /// <param name="builder">The builder used for this request.</param>
        /// <param name="cancellationToken">The cancellation token for stopping the request.</param>
        /// <returns>A <see cref="HttpCallResponse{T}"/> to consume.</returns>
        public static Task<HttpCallResponse<T>> GetAsync<T>(this IRequestBuilder<T> builder, CancellationToken cancellationToken = default(CancellationToken)) =>
            SendAsync(builder, HttpMethod.Get, cancellationToken);

        /// <summary>
        /// Issue the request as a POST.
        /// </summary>
        /// <typeparam name="T">The return type.</typeparam>
        /// <param name="builder">The builder used for this request.</param>
        /// <param name="cancellationToken">The cancellation token for stopping the request.</param>
        /// <returns>A <see cref="HttpCallResponse{T}"/> to consume.</returns>
        public static Task<HttpCallResponse<T>> PostAsync<T>(this IRequestBuilder<T> builder, CancellationToken cancellationToken = default(CancellationToken)) =>
            SendAsync(builder, HttpMethod.Post, cancellationToken);

        /// <summary>
        /// Issue the request as a PUT.
        /// </summary>
        /// <typeparam name="T">The return type.</typeparam>
        /// <param name="builder">The builder used for this request.</param>
        /// <param name="cancellationToken">The cancellation token for stopping the request.</param>
        /// <returns>A <see cref="HttpCallResponse{T}"/> to consume.</returns>
        public static Task<HttpCallResponse<T>> PutAsync<T>(this IRequestBuilder<T> builder, CancellationToken cancellationToken = default(CancellationToken)) =>
            SendAsync(builder, HttpMethod.Put, cancellationToken);
    }

    public class HttpClientException : Exception
    {
        public HttpClientException() { }
        public HttpClientException(string message) : base(message) { }
        public HttpClientException(string message, Exception innerException) : base(message, innerException) { }        
        protected HttpClientException(SerializationInfo info, StreamingContext context) : base(info, context) { }
    }
}

Really enjoying this syntax.

There is an issue with SendAsync success path always returning a HttpCallResponse with Success set to false and a false error message. I removed the optional error parameter from the HttpCallResponse constuctor and added an overload public HttpCallResponse(HttpResponseMessage response) : base(response) { } to prevent overwriting the Success value in the base constructor.