gistfile1.md

S3/CloudFront problems

• CloudFront: updates are very slow, taking about 20 - 30 minutes each time you try to change something to fix some of the things listed here
• CloudFront: no support for Cache-Control: stale-while-revalidate (would be nice for index.html etc. that can't be hashed)
• CloudFront: does not forward the original host name to a custom origin (would be useful for creating a S3 proxy origin to fix some of the issues here, instead of trying to implement them w/ lambda@edge)
• S3: Vary: Origin header is not added for non-CORS requests    * this is needed to fix a browser caching issue where if you load an image via <img src="...">, the browser will cache the response to that. Then, when you do cross origin XHR to the same object to receive metadata, the browsers will use the cached response and fail, because it didn't contain Access-Control-Allow-Origin header for the original request made by <img src="..."> - and because there was no Vary: Origin, browser doesn't think it could get a different response from the server by redoing the request with the Origin header
• CloudFront: needs lambda@edge pretty much for everything:
  • to support www redirects
  • to support any custom headers (Strict-Transport-Security, Content-Security-Policy, X-Frame-Options, ...)
  • to support rewriting requests from "/foo" to "/foo/index.html" (CloudFront only supports that for the root)
  • to support basic auth
• CloudFront: issues with Range requests:
  • CloudFront does not cache the full object to the edge when client does a Range request
  • Range: 0-100 comes in, CloudFront fetches bytes 0 - 100 from origin
  • Range: 100-200 comes in, CloudFront fetches bytes 100 - 200 from origin
  • This has very bad performance if the origin is in us-east-1 and CloudFront edge is in Tokyo for example
  • Adaptive bitrate video playback doesn't work well with this behaviour, because when it sees that downloading some part is slow, it'll lower the quality. So, when the player asks for Range: 0 - 100 of a high quality stream, it decides it was too slow coming and downgrades quality (usually all the way down to the lowest quality). But then when the next user comes, Range: 0 - 100 of the high quality stream is in the edge, so it tries loading Range: 100 - 200 which is again slow, so the next user will get downgraded to low quality as well. This goes on until enough users have tried to view the high quality stream. If there's 5 quality versions and 1000 parts per version, you need 5000 users per CloudFront edge to view the video from beginning to finish to get everything to the edge location
    • Fixable by having the content replicated to many locations around the world
    • Or by prewarming the edges by mimicking the Range requests the player would make (hard mode)