Skip to content

Instantly share code, notes, and snippets.

@osy
osy / README.md
Last active January 26, 2024 18:18
Local caching for GitHub Actions self hosted runner using Squid Proxy

One of the biggest issues with using a self hosted GitHub runner is that actions that require downloading large amounts of data will bottleneck at the network. [actions/cache][1] does not support locally caching objects and artifacts stored on GitHub's servers will require a lot of bandwidth to fetch on every job. We can, however, set up a content proxy using [Squid][2] with SSL bumping to locally cache requests from jobs.

Patching Squid

A major challenge is that [actions/cache][1] uses Azure storage APIs which makes HTTP range requests. While Squid supports range requests, it is not good at caching them. There is an option, range_offset_limit none which, according to the [documentation][3]:

A size of 'none' causes Squid to always fetch the object from the beginning so it may cache the result. (2.0 style)

However, after extensive debugging, I discovered that the feature does not work if the client closes the connection. When range_offset_limit is set, Squid will make a full request to the server,

@michael1011
michael1011 / cmds.go
Created July 11, 2023 09:10
LND Compactor
package main
import (
"encoding/binary"
"errors"
"fmt"
"github.com/urfave/cli/v2"
"os"
"strconv"
"time"