I have 2 SmartOS servers at home with a total of 17 zones
running persistently - a majority of these provisioned with the latest
LTS support release
of pkgsrc: 2014Q4. All zones are setup to download a similar set of "bootstrap"
packages to make them ready to be used. Packages like
git, etc. I pull to all
of my zones upon creation.
The problem, however, is my home internet is not the fastest - it usually can pull
packages at around 100KB/s.
gcc alone are responsible for over
100MB of compiled binary data, meaning these initial package downloads can take
over 15 minutes per new zone.
A possible solution to speed up initial package downloads I investigated was to
rsync the entire pkgsrc tree to a local server, and host it over an internal
HTTP server. This, however, required a massive amount of storage dedicated
to packages I am almost guaranteed to never use. I could have just pulled
2014Q4, but then I would have been required to add new releases whenever I
wanted to upgrade, and keep around the old packages until all my old zones have
Instead, I created a Node.JS HTTP caching server to proxy
HEAD requests to Joyent's pkgsrc server, and cache the data on the
local filesystem. The requests will be proxied if the file does not exist locally,
and will be dual-written to the requesting client, and the local filesystem. Subsequent
requests for the same resource will be streamed from the local filesystem without
every making an outbound request.
The program is written to be generic - it allows it to act as a caching
proxy to any website, and also allows the user to specify a regex to test a URL when
deciding if the request should be cached, or just proxied directly with no cache.