Mitigate the Docker Dilemma with a Proxy Cache

Mitigate the Docker Dilema with a Proxy Cache

The Docker Dilemma

For the better part of a decade, we developers and operators have relied on the Docker Hub and their image repository for our needs. This has been great for the growth of the technology, but this growth has necessitated a change to how Docker makes money. As such, Docker is rate limiting anonymous and authenticated users.

For the average individual developer who is building software, there is likely little impact to their daily work. But for those of us running Kubernetes and applications at scale, this change will likely break our larger clusters, especially those that use the Always ImagePullPolicy.

As organizations that are air gapping their setups, this is of little consequence. They're already accustomed to regularly updating their internal repositories. But, for those of us who need to, we need a way to quickly and easily serve images to our clusters and developers and limit the amount of traffic to Docker Hub. A proxy cache is a great tool to effectively accomplish this goal.

What is a Proxy Cache?

An image proxy cache is a feature of many container image repository server technologies. It allows servers and developers to pull images from a centralized location, and uses upstream connections to upstream image repositories such as Docker Hub to make sure the local copy of the image is up to date. If it is, no pull request is made to the upstream repository, so the server or developer's pull does not count against the upstream servers rate limit. If the upstream image has changed, it is add pulled once to the local proxy cache and then delivered to the server or developer requesting it.

Setting up a proxy cache is technology specific and outside the scope of this blog post, but know there are many options available. Here are a few examples.

At this point, it's just a matter of setting up your Kubernetes clusters to use the a private repository. Let's take a look at how to do that with RKE, K3s and RKE Government.

Setup RKE to Use an Image Proxy Cache

Per the RKE Documentation you'll use the private_registries setting to set up each private registry by its url. An example of this might look like this.

private_registries:
    - url: mydockermirror.local
      user: username
      password: password
    - url: myquaymirror.local
      user: myusername
      password: mypassword

If you're using the Terraform RKE Provider, you would use the private_registries argument of the cluster resource.

The same example from above would look like this.

  private_registries {
    url      = "mydockermirror.local"
    user     = "username"
    password = "password"
  }
  private_registries {
    url      = "myquaymirror.local"
    user     = "myusername"
    password = "mypassword"
  }

Set Up K3s to Use an Image Proxy Cache

Per the K3s Documentation, each node in a K3s cluster will need a registries.yaml configuration file at /etc/rancher/k3s/registries.yaml. This includes both the server and the agent, if the server is not tainted to prevent deployments. This registries file allows for multiple endpoints per url. This allows you to specify the local registry, but you could include the public endpoint as a backup. In addition, if the local registry is secured via TLS, you may need to specify certificates and use https as part of the URL. Here's the above registry example, configured for K3s. We'll assume the quay mirror is unsecured for this example.

mirrors:
  docker.io:
    endpoint:
      - "https://mydockermirror.local"
      - "https://docker.io"
  quay.io:
    endpoint:
      - "http://myquaymirror.local"
      - "https://quay.io"
configs:
  "mydockermirror.local":
    auth:
      username: username 
      password: password
    tls:
      cert_file: /mycerts/client.crt
      key_file:  /mycerts/clientkey.crt
      ca_file:   /mycerts/ca.crt
  "docker.io":
    auth:
      username: dockerhubusername
      password: dockerhubpassword
  "myquaymirror.local":
    auth:
      username: myusername 
      password: mypassword
  "quay.io":
    auth:
      username: quayiousername
      password: quayiopassword

Set Up RKE Government to Use an Image Proxy Cache

Per the RKE Government/Documentation, the method is nearly identical to the method for K3s. Each node in an RKE Government cluster will need a registries.yaml configuration file at /etc/rancher/rke2/registries.yaml. Like a K3s cluster, if the server is not appropriately tainted, then this node will also require a registries file. This registries file allows for multiple endpoints per url. This allows you to specify the local registry, but you could include the public endpoint as a backup. In addition, if the local registry is secured via TLS, you may need to specify certificates and use https as part of the URL. Here's the above registry example, as it would look for RKE Government. Again, we'll assume the quay mirror is unsecured for this example.

mirrors:
  docker.io:
    endpoint:
      - "https://mydockermirror.local"
      - "https://docker.io"
  quay.io:
    endpoint:
      - "http://myquaymirror.local"
      - "https://quay.io"
configs:
  "mydockermirror.local":
    auth:
      username: username 
      password: password
    tls:
      cert_file: /mycerts/client.crt
      key_file:  /mycerts/clientkey.crt
      ca_file:   /mycerts/ca.crt
  "docker.io":
    auth:
      username: dockerhubusername
      password: dockerhubpassword
  "myquaymirror.local":
    auth:
      username: myusername 
      password: mypassword
  "quay.io":
    auth:
      username: quayiousername
      password: quayiopassword

Conclusion

Using these methods alongside highly available image repositories may prevent you from over running your image repository rate limits and give you more control of your upstream container image dependencies.