DevOps

The biggest footgun I encounter every time I set up a raspberry pi or other linux host for a side project is forgetting that Docker doesn't do log rotation for containers' logs by default, which results in the service going down and seeing a sweat inducing ENOSPC error when you ssh in to check it out.

You can configure this by creating /etc/docker/daemon.json and either setting up log rotation with log-opts or using the local logging driver (it defaults to json) if you're not shipping container logs anywhere and just read the logs locally. The local driver compresses the logs and automatically does log rotation:

{
  "log-driver": "local",
  "log-opts": {
     "max-size": "10m",
     "max-file": "3"
  }
}

A lot of people don't don't realize that the port configuration of 127.0.0.1:8080/ localhost:8080 is different than just 8080. One binds to localhost, meaning remote machines can't access it and the other does not.Combine this with the fact that people don't realize docker bypasses firewalls like ufw or firewalld, editing ip/nftables rules directly...

If you use an ip/port search engine like shodan or zoomeye, you can find loads of docker containers that probably weren't supposed to be on the net.

Using Docker Desktop at work without a license. Use Rancher Desktop instead. It's essentially like what Oracle did with the Oracle JDK. To my knowledge they haven't gone after anyone but it is technically a license violation to use it for work without a license. I could not (easily) find a way to install Docker on Mac without using Docker Desktop but Rancher Desktop worked fine.

Also, podman exists as a drop in replacement for Docker for the curious. I haven't tried it myself though so this isn't a recommendation.

I still don't really know how to get mounted folders to not wreck permission on the files. You can use a non root user, but that requires users have UID 1000 when you distribute an image.

Be really careful when building images that require secrets for build configuration. Secrets can be passed in as build args, but you MUST UNSET THEM IN THE DOCKERFILE and then repass them in as environment variables at runtime (or else you are leaking your secrets with your image).

Also, image != container. Image is the thing you publish to a registry (e.g. dockerhub). Container is an instance of an image.

Spending hours and hours debugging an issue where files weren’t being written before finally realising I was looking in my host file system not the container… fml.

Forgetting about SELinux.

Early in the history of docker, a lot of bits and bobs hadn't been worked out yet, and I had a bug land on my desk where a service was leaking memory until it crashed, but only when running in a container. Turns out, the jvm at the time just never collected in a container because the /proc directory was mounted from the host rather than the k8s scheduler. So it would only collect if it did not receive a second allocation request during the GC.