How do you handle JDK/JRE patch updates for Java apps on K8s?

I’m curious how people running Java workloads on Kubernetes handle JDK/JRE updates and security patches without rebuilding every app image.

Background: in Mesos (https://en.wikipedia.org/wiki/Apache_Mesos) times, we used to keep the JDK on the runner nodes. When a CVE or patch came out, we updated the host JDK, and all apps picked it up. That was convenient for fast security rollouts. On k8s, almost everyone I see bakes the JDK into the container image, which means: new JDK → rebuild base image → rebuild app images (or at least rebuild base) → push → roll out. That is reliable and reproducible, but it is impossible to update the JDK version for, for example, 2000 apps quickly.

Questions I have for people who run Java on k8s at scale:

Do you rebuild images for every JDK patch?

If so, how do you keep the pipeline fast/automated?

What approaches we talked about (still looking for something better):

- Rebuild images on every JDK patch (CI pipeline that automatically bumps base image + rebuilds): reproducible but heavy and slow.

- Host-provided JDK (like Mesos) via hostPath or a shared volume (every path version must be available): fast patches, but brittle (node drift, version chaos between k8s nodes, less reproducible, potential security/permission problems).

- Base, standard image for all java apps (alpine+java) that our platform updates and init container downloading user app on startup, so that we can update it in the background.

- Sidecar or init-container that places a JDK into a shared volume, and the app container uses that volume: mutable runtime without rebuilding images — how well does this work in practice?

6 points

・

bgalek

・

3 days ago

5 comments

matt_s ・ 3 days ago

There's no way getting around needing to install the patched versions of the compiler or runtime. Someone has to do it and in my opinion it should be done internally so its controlled and has verified/known sources.

I think option/bullet 2 keeps things as streamlined as possible - base standard image for java apps that apps then use as their base when they build their container images to run in k8s containers. Maybe there's a way to tag base images with major.minor so that apps just pull that image and if its cached its faster and if there was a major.minor.patch just applied and you're the first app to deploy then that app pays the penalty of waiting for the underlying base image to be rebuilt.

A volume is an interesting idea but are there possible issues for OS library dependencies like openssl, database drivers, etc. that might not be the right version for a JDK mounted on a shared volume?

comprev ・ 3 days ago

A patch is a change and for changes to be deployed they must go through the pipelines.

The new images are then rolled out in exactly the same controlled manner as any other new release.

SamInTheShell ・ 3 days ago

Java is the worse language to deal with in Kubernetes. You might want to look into Quarkus. I’m of the opinion that ditching Java for Go is better if possible.

  • 0x54MUR41 ・ 3 days ago

    Why did you say so? Do you mind to elaborate more?

    • SamInTheShell ・ 3 days ago

      Just my experience, but slow start times, high memory usage, tracking down GC pauses, getting the memory flags in line with the k8s resources config, and fat image sizes. All things I enjoy living without.

      Also regarding Quarkus, that's java stuff. I don't use it, but I know people do use it for shortcomings of Java in k8s. Perhaps GraalVM is the thing that makes it worth-while, idk.

      There's also a distroless Java container from Google that could be of value.