Introducing StableBuild – freeze and pin all dependencies for enhanced stability.

Launching StableBuild – freeze and pin all your dependencies

StableBuild has introduced a new set of tools aimed at assisting developers in creating reliable and deterministic builds. This is achieved by providing the ability to effortlessly freeze and pin Docker images, operating system packages, Python packages, and other essential build dependencies.

‍Docker and deterministic builds

StableBuild has facilitated the process of shipping application code through Docker, enabling developers to package their applications along with all necessary dependencies, including the operating system and Python packages, into a single deployable container. Dockerfiles provide a clear specification on how to build the application, which appears deterministic at a glance, as the same Dockerfile yields the same Docker container. However, in practice, this determinism is compromised by reliance on numerous mirrors, package registries, and repositories, any of which can undergo changes unpredictably. Here’s an illustration:

# Base this Dockerfile on Ubuntu 20.04
FROM ubuntu:20.04

# Install some OS level packages
RUN apt update && apt install -y curl software-properties-common

# We need a newer Python version, grab it from Deadsnakes PPA
RUN add-apt-repository -y ppa:deadsnakes/ppa && \
    apt update && \
    apt install -y python3.9 python3.9-distutils

# Install pip (the Python package manager)
RUN curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py && \
    python3.9 get-pip.py

# Install a Python package through pip
RUN pip3 install onnx==1.14.0

This Dockerfile relies on five separate services:

1. It fetches a base image from Docker Hub, where images are not immutable. This means the base image could be replaced with a new OS version or removed entirely.
2. It installs packages via apt from the Ubuntu package registry, which is regularly updated. Consequently, it will install the latest available versions of curl and software-properties-common.
3. It obtains a newer Python version from a third-party package registry (PPA), which is also subject to frequent updates. Therefore, it will fetch the latest available Python 3.9.x release, with no assurance that the PPAs will remain accessible.
4. It downloads get-pip.py from the internet, where the URL might return a different file or be taken down at any time.
5. It utilizes pip to install packages from the Python package registry. Although a specific version is specified for onnx, onnx relies on protobuf>=3.20.2, meaning it will install the latest version of protobuf available.

The situation arises where the same Dockerfile can generate significantly different containers depending on when it’s built. For instance, today the Dockerfile is used to build and test the application successfully. However, rebuilding the container tomorrow, even from the same Dockerfile (for example, due to code updates or additional dependencies), may result in a new OS version, updated package versions from apt, a different Python version, and potentially unresolved URLs for Python packages like get-pip.

This poses a significant maintenance challenge for larger software teams, especially since it’s all reactive. A seemingly minor code change triggers a rebuild of the container on the build server, only to discover that the build is now broken due to an unrelated error. Consequently, immediate action is required to update the application to accommodate the new dependency and rectify the broken build.

StableBuild to the rescue

StableBuild addresses this issue by providing a collection of mirrors and package registries dedicated to ensuring the reliability and predictability of container builds. Essentially, StableBuild enables developers to freeze any dependency, guaranteeing that the same Dockerfile will consistently produce the same Docker container. Currently, this functionality extends to:

• Docker base images, facilitated by managing an immutable Docker Hub mirror.
• Packages sourced from the Ubuntu, Debian, and Alpine package registries (utilized via apt or apk installation), maintained through daily mirroring of the entire package registry.
• Popular third-party package registries (PPAs) such as deadsnakes (for Python) and Nvidia’s CUDA registry, supported through comprehensive daily mirroring.
• Python packages sourced from PyPI, with the complete PyPI index cached daily and on-demand caching of any installed package.
• Any arbitrary URL or file from the internet, facilitated through our file mirror.

Integrating StableBuild into your existing build toolchain takes just 5 minutes. For instance, consider the following Dockerfile, which is now fully cached and built using StableBuild:

# Pull ubuntu:20.04 from StableBuild's immutable Docker Hub mirror
FROM your-prefix.dockermirror.stablebuild.com/ubuntu:20.04

# Pin any `apt` packages to this specific date. As we create a full
# daily copy of the registry, this will always return the exact same
# packages.
ARG APT_PIN_DATE=2024-02-14T10:40:01Z
ARG SB_API_KEY=your-prefix

# Load the Ubuntu package registry and the deadsnakes PPA
COPY ./sb-apt.sh /opt/sb-apt.sh
RUN bash /opt/sb-apt.sh load-apt-sources ubuntu deadsnakes

# Install some OS level packages (same as always)
RUN apt update && apt install -y curl software-properties-common

# We need a newer Python version, grab it from Deadsnakes PPA 
# (no need to manually add the repo anymore) - same as always
RUN apt update && apt install -y python3.9 python3.9-distutils

# Prefix any URL with StableBuild's file mirror URL, and it's automatically
# cached. Then use StableBuild's PyPI mirror for extra dependencies
RUN curl https://your-prefix.httpcache.stablebuild.com/my-first-tutorial/https://bootstrap.pypa.io/get-pip.py -o get-pip.py && \
    python3.9 get-pip.py -i https://your-prefix.pypimirror.stablebuild.com/2024-02-14/

# Install a Python package through pip, pinned to 2024-02-14,
# this will install the package list as it was on that specific date.
RUN pip3 install \
    -i https://your-prefix.pypimirror.stablebuild.com/2024-02-14/ \
    install onnx==1.14.0

Done. This container is now completely pinned, ensuring consistent installation of the same OS version, package list, and Python dependencies.

‍Link to the Article

https://www.stablebuild.com/blog/launching-stablebuild-freeze-and-pin-all-your-dependencies