Fedora change aims for 99% package reproducibility

That's mentioned at the bottom of the README.

Rather, it's the version without a dependency on PERL.

I'm not confident that I understand what you're asking for, but couldn't you just sed off the timestamp from every line? Or for a more extreme example, I have occasionally used ... tr, I think? to completely remove all numbers from logs in order to aggregate error messages without worrying about the fact that they kept including irrelevant changing numbers (something like tail -5000 logfile | tr -d [0-9] | sort | uniq -c | sort -n or so).

A different but more powerful method of ensuring reproducibility is more rigorous compilation using formally verifiable proofs.

That’s what https://pi2.network/ does. It uses K-Framework, which is imo very underrated/deserves more attention as a long term way of solving this kind of problem.

You could just trap time systems calls to always return zero via LD_PRELOAD

I'd far rather a static binary than a bundled vm for a single app which produces all the same bad points of a static binary plus 900 new bad points on top.

Packaging guidelines from a distros docs like this are not any kind of counter argument to that comment.

This is the current orthodoxy, so obviously all docs say it. We all know the standard argument for the current standard. Their comment was explicitly "I'd like to see a change from the current orthodoxy". They are saying that maybe that argument is not all it promised to be back in the 90's when we started using dynamic libs.

They use Windows. \s

Even though the idea is much older, shared libraries were only introduced on Unix systems on SunOS 4.x and System V release 3 (?). Sun paper from 1988: https://www.cs.cornell.edu/courses/cs414/2001FA/sharedlib.pd...

They'd still be duplicated in RAM in the page cache… which is the real problem with static linking.

I don't know this area, but it seems to me it might be a boon to security? So that you can tell if components have been tampered with?

You know what does not give me confidence? Updating software, but whats that, its still printing the same build date? Of course hours later tens of files deep I found out some reproducability goof just hardcoded it.

So far, reproducible builds are heavy on the former, zero on these bugs you mention and zero on supply chain attacks.

Some of these supposed use cases make no sense. You update the compiler. Oh no, all the code is different? Enjoy the 16h deep dive to realize someone tweaked code generation based on the cycle times given on page 7893 of the Intel x64 architecture reference manual.

I don't think it is that unlikely that build hosts or some related part of the infrastructure gets compromised.

The supply chain attacks you have to most worry about are not someone breaking into Fedora build machines.

It's the attacks on the upstream packages themselves.

Reproducible builds would absolutely not catch a situation like the XZ package being compromised a year ago, due to the project merging a contribution from a malicious actor.

A downstream package system or OS distro will just take that malicious update and spin it into a beautifully reproducing build.

Bingo.

True, but every step we add makes the others harder too. It is unlikely Ken Thompson's "trusting trust" compiler would detect modern gcc, much less successfully introduce the backdoor. Even if you start with a compromised gcc of that type there is a good chance that after a few years it would be caught when the latest gcc fails to build anymore for someone with the compromised compiler. (now add clang and people using that...)

We may never reach perfection, but the more steps we make in that direction the more likely it is we reach a point where we are impossible to compromise in the real world.

In this attack, the compiler is not a reproducible artifact? Or does backdooring use another technique?

I was thinking more from a reference point along the lines of LLVM type performance optimizations when I was speaking about optimizations, if that sufficiently clarifies.

The Performant claim there is counter to research I have heard. Plus as the PGO profile data is non-deterministic in most cases, even when compiled on the same hardware as the end machine "Committing profiles directly in the source repository" is the reason why they are deleted or at least excluded from the comparison.

A quote from the paper that I remember on the subject[1] as these profiles are just about as machine dependent as you can get.

> Unfortunately, most code improvements are not machine independent, and the few that truly are machine independent interact with those that are machine dependent causing phase-ordering problems. Hence, effectively there are no machine-independent code improvements.

There were some differences between various Xeon chip's implementations of the same or neighboring generations that I personally ran into when we tried to copy profiles to avoid the cost of the profile runs that may make me a bit more sensitive to this, but I personally saw huge drops in performance well into the double digits that threw off our regression testing.

IMHO this is exactly why your link suggested the following:

> Your production environment is the best source of representative profiles for your application, as described in Collecting profiles.

That is very different from Fedora using some random or generic profile for x86_64, which may or may not match the end users specific profile.

[1] https://dl.acm.org/doi/10.5555/184716.184723

Yo, the attacker has access to the same binaries, so only runtime address randomization is useful.

There's an interesting thought - in addition to aiming for 99% of all packages, perhaps it would be a good idea to target 100% of the packages that, say, land in the official install media? (I wouldn't even be surprised if they already meet that goal TBH, but making it explicit and documenting it has value)

I think that's a consequence of programmers making tools for programmers. It's something I've come to really dislike. Programmers are used to doing things like editing configurations, setting environment variables, or using custom code to solve a problem. As a result you get programs that can be configured, customized through code (scripting or extensions), and little tools to do whatever. This is not IMHO how good software should be designed to be used. On the positive side, we have some really good tooling - revision control in the software world is way beyond the equivalent in any other field. But then git could be used in other fields if not for it being a programmers tool designed by programmers... A lot of developers even have trouble doing things with git that are outside their daily use cases.

Dependency management tools are tools that come about because it's easier and more natural for a programmer to write some code than solve a bigger problem. Easier to write a tool than write your own version of something or clean up a complex set of dependencies.

"Security" and "Convenience" is always a tradeoff, you can never have both.

I use nix package manager on 3 of the systems I'm working daily (one of them HPC cluster) and none of them run NixOS. It's possible to carefully sculpt one's tools and use latest and greatest.

That sounds like an incredibly annoying way to manage software. I don't have a single thing installed that way.

Rightly so. The idea that all software should be packaged for all distros, and that you shouldn't want to use the latest version of software is clearly ludicrous. It only seems vaguely reasonable because it's what's always been done.

If Linux had evolved a more sensible system and someone came along and suggested "no actually I think each distro should have its own package format and they should all be responsible for packaging all software in the world, and they should use old versions too for stability" they would rightly be laughed out of the room.

They do not. Even the vast majority of Arch users thinks the policy of "integrate breakage anyway and post the warning on the web changelog" is pants-on-head insane, especially compared to something like SUSE Tumbleweed (also a rolling distro) where things get tested and will stay staged if broken.

Distros get real hate for being so out of date that upstream gets a stream of bug reports on old and solved issues.

Prime example of this is what the Bottles dev team as done.

It isnt an easy problem to solve.

That's, in my experience, mostly Debian Stable.

System rustc is also good on Arch Linux. I think system rustc-web is also fine on Debian.

Sure, there's never malware on the stores...

> Shipping what randos push works great for iOS and Android too.

App store software is excruciatingly vetted, though. Apple and Google spend far, far, FAR more on validating the software they ship to customers than Fedora or Canonical, and it's not remotely close.

It only looks like "randos" because the armies of auditors and datacenters of validation software are hidden behind the paywall.

At work we went back to a Docker build to make reproducible images. The primary reason is poor cross-compilation support in Nix on Arm when developers needed to compile for an amd64 service and derive image checksums that are put into tooling that are run locally for service version verification and reproducibility.

With Docker it turned out relatively straightforward. With Nix even when it runs in Linux Arm VM we tried but just gave up.

I agree that NixOS/nixpkgs would not be a good a basis for comparison. Do you have an opinion about the use of nix by app devs to specify their builds, i.e. as a make alternative, not as a Fedora alternative?

Quoting the article:

> Irreproducible bits in packages are quite often "caused by an error or sloppiness in the code". For example, dependence on hardware architecture in architecture-independent (noarch) packages is "almost always unwanted and/or a bug", and reproducibility tests can uncover those bugs.

This is the sort of thing that nix is good at guarding against, and it's convenient that it doesn't require users to engage with the underlying toolchain if they're unfamiliar with it.

For instance I can use the command below to build helix at a certain commit without even knowing that it's a rust package. Although it doesn't guarantee all aspects of repeatability, it will fail if the build depends on any bits for which a hash is not known ahead of time, which gets you half way there I think.

     nix build github:helix-editor/helix/340934db92aea902a61b9f79b9e6f4bd15111044

That's such a weird characterization of this article, which (in contrast to other writing on this subject) clearly concludes (a) that Nix achieves a very high degree of reproducibility and is continuously improving in this respect, and (b) Nix is moreover reproducible in a way that most other distros (even distros that do well in some measures of bitwise reproducibility) are not (namely, time traveling— being able to reproduce builds in different environments, even months or years later, because the build environment itself is more reproducible).

The article you linked is very clear that both qualitatively and quantitatively, NixOS has made achieved high degrees of reproducibility, and even explicitly rejects the possibility of assessing absolute reproducibility.

NixOS may not be the absolute leader here (that's probably stagex, or GuixSD if you limit yourself to more practical distros with large package collections), but it is indeed very good.

Did you mean to link to a different article?

Sure, but in terms of absolute number of packages that are truly reproducible, they outnumber Debian because Debian only targets reproducibility for a smaller fraction of total packages & even there they're not 100%. I haven't been able to find reliable numbers for Fedora on how many packages they have & in particular how many this 99% is targeting.

By any conceivable metric Nix really is ahead of the pack.

Disclaimer: I have no affiliation with Nix, Fedora, Debian etc. I just recognize that Nix has done a lot of hard work in this space & Fedora + Debian jumping onto this is in no small part thanks to the path shown by Nix.

Yes, I know that. But when talking about reproducible packages, we can and should learn from existing techniques. Is Fedoras system capable of this goal? Is it built the right way? Should it adopt an alternate package manager to achieve this with less headache?

The same LWN article says:

> The contents, however, should still be "bit-by-bit" identical, even though that phrase does not turn up in Fedora's definition.

So, according to the literal interpretation of the article, signatures inside the payload (e.g., files that are signed using an ephemeral key during the build, NOT the overall RPM signature) are still a self-contradictory area and IMHO constitute a possibly-valid reason for not reaching 100% payload reproducibility.

I learned this from an ansible molecule test env setup script for use in containers and VMs years ago; because `which` isn't necessarily installed in containers for example:

  type -p apt && (set -x; apt install -y debsums; debsums | grep -v 'OK$') || \
  type -p rpm && rpm -Va  # --verify --all

  [updates]
  name=Fedora $releasever - $basearch - Updates
  #baseurl=http://download.example/pub/fedora/linux/updates/$releasever/Everything/$basearch/
  metalink=https://mirrors.fedoraproject.org/metalink?repo=updates-released-f$releasever&arch=$basearch
  enabled=1
  countme=1
  repo_gpgcheck=0
  type=rpm
  gpgcheck=1
  metadata_expire=6h
  gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$releasever-$basearch
  skip_if_unavailable=False

  gpgkey
  gpgkey_dns_verification
  
  repo_gpgcheck
  localpkg_gpgcheck
  gpgcheck

2. https://docs.ansible.com/ansible/latest/collections/ansible/... lists a gpgcakey parameter for the ansible.builtin.yum_repository module

For Debian, Ubuntu, Raspberry Pi OS and other dpkg .deb and apt distros:

  man sources.list
  man sources.list | grep -i keyring -C 10
  # trusted:
  # signed-by:
  # /etc/apt/ trusted.gpg.d/
  man apt-secure
  man apt-key
  apt-key help
  less "$(type -p apt-key)"

From "New requirements for APT repository signing in 24.04" (2024) https://discourse.ubuntu.com/t/new-requirements-for-apt-repo... :

> In Ubuntu 24.04, APT will require repositories to be signed using one of the following public key algorithms: [ RSA with at least 2048-bit keys, Ed25519, Ed448 ]

> This has been made possible thanks to recent work in GnuPG 2.4 82 by Werner Koch to allow us to specify a “public key algorithm assertion” in APT when calling the gpgv tool for verifying repositories.

>Defaulting to Android-style nanny sandboxing ("you can't grant access to your Downloads folder because we say so" etc.) is unlikely to go over well with the average Linux distro userbase.

If you market it that way. Plenty of Linux users say they care about security, don't want maleware, etc. This is a step towards those desires. Users have been conditioned to use tools badly to designed for security for decades so there will be some growing pains, but it will get worse the longer people wait.

>Just use Podman and only mount your Wayland socket + any working files.

This won't work for the average user. Security needs to be accessible.

> Just use Podman and only mount your Wayland socket + any working files.

If only it was that simple…

Both are good for security, but prioritization is important. The areas that are weakest in terms of security should get the most attention.

This only secures between vms. This side steps the problem and people can still easily run multiple applications in the same qube.

Sure, that is one solution, but it still needs a lot of work both to patch up holes in it and to fix apps to be better designed in regards to security.