#3205 Request exception to permit shipping pre-built, signed SGX enclave binaries in Fedora
Closed: Insufficient data by sgallagh. Opened by berrange.

I am working on an attempt to bring support to Fedora for running confidential virtual machines (hereafter called 'CVM') using Intel's TDX technology. At a high level a CVM is just like any traditional VM, but the guest RAM is encrypted, such that no host/hypervisor software can read guest memory, only the physical hardware and guest software can see guest memory in plaintext.

Integrating software to enable the launch of confidential virtual machines is straightforward, however, one of the use cases for CVMs requires a remote user / service to be able to "prove" that a CVM is running in the confidential hardware environment it claims to be - this is called "remote attestation". In establishing this proof of/trust in the hardware (and its vendor), the user eliminates (much of) the need to trust either the owner of the host, or the software vendor of the hypervisor/host OS.

For current generations of TDX, the attestation processes builds on top of the Intel SGX technology.
At a high level SGX provides a way for a userspace process to carve off a portion of RAM to be used to execute code in an "enclave". Trusted code is loaded into this enclave and runs some computation tasks, guarantee safe from interference from untrusted code in the regular userspace process or OS kernel. The enclave code exchanges data with its owning process. Code running inside the enclave does NOT link to any of the OS libraries. There are completely separate embedded C & C++ runtime libraries (and some supporting libs for crypto & data exchange) for SGX enclaves to link against, which cannot be substituted out with glibc or any other standard distro library. We're still compiling for x86_64 using regular GCC/binutils x86 targets, but a dedicated runtime (ie gcc -nostdlib -standalone flags are used) as you would have in firmware or kernel or other embedded scenarios. The code to be run inside an SGX enclave will have a signature attached from the vendor of the code, which is validated when the enclave is launched.

It is possible to verify that an application provided enclave is running code signed by its particular vendor's certificate. This involves using some standard / fundamental SGX enclaves built and signed by Intel with a well known certificate, tieing the root of trust to the Intel hardware platform.

To support remote attestation of CVMs using Intel TDX, thus requires packaging much of the SGX software stack in Fedora. Packaging SGX software in Fedora requires packaging the small set of standard enclaves built and crucially signed by Intel - id_enclave.so le.so le.so.orig pce.so pve.so qe.so qe3.so qve.so tdqe.so.

While the enclaves are built into ELF so files, this is just a convenient format to package the code and signature together. These .so files are not loadable by Linux userspace ld-linux.so. The SGX host software will extract the executable code from the ELF file and place it into the enclave memory and verify the signature. The prebuilt binary enclave .so files are available at

https://download.01.org/intel-sgx/sgx-linux/2.24/prebuilt_ae_2.24.tar.gz
https://download.01.org/intel-sgx/sgx-dcap/1.21/linux/prebuilt_dcap_1.21.tar.gz

which were respectively created from enclave sources in

https://github.com/intel/linux-sgx/
https://github.com/intel/SGXDataCenterAttestationPrimitives

You can see from git that the code for these enclaves is all open source, and comprises a mixture of licenses, all of which are approved for inclusion in Fedora. It is possible to build the enclaves on Fedora, however, it would not be possible to use the result since the binaries would lack the required Intel cryptographic signatures the establish the root of trust back to the hardware.

This creates a packaging guideline compliance problem for us, given the normal requirement to build from source and not ship pre-built binaries in Fedora.

The binaries in the tarballs above were built from corresponding git tags in the repos above using a toolchain and some supporting packages from the NixOS distro that enables byte-perfect reproducibility:

https://github.com/intel/linux-sgx/blob/main/linux/reproducibility/Dockerfile#L24

NB, that dockerfile is derived from Ubuntu, but that's just for bootstrapping the NixOS installation, which is what the enclaves actually build with.

I've considered / evaluated a number of options for SGX enclave packaging in Fedora

  • Declare that the standard Intel signed SGX enclaves get a general exception from the requirement to build from source. The maintainer would promise to run the reproducible build process under NixOS before importing any new releases, to prove they come from the source they claim to come from. This verification would not involve any Fedora toolchain/software though, and would be a manual process outside the context of Koji. The licenses don't require providing the source (no copyleft code goes into the enclaves), but we could still provide the source tarballs if desired, even if not building from them. This is similar conceptually to how we grant pre-built firmware blobs a general exception, with the added verification of the build process, since we are fortunate to actually have the source available.

  • Build the standard SGX enclaves with normal Fedora toolchain and package dependencies. Send the resulting enclaves to Intel to be verified and signed by them. Receive back signed enclaves which we package. This is similar conceptually to how we roundtrip 'shim' through Microsoft for signing. The difference is that with Shim there's value in that each distro vendor is building shim with their own set of certificates embedded. With SGX enclaves each distro would be building the exact same source with no local modifications - only the gcc/binutils would really differ. I've several spent months in discussion with Intel on the possibility of them signing our builds, but this has ultimately been rejected as something they're unwilling to do. It is not scalable for them to audit and verify the full toolchain and build process used across countless Linux distros, and across multiple releases of each distro, in order to sign their respective enclave builds. From an end user POV it would have no functional benefit either, since distros would not be permitted to make any source changes. Effectively the only differences would be what gcc/binutils versions are used. At that point there's no upside to signing something again from their POV, only risk of undermining the root of trust by mistakenly signing a bad build from a distro, a scenario that is difficult to recover from.

  • Add custom RPM builds of binutils, gcc (& a couple of other pkgs), using the exact same versions as present in Intel's NixOS environment. Build the standard SGX enclaves with this alternative toolchain in Fedora. Bundle Intel's existing pre-built & signed enclave binaries in the RPM as additional "sources". Verify that the freshly built enclaves are byte-for-byte identical with Intel's pre-built enclaves from NixOS as part of the RPM build %check step. This is effectively (1), but with all toolchain needed for verification packaged up in Fedora so we're not relying on NixOS / Ubuntu containers, or manual maintainer verification. The verification of signed pre-built enclaves would be done automatically inside Koji as part of the RPM build process. This gives us a much stronger guarantee that we're shipping what we claim to, along with its corresponding source (even though the licenses don't require that in this case).

  • Don't ship any SGX enclaves at all in Fedora. Request Intel to host their own YUM repo metadata with packages for the enclaves that are installed in a location that matches Fedora's needs. Fedora packages the rest of the non-enclave SGX code. Intel has historically provided RPMs for old RHEL with all SGX content, however, the contents are NOT following Fedora RPM packaging guidelines in any way, so are not desirable to consume. We would have to ask Intel to change the packaging of SGX enclaves to a manner we are happy telling users to consume.

  • Don't ship any SGX enclaves at all in Fedora. Require anyone who wants to run TDX CVMs to acquire the pre-built enclaves from Intel's GitHub download site. Possibly add a script to Fedora to download this automatically on boot or first use. I include this for completeness, as it is the fallback if all the other options above are rejected. I think it would be an unfriendly solution for Fedora's users though, so I'd like to avoid it.

With all this in mind, I am looking for the least bad option that is acceptable to Fedora, and thus asking FPC to consider granting packaging exception to allow option (3):

  • Pre-built binaries for the standard / fundamental SGX enclaves, signed and distributed by Intel, can be packaged Fedora, with the pre-condition that their payload is verified to be byte-for-byte identical to unsigned binaries fully packaged and built from source in koji using the designated SGX toolchain and runtime for reproducible builds.

The main downside to this exception is that while the source is open, there's no practical benefit if you build your own versions / make changes in context of Fedora, as while you can create builds, you can't sign them and thus can't use them. The only way to get changes is to submit them back upstream to Intel and hope they'll be accepted for the next upstream release. This is a difference from shim, where you can build your own shim and sign it yourself, provided your firmware lets you change trusted secureboot keys. SGX/TDX has no concept of custom root keys, since the goal is precisely to let users establish trust of a hardware platform they do not own - trusting hardware owner provisioned keys is not desirable.

In case it wasn't clear, all of the above applies only to the SGX enclave code. The SGX software stack includes a lot of other code that is just built and run in normal OS distro context. This host OS software part satisfies Fedora packaging guidelines without issue.

Also there other SGX enclaves app developers have created for various tasks. I am NOT proposing for these to be covered by the exception. Only the fundamental SGX enclaves created & signed by Intel that bootstrap the platform should be covered by any exception.

BTW, you may have read that SGX is deprecated/removed from Intel CPUs. This only applies to desktop class 'Core' CPUs. Server class 'Xeon' CPUs retain SGX, and TDX is present only in the latter as a result. So this is an relevant technology for the forseeable future (however long current generation Xeon's remain in active use by Fedora userbase).

Sorry for the long read, I wanted to give as much background detail as practical, to make an informed decision, since I expect most people are unlikely to be very familiar with Intel's SGX or TDX technologies.

NB, i previously requested this at https://pagure.io/packaging-committee/issue/1364 but FPC decided they don't have authority to grant the request, and I need to speak to FESCO.


In general, my sentiment is that we want to make it easy for users to use this technology, i.e. run CVMs on Fedora and/or Fedora in CVMs, and that we should provide the necessary "blobs" to make this work relatively painlessly for users who choose to use this. Thus, I think we should package the signed blobs. As you say, otherwise users would have to download the blobs from somewhere else, making the whole process painful.

Add custom RPM builds of binutils, gcc (& a couple of other pkgs), using the exact same versions as present in Intel's NixOS environment. Build the standard SGX enclaves with this alternative toolchain in Fedora. Bundle Intel's existing pre-built & signed enclave binaries in the RPM as additional "sources". Verify that the freshly built enclaves are byte-for-byte identical with Intel's pre-built enclaves from NixOS as part of the RPM build %check step.

Using a reproducible build for this is the most reasonable approach.

The proposal says "payload is verified to be byte-for-byte identical", but doesn't say where the verification happens. Would this be manual by the packager, or part of the automatic koji package build? The latter would be much more attractive.

A custom build of gcc, binutils, and other packages is a lot of work. But I don't think we'd want to distribute binaries we cannot build, so it seems an acceptable price to pay in this case.

FWIW, I'm working also working on increasing build reproducibility in Fedora packages, and based on what I've seen so far, I have some doubts whether packaging the same versions of compiler and other tools will be enough to achieve reproducibility. Build artifacts tend to be very sensitive to paths and various bits of the environment, that'd necessarilly differ between a build in koji and the nixos-on-ubuntu-in-docker thingy. Did you actually get it working already?

The proposal says "payload is verified to be byte-for-byte identical", but doesn't say where the verification happens. Would this be manual by the packager, or part of the automatic koji package build? The latter would be much more attractive.

My desire would be to have verification performed in a "%check" phase of the RPM spec that packages the pre-signed enclave binaries, such that we can't build the signed enclave package unless it is proved to verify against the sources we expect.

A custom build of gcc, binutils, and other packages is a lot of work. But I don't think we'd want to distribute binaries we cannot build, so it seems an acceptable price to pay in this case.

FWIW, I'm working also working on increasing build reproducibility in Fedora packages, and based on what I've seen so far, I have some doubts whether packaging the same versions of compiler and other tools will be enough to achieve reproducibility. Build artifacts tend to be very sensitive to paths and various bits of the environment, that'd necessarilly differ between a build in koji and the nixos-on-ubuntu-in-docker thingy. Did you actually get it working already?

I have a work in progress on this front. With the required gcc and binutils versions, I can reproduce about 95% of the '.o' files that go into the binary enclave binaries. I'm fighting a remaining problem where GCC uses a different ELF section name, for static variables, variable based on a whole heap of context in the header files. This may require me also building a specific glibc + kernel-headers versions, which is annoying as the build process doesn't need to actually link to glibc - it just needs a couple of data type definitions. If i'm lucky though I might be able to tweak the build system to avoid this.

The reproducibility is not as bad as the Fedora common case, because most of the code builds with -nostdlib -ffreestanding. The only wierd path is that the SGX enclave ELF contains contain a (unused & redundant) path to the NixOS ld.so binary. I'm going to report that one to Intel, since it is a harmless mistake that can likely be eliminated in a future release.

TL;DR: reproducibility has some definite pain points, but it should be viable from what I've seen so far.

BTW, you may have read that SGX is deprecated/removed from Intel CPUs. This only applies to desktop class 'Core' CPUs. Server class 'Xeon' CPUs retain SGX, and TDX is present only in the latter as a result. So this is an relevant technology for the forseeable future (however long current generation Xeon's remain in active use by Fedora userbase).

For someone like me that is not that familiar with Intel's CPU products. Can provide the production date ranges for CPUs that support this feature?

For someone like me that is not that familiar with Intel's CPU products. Can provide the production date ranges for CPUs that support this feature?

Intel SGX was introduced in Intel Core in 2015 based on Skylake microarchitecture. It was discontinued in 11th/12th generation Core around 2021/2022, but kept in Xeon.

Intel TDX, which is the bit I care about, building on top of SGX, is in Xeon based on Emerald Rapids miroarch, end of 2023. There is some access in the previous Sapphire Rapids, but IIUC that's limited to a few big cloud vendors, not the general public. So Emerald Rapids is what matters from POV of Fedora TDX support, but if we introduce the SGX enclaves in Fedora they'll be usable outside context of TDX too, if users so desire.

verification performed in a "%check" phase of the RPM spec that packages the pre-signed enclave binaries, such that we can't build the signed enclave package unless it is proved to verify against the sources we expect.

That's nice.

As I said in the FPC ticket, I don't particularly want to grant an exception to allow pre-built SGX enclaves. I would rather us package up the toolchain and use it to build it ourselves, because then I feel we can trust it.

I think that if we are building binaries in koji and putting those into RPMs (and not binaries included in the 'sources'), then the proposed solution is fine. I acknowledge that we would be limited by the signing requirements an unable to fix bugs or CVEs unless Intel did first. We also carry additional risk, since to fix a CVE we need Intel to fix it first and we need to produce byte-for-byte identical builds for the fix, which may not be trivial.

As for the custom toolchain, I think that it should be built as part of the SGX package, but not distributed into any RPMs (even if they are buildroot-only). I don't think it's good practice to start building RPMs with custom toolchains in Fedora or else everyone is going to want one.

This "feature" is a perfect example of Treacherous Computing. Those binaries are also effectively not Free Software, even if they are under a Free Software license, because the hardware does not allow you to run modified versions. So this kind of anti-feature goes against the Fedora core values (Freedom) and has no business being in Fedora at all.

What is role of the TDX code? Where it is executed, and where it has access to? If I understand https://en.wikipedia.org/wiki/Trust_Domain_Extensions correctly, TDX code executed us in TDX module. TDX module is used as a gateway between CVM (encrypted GSX enclave with a user-supplied operating system) and a hypervizor. Is the TDX module is dedicated processor, is is just another GSX enclave running on the same CPU? Does the remote attestation only verify integrity of the TDX module, or does it aspire for verifying integrity of CVM? In the later case TDX module has to have unlimited an access to CVM. That would be very different from security point of view from other firmwares we ship. Other firmwares ares usually guarded by MMIO unit.

@ppisar you're mixing two distinct bits of code here. The "TDX Module" is what actually implements the VM security isolation, and is a part of the host firmware, that runs beneath the OS. You can think of it as somewhat similar to SMM code in terms of where it runs and its privileges.
The "TDX Module" is NOT what this ticket is about.

The SGX enclaves, provide a way to run unprivileged in a secure manner, such that it is protected from the regular untrusted application code, while still being effectively ring-3. It is running on the same CPU as all the other application code, but the only way for the SGX enclave and application code to communicate is through specific RPC based APIs. In the case of TDX, the SGX enclaves are used by a daemon called "QGS" which provides a signed attestation report to a guest, proving that the guest is running on a host and protected by the Intel TDX platform.

IOW, without the signed SGX enclaves, you can still launch TDX VMs, but the VM is incapable of proving to its (remote) owner, that it is truly running on a Intel TDX platform as claimed.

To elaborate on my previous comment:

IMHO, the only approach compatible with the Fedora project's Freedom goal is: Don't ship any SGX enclaves at all in Fedora. Require anyone who wants to run TDX CVMs to acquire the pre-built enclaves from Intel's GitHub download site or some Intel or third-party repository (but do not link to those from any Fedora documentation). Do not add any scripts to Fedora to download this automatically on boot or first use.

This is effectively proprietary software, because, as you wrote, "The main downside to this exception is that while the source is open, there's no practical benefit if you build your own versions / make changes in context of Fedora, as while you can create builds, you can't sign them and thus can't use them." In other words, the hardware prevents you from making any use of the freedoms theoretically granted by the software license, making the Free Software license effectively useless. Fedora does not distribute or promote proprietary software. This should be no exception.

Fedora does not distribute or promote proprietary software. This should be no exception.

It is possible to consider that the SGX enclaves are providing firmware like features. Firmware is running critical pieces of trusted code that expose certain services to the OS. The difference that instead of using an elevated CPU privilege level to protect the "firmware" from OS code, the SGX technology protects the "firmware" from OS code while letting it run at a low privilege level, which is better as there's less scope for bad SGX enclaves to take out the whole machine.

Fedora already permits firmware to be distributed in binary form with absolutely zero source code, so by comparison the SGX enclave code is significantly better

The way this software is designed to operate is clearly inconsistent with the definition of "firmware". While the boundaries of "firmware" are somewhat murky (no) thanks to CPU microcode, UEFI, and the like, this software:

  • runs on the main CPU,
  • runs at a time where the main (host) OS is already running,
  • is not necessary to get the (host) OS to boot,
  • does not enable any peripheral hardware device to work (but a CPU "feature"),
  • talks to the Internet (necessarily has to, for remote attestation),

all of which speaks against this software being "firmware".

Fimware is normally either something that allows the main CPU to boot an OS (and I consider microcode, which does the instruction decoding, to be part of that, even if the CPUs carry an old buggy version of the microcode on chip, making the microcode update technically unnecessary to boot) or something that enables a peripheral hardware to work and runs on that peripheral hardware's processor, not on the main CPU. This software is neither. It is instead the core part of a "Trusted Computing", i.e., Treacherous Computing, implementation, something Fedora should be fighting against, not for.

Firmware also has no business talking to the Internet unless of course it is the firmware for a network card/stick (Ethernet, WiFi, modem, or the like).

talks to the Internet (necessarily has to, for remote attestation),

This is wrong. The SGX enclave code has no network connectivity at all and just does very small tasks, mostly just related to cryptographic signature creation / verification, and other targeted data verification tasks.

The guest OS requests a "TDX report", and passes this to QEMU in the host, QEMU sends it to the 'qgs' daemon, and that 'qgs' calls the 'tdqe' SGX enclave. The enclave verifies that the "TDX report" is valid and if so adds a cryptographic signature, and sends the signed report back to qgs, which sends it back to QEMU, which sends it back to the guest OS. The user/application in the guest OS can now pass this signed attestation report to an external party on a remote host, which is now able to verify the attestation report thus providing that the VM was running with TDX protection.

In the end, it is the enclave which signs the thing and the remote server which verifies the signature, so the enclave is communicating with the remote server, even if the Internet connectivity is handled by a middleman (QEMU).

In the end, it is the enclave which signs the thing and the remote server which verifies the signature, so the enclave is communicating with the remote server, even if the Internet connectivity is handled by a middleman (QEMU).

That's a really bizarre way of looking at it - it is like saying that /usr/bin/openssl is communicating with a remote server if it happens to be invoked by an apache CGI script to sign some data that the CGI script returns to the client.

This particular Treacherous Computing implementation is marketed as a "feature" for datacenter operators to provide "trusted" services to their paying customers, but that is just the trick to make it palatable to entities like Fedora, and it looks like Fedora is falling for this scheme.

The ultimate goal of Treacherous Computing is pretty clear: Allow the hardware manufacturer (in this case, Intel) to completely control what software you run, effectively eliminating all Free Software or turning it effectively proprietary. Remote attestation is envisioned as the way to exclude Free Software from large parts of the Internet. This has already started to happen on Android, e.g., more and more banking apps refuse to run on anything other than an unmodified unrooted Android on an officially supported device, using Google's remote attestation "security" "feature" to enforce that. Is this something Fedora wants to bring to the PC world?

I would urge you to read Can You Trust Your Computer?, The Right to Read, etc.

Allow the hardware manufacturer (in this case, Intel) to completely control what software you run, effectively eliminating all Free Software or turning it effectively proprietary. Remote attestation is envisioned as the way to exclude Free Software from large parts of the Internet. This has already started to happen on Android, e.g., more and more banking apps refuse to run on anything other than an unmodified unrooted Android on an officially supported device, using Google's remote attestation "security" "feature" to enforce that. Is this something Fedora wants to bring to the PC world?

Remote attestation in general is nothing new. The technology for it already exists in Fedora for many, many years, via TPM support integrated across various software packages in Fedora. The TDX remote attestation technology does not not restrict what is run in your VM - it is entirely at the discretion of the owner of the VM what is run there. It is providing a mechanism for the owner of the VM to prove that their requested software is actually what's running, and has not been compromised by a broken/compromised hypervisor host software stack/admin.

That is the disguise under which they are pushing this stack into distributions. Once everyone ships it, they will use it to restrict our freedom.

I will make this very simple: my position on this is that if we can't even do the absolute basic thing of building and shipping our own software for this (we are able to do that for shim and that thing is a nightmare), then this is not worth having in Fedora or any Linux distribution.

It is effectively a misfeature and a pathway to forcing us to ship unusable FOSS. It is important that we have software we can use, remix, and redistribute. It is not firmware. Intel has stated it is technically possible to support third party builds, they just don't want to.

Then we can equally say we won't support the feature. It's not useful if it's not trustable. End of statement.

As an aside, I'm very disappointed in how people are using "Confidential Computing" as a way to do exactly the opposite. Nobody can practically trust the computing environment with something like these non-trustable enclaves software solutions.

This is not like firmware where we effectively lose the ability for a computer to function properly. This is for a thing that has nebulous benefits that would in practice be voided by binary blobs that everyone has the sources for but can't practically use.

Sorry, no.

-1

Metadata Update from @ngompa:
- Issue tagged with: meeting

Metadata Update from @ngompa:
- Issue untagged with: meeting

Metadata Update from @ngompa:
- Issue tagged with: meeting

As an aside, I'm very disappointed in how people are using "Confidential Computing" as a way to do exactly the opposite. Nobody can practically trust the computing environment with something like these non-trustable enclaves software solutions.

"Nobody" doesn't seem to be entirely accurate here. From the description in the proposal:

The code to be run inside an SGX enclave will have a signature attached from the vendor of the code, which is validated when the enclave is launched.

It is possible to verify that an application provided enclave is running code signed by its particular vendor's certificate. This involves using some standard / fundamental SGX enclaves built and signed by Intel with a well known certificate, tieing the root of trust to the Intel hardware platform.

The hardware manufacturer can evidently authenticate the code, and presumably the software vendor can check whether it is running in an SGX enclave. SGX does however prevent the hardware owner from interposing between the hardware and the software. But since end users also don't manage the bare metal machines in AWS where SGX is likely to be deployed, how much can they trust their computing environment right now?

It all comes down to: Do you trust Intel more than you trust your VPS/"cloud" vendor?

It all comes down to: Do you trust Intel more than you trust your VPS/"cloud" vendor?

I really don't want to open this can of worms, however...
If you can't trust the hardware vendor, especially the one which provides the CPU, you have much bigger problems on your hands...

That is not the question we're asking. We're asking "can we trust that a software blob that isn't needed that is open source that can't be authenticated or replaced is trustable in Fedora?"

That is not the question we're asking. We're asking "can we trust that a software blob that isn't needed that is open source that can't be authenticated or replaced is trustable in Fedora?"

Could you rephrase that, Neal? It's a run-on sentence and it's not clear to me what you're saying.

that can't be authenticated

Doesn't the rebuild comparison accomplish that? Or what else do you mean?

To break it down, this blob is:

  • open source, but...
  • not replaceable with our own builds
  • cannot be fully verified because we cannot run our own builds
  • runs on the main hardware environment alongside the OS

We have never allowed this in Fedora before. The only reason this is coming to us is because Intel is saying they don't want to sign our own builds.

This creates the problem that we get all the blame but have no power to fix things when it's bad. This also creates a problem that we don't even have with shim where we are backing something we cannot build. We've made a lot of allowances for things that don't run in the main system environment so that hardware works, but this stuff is part of the virtualized system environment. Indeed, it's merely carved out from the system environment the machine operates in.

From my perspective, this is not okay to ship in Fedora at all, and we should be pushing Intel to do better here.

cannot be fully verified because we cannot run our own builds

This proposal is specifically indicating that we WILL run our own builds, AND verify that the output is a 100% byte-for-byte match of the pre-signed binary. The only difference will be the cryptographic signature on the binary that establishes the chain of trust to the hardware platform.

But "our own builds" will only work if they happen to match byte for byte whatever blob Intel ships. Sounds like "you can have any color provided that it is black". This completely defeats the point of doing our builds at all. We are not allowed to make any changes whatsoever to this code (and in fact Intel made it pretty clear that they would not allow any modifications even if they were to sign our own builds, which is one of the reasons they refuse to sign our builds, they just do not see the point given this constraint that they are not willing to budge an inch on), which makes it effectively proprietary software. The whole point of Free Software is the freedom to modify, which is effectively nonexistent here, despite the FOSS license.

So this should be treated like any other non-Free/proprietary software: not allowed in Fedora.

From an end user POV it would have no functional benefit either, since distros would not be permitted to make any source changes.

Then this software is not free and open source. It should be treated as proprietary and rejected from inclusion.

It's not clear to me whether Fedora in this scenario would run as the virtualization host or as the virtual machine. I think I've at least figured out that the client that receives the remote attestation is not the one that would use these pre-built binaries.

Anyway, as far as I understand:

  • Freedom to use the software for any purpose: Yes.
  • Freedom to study the software: Yes.
  • Freedom to share the software with others: Yes.
  • Freedom to improve the software: No, because the processor will reject any modified version that isn't signed by Intel.

Thus the pre-built binaries are unfree.

Berrange wrote:

no copyleft code goes into the enclaves

That's no coincidence. I'm sure they took care to avoid code that they're not allowed to include in unfree software.

Jrelvas wrote:

If you can't trust the hardware vendor, especially the one which provides the CPU, you have much bigger problems on your hands...

I'd love to have an affordable fabber at home that could turn free-as-in-freedom schematics into electronic components. Since that isn't practical, my choices are to trust various hardware vendors or become a hermit. Running my programs on my own computer instead of a cloud vendor's computer is immensely more practical than manufacturing my own electronics, and not wanting to give hardware vendors more control than necessary is a perfectly valid position.

The key question seems to be: Is the push to move all computation from users' own computers to big remote datacenters important enough to justify an exception from the Fedora Project's "freedom" foundation?

This is marked as meeting, but we already discussed this extensively during two meetings, and it seems that we're waiting for a real change proposal to be filed, and it seems we'll not make any progress until that happens. So I'll unset the meeting tag for now.

Metadata Update from @zbyszek:
- Issue untagged with: meeting

It's not clear to me whether Fedora in this scenario would run as the virtualization host or as the virtual machine. I think I've at least figured out that the client that receives the remote attestation is not the one that would use these pre-built binaries.

There are three distinct OS installations involved here, and intent is that Fedora is able to be used in all three roles

  • Virtualization host
  • Confidential virtual machine
  • Attestation service host

The pre-built binaries discussed here are only relevant for the virtualization host, they would not be used in any other deployment.

  • Freedom to use the software for any purpose: Yes.
  • Freedom to study the software: Yes.
  • Freedom to share the software with others: Yes.
  • Freedom to improve the software: No, because the processor will reject any modified version that isn't signed by Intel.

To clarify this point - the processor will happily run a build of the SGX enclaves that is signed by a non-Intel key. The problem is that, AFAICT, the attestation report provided by virtualization host is then not able to be successfully validated by the attestation service host. Unless I can find a solution to this limitation, the use of a non-Intel signing key makes the deployment unverifiable to the end user.

Thus the pre-built binaries are unfree.

No that's not correct. Consider a laptop whose BIOS runs with SecureBoot enforcing, and does NOT include the Microsoft 3rd party signing key. Such a laptop will be unable to run Fedora, because its BIOS will not trust the SecureBoot signature on shim. By your logic here, that means the Fedora binaries are unfree.

The problem is not the software - it is that the hardware design and/or implementation is hostile to free software.

I'd love to have an affordable fabber at home that could turn free-as-in-freedom schematics into electronic components. Since that isn't practical, my choices are to trust various hardware vendors or become a hermit. Running my programs on my own computer instead of a cloud vendor's computer is immensely more practical than manufacturing my own electronics, and not wanting to give hardware vendors more control than necessary is a perfectly valid position.

The key question seems to be: Is the push to move all computation from users' own computers to big remote datacenters important enough to justify an exception from the Fedora Project's "freedom" foundation?

This proposal is absolutely not about pushing people over to remote data centers instead of their own hardware. Precisely the opposite. It will be possible to run Fedora guest VMs on a Intel TDX protected public clouds regardless, as that use case requires no special work.

Rather this proposal allows Fedora users to setup their own private data center/cloud hosts, and establish trust in their own managed hardware, with a greater level of freedom than they can ever get from the public cloud provider.

IOW, by NOT accepting functionality needed for TDX virtualization hosts, we prevent Fedora users from establishing trust in their own hardware, which is shooting ourselves in the foot and will encourage more use of 3rd party hardware fully outside their control with much less open source code in use.

An end to end solution in Fedora for Intel TDX across the virtualization host, confidential VM, and attestation service will be the most open confidential computing platform available anywhere today. Yes, I still hate that this pre-built signed binary exists, yet all the other non-Intel platforms will have the same kind of pre-built signed binary blob - it'll just be invisible to Fedora as it is bundled with the host firmware. Intel is more open than its hardware peers by having this actually done in OSS software.

To clarify this point - the processor will happily run a build of the SGX enclaves that is signed by a non-Intel key.

They must be signed with a Fedora key. Instead of uploading pre-built non-free blobs, the proposal owners should work with virtualization providers to trust such a key.

Consider a laptop whose BIOS runs with SecureBoot enforcing, and does NOT include the Microsoft 3rd party signing key. Such a laptop will be unable to run Fedora, because its BIOS will not trust the SecureBoot signature on shim.

Such laptops should be banned by antimonopoly committees. Users should be able to install any OS they want.

By your logic here, that means the Fedora binaries are unfree.

Definitely not. Only firmware can be considered proprietary and non-free.

The problem is not the software - it is that the hardware design and/or implementation is hostile to free software.

Fedora should not endorse this design, otherwise it will be used more often.

Daniel P. Berrange wrote:

The pre-built binaries discussed here are only relevant for the virtualiz=
ation host, they would not be used in any other deployment.

Thanks for the clarification.

  • Freedom to use the software for any purpose: Yes.
  • Freedom to study the software: Yes.
  • Freedom to share the software with others: Yes.
  • Freedom to improve the software: No, because the processor will r=
    eject any modified version that isn't signed by Intel. =20
    =20
    To clarify this point - the processor will happily run a build of the SGX=
    enclaves that is signed by a non-Intel key. The problem is that, AFAICT, t=
    he attestation report provided by virtualization host is then not able to b=
    e successfully validated by the attestation service host. Unless I can find=
    a solution to this limitation, the use of a non-Intel signing key makes th=
    e deployment unverifiable to the end user.

So you're a cloud provider running Fedora on your virtualization hosts.
You allow your customers to run confidential virtual machines relying on
SGX enclaves. You fix a bug in an enclave and install the fixed version.
It runs, but now your customers' attestations fail. The customers have
no way to distinguish your bugfix from an attack on the confidentiality
of their virtual machines. You lose your customers.

Although the modified enclave technically runs, it doesn't fill its
function. In practice you have no choice but to use Intel's signed
binaries.

Consider a laptop whose BIOS runs with SecureBoot enforcing, and does NOT=
include the Microsoft 3rd party signing key. Such a laptop will be unable =
to run Fedora, because its BIOS will not trust the SecureBoot signature on =
shim. By your logic here, that means the Fedora binaries are unfree.

What I've heard is that it's supposed to be possible to install one's
own key in a TPM, sign one's own modified bootloader or shim with that
key, and have the bios enforce that only code signed with that key gets
to run. If what I've heard is true, then users have the freedom to
improve Shim and use their improved version (though in practice it may
be more difficult than we would like). You say it's not possible to do
that with SGX enclaves, so the cases are not equivalent.

Rather this proposal allows Fedora users to setup their own private data =
center/cloud hosts, and establish trust in their own managed hardware, wi=
th a greater level of freedom than they can ever get from the public cloud =
provider.=20

I'm afraid I don't see the point. The purpose of the enclaves is to
process confidential data on a cloud provider's computer without
trusting the cloud provider, is it not? Running one's own computer
takes the cloud provider out of the picture. Users must trust the
hardware either way. Who is the untrusted party in the user's own
computer?

Rather this proposal allows Fedora users to setup their own private data
center/cloud hosts, and establish trust in their own managed hardware,
with a greater level of freedom than they can ever get from the public cloud
provider.

I'm afraid I don't see the point. The purpose of the enclaves is to
process confidential data on a cloud provider's computer without
trusting the cloud provider, is it not? Running one's own computer
takes the cloud provider out of the picture. Users must trust the
hardware either way. Who is the untrusted party in the user's own
computer

Confidential computing eliminates the need to trust the cloud provider admins and software. IOW, you can prove that your VM is fully confidential, even if something has compromised they hypervisor software stack. Thus, even if you own your own computer, and thus trust yourself as an admin, it still improves your safety as it protects you from exploitable software flaws in the hypervisor software (ie linux kernel + QEMU userspace), of which there have been quite a few over the years.

There is no way you can eliminate trusting the cloud provider admins and the software they run. The cloud providers can do whatever they want to the hardware stack and the hypervisor stack, including faking TDX/SGX.

There is no way you can eliminate trusting the cloud provider admins and the software they run. The cloud providers can do whatever they want to the hardware stack and the hypervisor stack, including faking TDX/SGX.

While you can emulate TDX/SGX hardware, this is trivially detectable since the faked hardware will not be capable of providing attestation reports with a digital signature that can be verified against Intel's root of trust.

To clarify this point - the processor will happily run a build of the SGX enclaves that is signed by a non-Intel key.

They must be signed with a Fedora key. Instead of uploading pre-built non-free blobs, the proposal owners should work with virtualization providers to trust such a key.

This is my position as well. For completeness - is this remotely feasible? Seems like a much more available alternative to option (2) which Intel rejected.

I'm closing this ticket for now. Once a formal Change Proposal is filed, the conversation will be held via the usual mechanisms and then brought up for a FESCo vote.

Metadata Update from @sgallagh:
- Issue close_status updated to: Insufficient data
- Issue status updated to: Closed (was: Open)

Metadata