#4446 rpm v6 support
Merged by mikem. Opened by mikem.
mikem/koji rpm-v6-support  into  master

Download 4446.patch

This adds initial support for v6 rpms in Koji. The main issue is Koji's past reliance on the sigmd5 field, which does not exist in v6. The major workhorses here are new get_rpm_ident() and related functions which are used numerous places where we previously used sigmd5 to identify rpms. Additionally, we have added new fields in the rpminfo table to record the various hashes that might be contained in the signature header.

Overall, this problem is trickier that it might as first seem.

Fixes https://pagure.io/koji/issue/4200

One notable omission, the oz/imagefactory feature (i.e. createImage) is not covered here (it will fail if an image includes v6 rpms). This will be deferred as a separate item.

rebased onto a2d2e35dc1764b508d601289265e2fc92152dde3

4 new commits added

  • use koji.get_rpm_ident to id rpms in builder plugins
  • use koji.get_rpm_ident to id rpms in kojivmd
  • use koji.get_rpm_ident to id rpms in kojid
  • use "Digest" in web ui

8 new commits added

  • unit test updates
  • partial work on utilizing new rpm digest fields
  • schema updates
  • accept rpm ids in cg import
  • update metadata doc
  • drop duplicate definition
  • use filedigests header name instead of filemd5s alias
  • add missing test file

The current batch of changes are still incomplete, but give an idea of where I'm going.

I started out just trying to remap the payloadhash field to be determined by alternate fields, but remain singular. However, when I got to the point of sorting out how this would look for CG imports, I realized that we needed to be more flexible in how the CG specifies component/buildroot rpms

I've noticed RPM v6.0.0 Supports multiple OpenPGP signatures per package, Does Koji have any plans to support multi-sign?

rebased onto c19d21a94b40d111aed60b5b2c8057c1caf95aaf

I've noticed RPM v6.0.0 Supports multiple OpenPGP signatures per package, Does Koji have any plans to support multi-sign?

This is handled by #4373 and its followups.

Koji doesn't really care how many signatures are in an rpm. That said, Koji needs a single unique string key for each different signature header that it stores (used for lookup and in paths). Historically, Koji has auto-chosen this key by parsing RPM_SIGTAG_{GPG,PGP,DSA,RSA} from the signature header. For more complex v6 signatures, the client that is importing the signature should specify a consistent sigkey value at import time.

This (size) field got lost and is required later in the process

1 new commit added

  • include size in rpm ident fields

The size field in rpminfo is unfortunately inconsistent. In import_rpm, we populate it with the file size, not the size field from the header. Otoh, in add_external_rpm we use the size reported by the caller, which might be either the file size (e.g. for some image components) or the header value (e.g. for rpm buildroot components).

To further complicate matters, the size field has been removed in the v6 spec (rpm will report None).

I don't think my last commit above is the right fix. Will try another approach shortly.

1 new commit added

  • allow rpminfo.size to be null

Some debug artifact?

1 new commit added

  • drop debug output in rpminto page

Some debug artifact?

Ah, yes. An earlier version had got_digest defined by a set in the for loop, and that was me figuring out that this value did not persist after the loop in jinja.

Fixed now.

This works for me now. :thumbsup:

How is the situation with multiple GPG signatures? It seems to me that we've another family of issues there. E.g. get_sighdr_key returns only first one and we're blind to others. So, also sigcache, data/signed are referred by only first one (while sighdr contains more GPG signatures).

How is the situation with multiple GPG signatures?

See above comment - https://pagure.io/koji/pull-request/4446#comment-227271

Longer term, we could consider expanding the automatic sigkey choice to be aware of a broader range of signatures, however this could get quite complex. To do this right we'd also need to consider ima, which we've been ignoring for a while (and hypothetically, the header structures allows for each file in the rpm to be ima-signed differently, though rpmsign itself behaves more consistently).
The current code allows the client to specify the sigkey, which dodges this complexity.

Metadata Update from @tkopecek:
- Pull-request tagged with: testing-custom

Metadata Update from @mfilip:
- Pull-request untagged with: testing-custom

Temporally untagged for testing purposes

Temporally untagged for testing purposes

Is there a merge issue? Lmk if I need to rebase something

Ah, I see this overlaps with #4472. The conflict is pretty straightforward to resolve. I'll rebase one whenever the other is merged.

Metadata Update from @mfilip:
- Pull-request tagged with: testing-custom

rebased onto fa4cb46cafae4d1ccae3b19d6d1fad708c343bcc

Rebased. Minor conflict with #4472

Metadata Update from @mfilip:
- Pull-request tagged with: testing-done

Commit 86057e1b fixes this pull-request

Pull-Request has been merged by mikem

Metadata