#36 rpkg v2 and v3 compatible spec syntax?
Closed by clime. Opened by praiskup.

I hoped that I can commit a fix to one of our Copr testing packages so we'll be able to build this both with rpkg-util v2 and v3:

https://pagure.io/copr/copr-hello/c/450749e5eef7e1da9d2ba12ac0383a65f967a306?branch=noluck

What I did wrong? Prior to that commit, the package worked with v2, but not with v3.

Rpkg v3:

[praiskup@7c8dcad83685 copr-hello]$ rpm -q rpkg
rpkg-3.0-2.fc35.noarch
[praiskup@7c8dcad83685 copr-hello]$ git checkout 3b4bbd3b8fb7e220946171915e4f31db4542ab4e
HEAD is now at 3b4bbd3 update
[praiskup@7c8dcad83685 copr-hello]$ rpkg srpm
Wrote: /tmp/rpkg/example-15-e6l96gtu/example.spec
error: Bad source: /tmp/rpkg/example-15-e6l96gtu/example-1.0.13.tar.gz: No such file or directory
[praiskup@7c8dcad83685 copr-hello]$ git checkout noluck
Previous HEAD position was 3b4bbd3 update
Switched to branch 'noluck'
Your branch is up to date with 'origin/noluck'.
[praiskup@7c8dcad83685 copr-hello]$ rpkg srpm
/usr/lib/rpkg.macros.d/library/log.bash: line 27: /dev/stderr: Permission denied
/usr/lib/rpkg.macros.d/library/log.bash: line 27: /dev/stderr: Permission denied
Wrote: /tmp/rpkg/example-16-rbrf4nts/example.spec
setting SOURCE_DATE_EPOCH=1518912000
Wrote: /tmp/rpkg/example-16-rbrf4nts/example-0.0.git.74.450749e5-1.fc35.src.rpm

Rpkg v2

[praiskup@6b2d5bc6fbd3 copr-hello]$ rpm -q rpkg
rpkg-2.10-1.fc34.noarch
[praiskup@6b2d5bc6fbd3 copr-hello]$ git checkout 3b4bbd3b8fb7e220946171915e4f31db4542ab4e
HEAD is now at 3b4bbd3 update
[praiskup@6b2d5bc6fbd3 copr-hello]$ rpkg srpm
Wrote: /tmp/rpkg/example-5-8c8fvarl/example.spec
Wrote: /tmp/rpkg/example-5-8c8fvarl/example-1.0.13.tar.gz
auto-packing: This function is deprecated and will be removed in a future release.
setting SOURCE_DATE_EPOCH=1518912000
Wrote: /tmp/rpkg/example-5-8c8fvarl/example-1.0.13-1.fc34.src.rpm
[praiskup@6b2d5bc6fbd3 copr-hello]$ git checkout noluck
Previous HEAD position was 3b4bbd3 update
Switched to branch 'noluck'
Your branch is up to date with 'origin/noluck'.
[praiskup@6b2d5bc6fbd3 copr-hello]$ rpkg srpm
git_dir_version: name cannot be empty.
git_dir_version failed with value 1

Since we have no option to keep auto_pack = True in Copr, we need to prepare our user-base for migration; though I don't know what I should suggest them to do now to get prepared...


It should work if you include:

Name: {{{ git_dir_name }}}

in spec file as well. Invoking the git_dir_name macro will populate rpkg macro context with git_name variable that other macros like git_dir_version use afterwards if they need it. It's described here:

https://docs.pagure.org/rpkg-util/v3/macro_reference.html#base-git-macros

in the paragraph starting: "Output of every base git macro call is cached..."

Another option is to pass name to git_dir_version macro directly:

Version: {{{ git_dir_version name=example }}}


I don't know what /usr/lib/rpkg.macros.d/library/log.bash: line 27: /dev/stderr: Permission denied error is about. For some reason > /dev/stderr fails with permission error...


Since we have no option to keep auto_pack = True in Copr, we need to prepare our user-base for migration; though I don't know what I should suggest them to do now to get prepared...

Spec templates (the {{{...}}} stuff in a spec file) was designed to replace auto-packing so I guess that's the thing that should be recommended as the replacement. Specifically {{{ git_dir_pack }}} as you use it in the example. If git_dir_pack is used, it's also useful to use git_dir_setup_macro as you do.

If git_dir_version (or git_dir_release, or git_dir_changelog) is used, it's also good to use git_dir_name first to populate the context with the package name - that's not needed if only git_dir_pack + git_dir_setup_macro are used (they don't need to know the name).

If git_dir_version (or git_dir_release, or git_dir_changelog) is used, it's also good to use git_dir_name first to populate the context with the package name - that's not needed if only git_dir_pack + git_dir_setup_macro are used (they don't need to know the name).```

Well, it's not needed for v3. In v2, the git_dir_pack + git_dir_setup_macro pair needed to have git_(dir_)name and git_(dir_)version previously invoked or having dir_name/source_name params passed in explicitly. Sorry, I forgot about this. So the way to have rpkg v2 and v3 compatible (templated) spec file is to use all four git_dir_name, git_dir_version, git_dir_pack, git_dir_setup_macro.

Ok, so we need to suggest using all git_dir* statements (or nothing). Just please note that the behavior
should be more clear (a decent message what is going on would help).

Soo, the last problem is that the project copr-hello.git contains example.spec. When I use Name: {{{ git_dir_name }}}, the name of the package changes from example to copr-hello. Any opinion about this?

Ok, so we need to suggest using all git_dir* statements (or nothing). Just please note that the behavior
should be more clear (a decent message what is going on would help).

Well, for v2, there is the message:

git_dir_version: name cannot be empty.
git_dir_version failed with value 1

which points to the right direction. For v3, there should be the same thing but there seems to be some unrelated permission problem...

Soo, the last problem is that the project copr-hello.git contains example.spec. When I use Name: {{{ git_dir_name }}}, the name of the package changes from example to copr-hello. Any opinion about this?

For a custom name, you can use e.g.:

Name: {{{ git_name name=example }}}

Thanks, so last thing.... I updated the noluck branch, and it looks like it is working, but it isn't.

Rpkg v3 generates /tmp/rpkg/example-2-o4c0qnng/example-0.0.git.74.d77c8e03-1.fc35.src.rpm, while Rpkg v2 generates /tmp/rpkg/example-3-pyop52nb/example-0.0.git.9.d77c8e0-1.fc34.src.rpm.
```
See the invalid tag there. Any idea?

You can reproduce the stderr problem with:

$ podman run --rm -ti fedora:35
[root@4a951930fcbf /]# dnf install -y rpkg git python3-setuptools
...
[root@4a951930fcbf /]# useradd tester
[root@4a951930fcbf /]# su - tester
[tester@4a951930fcbf ~]$ git clone https://pagure.io/copr/copr-hello
Cloning into 'copr-hello'...
cdremote: Enumerating objects: 316, done.
remote: Counting objects: 100% (316/316), done.
remote: Compressing objects: 100% (170/170), done.
remote: Total 316 (delta 127), reused 258 (delta 106), pack-reused 0
Receiving objects: 100% (316/316), 5.32 MiB | 6.36 MiB/s, done.
Resolving deltas: 100% (127/127), done.
[tester@4a951930fcbf ~]$ cd copr-hello
[tester@4a951930fcbf copr-hello]$ git checkout noluck
Branch 'noluck' set up to track remote branch 'noluck' from 'origin'.
Switched to a new branch 'noluck'
[tester@4a951930fcbf copr-hello]$ rpkg srpm
/usr/lib/rpkg.macros.d/library/log.bash: line 27: /dev/stderr: Permission denied
/usr/lib/rpkg.macros.d/library/log.bash: line 27: /dev/stderr: Permission denied
Wrote: /tmp/rpkg/example-1-lk1c8wqv/example.spec
setting SOURCE_DATE_EPOCH=1518912000
Wrote: /tmp/rpkg/example-1-lk1c8wqv/example-0.0.git.74.d77c8e03-1.fc35.src.rpm

Thanks, so last thing.... I updated the noluck branch, and it looks like it is working, but it isn't.

Rpkg v3 generates /tmp/rpkg/example-2-o4c0qnng/example-0.0.git.74.d77c8e03-1.fc35.src.rpm, while Rpkg v2 generates /tmp/rpkg/example-3-pyop52nb/example-0.0.git.9.d77c8e0-1.fc34.src.rpm.
```
See the invalid tag there. Any idea?

It's because the version in tag is not in in an expected format:

example-1.0.14-1 - the version is 1.0.14, i.e. lead is 1.0 whereas default lead for git_version macro is 0. You can fix this for rpkg v3 by using {{{ git_dir_version lead= }}} in which case the lead is just derived from the tags. For v2 this fix is impossible so to have a compatible spec file, you would need to specify {{{ git_dir_version lead=1.0 }}}. lead is whatever there is before the last dot in the version.

You can reproduce the stderr problem with:

Somehow podman got stuck at "Storing signatures" at my computer when spawning the container. Does echo foo > /dev/stderr work for you? this seems to be some container-related permission issue. I guess try it without switching to tester user.

This is going to be a painful change... I'm afraid. I bet the reason rpkg was used
as default Copr method was that it was mostly doing rpkg srpm from git forges painless ... but honestly I still can't fix this. I really wanted to have the compatible spec,
but lead=1.0 did not help, rpkg v3 still provides: example-1.0.0.git.74.d01355c8-1.fc35.src.rpm

This is going to be a painful change... I'm afraid. I bet the reason rpkg was used
as default Copr method was that it was mostly doing rpkg srpm from git forges painless ... but honestly I still can't fix this. I really wanted to have the compatible spec,
but lead=1.0 did not help, rpkg v3 still provides: example-1.0.0.git.74.d01355c8-1.fc35.src.rpm

Basically, for packages with custom names (that do not derive their names from directories they are placed in) you shouldn't be using git_dir_* macros but just base git_ macros. In the case of example package, it would be {{{ git_version lead=1.0 }}} then.

You can find those answers also in documentation:
https://docs.pagure.org/rpkg-util/v3/macro_reference.html#git-dir-version

Also here: https://docs.pagure.org/rpkg-util/v3/macro_reference.html#git-dir-macros

NOTE: the git_dir macros are the recommended ones to use from all the macro types. Use them unless you would need to override a value that the the macro sets itself. In that case, the base git macros are a better choice.

The git_dir_version macro sets the name itself (overriding any cached value) so here you would need set the name yourself to example explicitly. Its therefore better to use the base git_ macro in this case.

You can keep using git_dir_pack and git_dir_setup_macro however as these do not work with the name attribute but they work with path only.

Here is the lead description btw.:
https://docs.pagure.org/rpkg-util/v3/macro_reference.html#git-version

Ok, now it seems circa OK, one difference is:

+ /tmp/rpkg/example-11-lioytkrk/example-1.0.13.git.9.45ded1e4-1.fc35.src.rpm
- /tmp/rpkg/example-5-3m2gxv2h/example-1.0.13.git.9.45ded1e-1.fc34.src.rpm

The new rpkg-util provides one more character in the git hash string. This is against
the "reproducibility" principle (if you get the same version of sources, you get the
same tarball or src.rpm, at least tito is kind of working here correctly).

I think I I have a patch that we'll suggest to Copr users when they are migrating (and the rest of the questions will be forwarded to @rpkg-util upstream :P).

Thank you for your time!

(I can't close this for some reason)

Ok, now it seems circa OK, one difference is:

diff + /tmp/rpkg/example-11-lioytkrk/example-1.0.13.git.9.45ded1e4-1.fc35.src.rpm - /tmp/rpkg/example-5-3m2gxv2h/example-1.0.13.git.9.45ded1e-1.fc34.src.rpm

The new rpkg-util provides one more character in the git hash string. This is against
the "reproducibility" principle (if you get the same version of sources, you get the
same tarball or src.rpm, at least tito is kind of working here correctly).

I don't think the extra char breaks reproducibility somehow or does it? 7 chars means occassional conflicts in the short hashes so I tried to avoid them...

I think I I have a patch that we'll suggest to Copr users when they are migrating (and the rest of the questions will be forwarded to @rpkg-util upstream :P).

Thank you for your time!

No problem. Sorry that it wasn't as smooth as you would probably imagine ;).

(I can't close this for some reason)

I'll close it then. Feel free to reopen/add any comments in case there are some further things to discuss...

Metadata Update from @clime:
- Issue status updated to: Closed (was: Open)

This issue has been migrated to GitHub: https://github.com/fedora-copr/rpkg-util/issues/36
Don't forget to subscribe for notifications about new comments.

Metadata