koji

#953 drop "rpm-py-installer" requirement from PyPI koji package
Closed: Fixed by tkopecek. Opened by ktdreyer.

Closed: Fixed
Reopen Issue

We run pip install koji in Travis CI for https://github.com/ktdreyer/product-listings-manager

The rpm-py-installer library often fails to install with hard-to-diagnose errors. For example it attemps to dynamically download the whole rpm-4.11.3-release.tar.gz and install it, and sometimes setuptools times out on the download.

The Koji client has a lot of useful features and only requires rpm for a very small percentage of the overall functionality. For example, if I just want to use koji's session object to make (possibly authenticated) basic RPC calls, I don't need to import rpm for that.

The main users of rpm seem to be the "koji import" and "koji import-sig" commands. These parse an RPM's headers before uploading the RPM file to the hub. These methods could just raise if import rpm fails.

Here are some other places the koji client uses the rpm module:

  • koji.RPM_SUPPORTS_OPTIONAL_DEPS uses rpm.__version_info__. The koji.get_header_field() method uses this variable. We could make get_header_field() raise at runtime if rpm is not available.

  • koji.RawHeader class's dump method uses rpm.__dict__. The only user outside kojihub.py is koji.get_sighdr_key(), and I don't see anything that uses get_sighdr_key. I don't see anything in Git history that ever called this method, so I think this may be dead code. RawHeader (and its use of rpm) could move to kojihub.py.

Metadata Update from @tkopecek:
- Issue tagged with: discussion

Related PR #965

Recently I found that that readthedocs.org cannot currently install rpm-py-installer during the docs build process, so anything that requires the koji pypi module fails there.

Doh, that's the case for all C-based modules on readthedocs.org, so never mind :)

@jaruga Would it be usable for you, if we will drop it as a hard dependency and you've to install rm-py-installer if rpm functions are needed?

@ktdreyer

We run pip install koji in Travis CI for https://github.com/ktdreyer/product-listings-manager

https://github.com/release-engineering/product-listings-manager/blob/master/.travis.yml

As a temporary solution, you can use like this, if the pip install koji is included in the install.sh.

install:
  - travis_retry ./.travis/install.sh

This unstable issue is known issue. I am recognizing. But sorry I have not fixed it yet.
https://github.com/junaruga/rpm-py-installer/issues/110

@jaruga Would it be usable for you, if we will drop it as a hard dependency and you've to install rm-py-installer if rpm functions are needed?

Yes, I am fine for the operation.

The rpm-py-installer library often fails to install with hard-to-diagnose errors. For example it attemps to dynamically download the whole rpm-4.11.3-release.tar.gz and install it, and sometimes setuptools times out on the download.

Right now rpm-py-installer installs RPM Python binding from source dynamically as a default behavior [1]. If setup.py.in [2] does not exist in the source code (This happens on RPM version <= 4.9.x), it installs from using the RPM package not from source.

We might be able to change this behavior.

[1] https://github.com/junaruga/rpm-py-installer/blob/v0.7.0/install.py#L185-L186
[2] https://github.com/rpm-software-management/rpm/blob/master/python/setup.py.in

sometimes setuptools times out on the download.

Maybe this reason is I had to wait more than 30 seconds to download RPM source code from rpm.org server right now.

Current logic is rpm-py-installer will try to download the RPM source archive file from the official web site rpm.org at first. Then if it gets HTTP error, try to download the source code from 2nd server github.com.
But right now rpm-py-installer (http client) timeout is too long or not set.

I have to adjust this timeout time to short time such as 3 seconds.

I released rpm-py-installer new version v0.7.1 now as a quick way to improve installation to prevent the timeouts of setuptools.

Previously The server to download RPM source archive was primary server: rpm.org, secondary server GitHub.
I changed the order to primary server: GitHub, secondary server rpm.org.
Because GitHub is faster.

This way is not good enough. But we can expect better situation than previous one.

Sorry for inconvenience.

@ktdreyer Does newer rpm-py-installer work for you, or should we pursue removing dependency completely?

@jaruga I'm really grateful for your work in rpm-py-installer.

It seems to me that we could still eliminate the rpm dependency from the koji client.

@ktdreyer you are welcome. Thank you for your feedback to improve rpm-py-installer.

It seems to me that we could still eliminate the rpm dependency from the koji client.

The rpm-py-installer library often fails to install with hard-to-diagnose errors. For example it attemps to dynamically download the whole rpm-4.11.3-release.tar.gz and install it, and sometimes setuptools times out on the download.

What is the exact use case you faced?

I downloaded your project https://github.com/release-engineering/product-listings-manager

$ git clone git@github.com:release-engineering/product-listings-manager.git
$ cd product-listings-manager
$ sudo dnf -y install postgresql-devel krb5-devel rpm-devel gcc python-devel

Then I did run below cases a few times.
rpm-py-installer installing process was less than 10 seconds.
But I could not see issues.
Could you share the output log for the future improvement if you have it?

$ rm -rf venv
$ python3 -m venv ./venv
$ source venv/bin/activate
(venv) $ pip install .
...
Collecting rpm-py-installer (from koji->product-listings-manager==0.2.0)
  Using cached https://files.pythonhosted.org/packages/6f/47/d7f5a2cd4977d517edc49f794b5f72cdf966240f7ef9abcf0e8acc0b02df/rpm-py-installer-0.8.0.tar.gz
...
Edited by jaruga

To check the issue and major the entire installing time, I am using this script. :)
https://gist.github.com/junaruga/7be29ef525b1111511812b00f98efe27

$ git clone git@github.com:release-engineering/product-listings-manager.git
$ cd product-listings-manager
$ sudo dnf -y install postgresql-devel krb5-devel rpm-devel gcc python-devel
$ sh install_repeatedly.sh 2>&1 | tee -a install.log

I added a feature to install RPM Python binding from the binary package with a environment variable RPM_PY_INSTALL_BIN=true to the project's master branch.
it is not released to pypi.

You can try like this for example in your virtualenv environment.

$ RPM_PY_INSTALL_BIN=true RPM_PY_VERBOSE=true python -c "$(curl -fsSL https://raw.githubusercontent.com/junaruga/rpm-py-installer/master/install.py)"

But there is a problem. I might need your help.

This feature works on Python 3.6, but does not work on Python 3.4 and Python 3.5.
Because the binary package is built with Python 3.6 and for Python 3.6.

The detail is here. https://github.com/junaruga/rpm-py-installer/issues/152

I remember that some people worked like this kind of approach for koji project in the past time, before using rpm-py-installer.
Do you know how to work on Python 3.5 or 3.4?

Thank you.

I've seen this one again (sic!) - is it still a problem?

Metadata Update from @tkopecek:
- Custom field Size adjusted to None

@tkopecek you meant RPM_PY_INSTALL_BIN=true issue? Recently I have not maintain rpm-py-installer well.

In my memory, maybe the success of RPM_PY_INSTALL_BIN=true option depends on the binary compatibility with a python.

We've also dropped rpm-py-installer from setup as it is no more hard requirement for koji lib. So I would close this ticket, if it is no more a problem for people.

Metadata Update from @tkopecek:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

This issue has been migrated to Fedora Forge:
https://forge.fedoraproject.org/koji/koji/issues/953

Please continue any further discussion there.
Metadata
None

discussion

None
None
Low
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.