I am writing to seek your expertise about using QEMU as a Koji Builder to build packages for the aarch64 on the fedora platform.
When Koji invokes mock for building, it spends a significant amount of time updating yum metadata(≈15mins, set as external repo) resulting in taking 30 minutes to init buildroot alone (download speed≈2.8MB/s). I tried to reconfigure "yum_cache" and "root_cache" through both koji and mock but it seems that kojid overrides and False them every time.
I would like to know if there is any information that I may have overlooked or any guidance or recommendations you may have to speed up init buildroot process on QEMU builder.
Caches are disabled as it could poison the build environment with incorrect package versions, etc. So, it is always disabled.
I would say that easiest way would be to setup some caching proxy (it could even live on builder, if there are no other better options) for kernel repo which would builder have fast access to.
Hackish (wrong) way (depend on your usecase) is to modify koji/__init__.py:genMockConfig to not override these values. But this comes with high risk.
koji/__init__.py:genMockConfig
Metadata Update from @tkopecek: - Custom field Size adjusted to None
Caches are disabled as it could poison the build environment with incorrect package versions, etc. So, it is always disabled. I would say that easiest way would be to setup some caching proxy (it could even live on builder, if there are no other better options) for kernel repo which would builder have fast access to. Hackish (wrong) way (depend on your usecase) is to modify koji/__init__.py:genMockConfig to not override these values. But this comes with high risk.
Thank you for your prompt reply and helpful suggestion. I appreciate your advice on setting up a local caching proxy to imporve Koji's performance. I will look into this option and try to configure my Koji instance accordingly.
Metadata Update from @tkopecek: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Metadata Update from @toshihiko-chen: - Issue status updated to: Open (was: Closed)
I set up squid as cache proxy for fedora mirror and modified /etc/kojid.conf: "yum_proxy =", however the "dnf --installroot" process still costs a lot of time, am I missing sth?
I hacked "koji/init.py" but the mock log says "cleaning package manager metadata" after "enabled package manager cache", is that normal? cuz it didn't speed up after I reconfigured.
Sorry for keep asking questions here, there's too little information on google about what I am doing.
Messages are ok - it will clean the cache inside the buildroot before it will connect system cache there. About if it really works - I can't do much to investigate it here. Are subsequent runs taking same time as first one (is cache populated?) Is time spent in downloading or in installation itself?
I timestamped the root.log, and most of the time was spent downloading yum metadata during "--installroot".
And how can I reconfigure "yum_cache", "root_cache" to False? I turned "mockbuild/config.py", "koji/init.py", "mock/site-defaults.cfg" to False and when I "$koji mock-config", it shows they are False, However when I "$koji build", the mock config generated in "/etc/mock/koji/xxxx.cfg" still shows True.
Also, I check the mock-doc, and the feature "nosync" may be helpful.
koji/__init__.py must be updated at builder also. Maybe that is missing? (it looks that machine where CLI is used it is updated)
koji/__init__.py
Metadata will be probably downloaded more often (but not the packages). Anyway, metadata are downloaded from hub, so it should be as fast as the link between.
You are right! I did miss the init.py at builder. Now it's correct.
May I ask if there is doc about how builder download metadata from hub? or is there some config parameters are related to that?
Nothing interesting related to that. Metadata are created for current buildroot (manually or via kojira). RPMs are then downloaded either from koji or from external repo. But repodata are just one referencing all of this. https://docs.pagure.org/koji/exporting_repositories/#koji-s-internal-repositories
Do you mean when koji build, the builder will go to get repodata in "/mnt/koji" which is generated by "$koji regen-repo"?
My external repo is not local, so every time when I regen-repo, the mergerepo_c process will also cost 30mins. maybe the builder didn't find the local repodata, so every time it has to download huge mirror metadata from 0?
Yes, but it goes there via http/s as it needn't be mounted. Ando for second question - no. It will always download data from the hub or fail. Maybe in such case it would be better to really make mirror version of that external repo (e.g. via some simple cron/rsync) and use that mirrored one as external repo. It would reduce the complexity of caching.
I pulled the fedora mirror to local and set it as the external repo, but the "dnf --installroot" of the build process still costs the same long time(20mins), and I tested the speed of IO from "/mnt/koji" to "kojibuilder", and it's approximate 60MB/s, so it's not netio or diskio bottleneck the speed of "dnf --installroot"?
netio - it is being downloaded via http/s (topurl in kojid.conf)
Do you have any idea where to config "yum_cache"'s directory?
In "/usr/lib/python3.11/site-packages/mockbuild/config.py".
I only found the config option "dir" of "root_cache" and "ccache", but there's no yum_cache's "dir"
It is a mock's plugin. I don't think, it is configurable: https://github.com/rpm-software-management/mock/blob/main/mock/py/mockbuild/plugins/yum_cache.py#L31
It's not configurable, but I compress the yum_cache into root_cache, and change root_cache directory to solid one, now it works!
But the build process still not fast, I am using qemu to emulate X86_64 to aarch64, is this the method of how "koji.fedora.org" build different arch rpms?
Do you know how "koji.fedora.org", set up and configure their "aarch64" kojibuilders?
Fedora's koji use real aarch64 HW, not cross-emulation. In fact I don't know about any deployment which uses cross-emulated VMs (but for some (s390x) it was reported that it could be of reasonable speed).
I see, It's better to give up this deployment then.
Thanks a lot.
Metadata Update from @toshihiko-chen: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
This issue has been migrated to Fedora Forge: https://forge.fedoraproject.org/koji/koji/issues/3735
Please continue any further discussion there.