pagure

#2504 Listing projects is too slow
Closed: Fixed by wombelix. Opened by ralph.

Closed: Fixed
Reopen Issue

See this:

~❯ time http get "https://src.fedoraproject.org/api/0/projects?namespace=rpms&page=1&per_page=50"

That takes ~30 seconds. There are 480 page. That means querying for all projects will take about 4 hours. Any script that needs to list all pagure projects (on fedora dist-git) is going to have a hard time.

Can we speed that up? Any analysis on why it takes so long?

Not sure what goes on, locally I can retrieve all the projects in ~1 minutes while it takes more than 10 minutes in prod :(

Metadata Update from @pingou:
- Issue tagged with: RFE

Metadata Update from @pingou:
- Issue untagged with: RFE
- Issue tagged with: bug

@kevin do you think tweaking the wsgi wrapper would help here like it did for datagrepper?

Huh, not sure, it's 5-10 seconds per load here... but it's the weekend now, so loading of the site might be causing it to be slow?

I think we could increase settings in wgsi, we have a lot of memory free still. If @pingou is ok I'd be happy to play with the settings.

I think we could increase settings in wgsi, we have a lot of memory free still. If @pingou is ok I'd be happy to play with the settings.

By all means, go for it :)

@kevin please let me know when we can do some trials against the changes.

The big one for me is getting this to return in a timely manner.

https://src.fedoraproject.org/api/0/projects?owner=orphan

The big one for me is getting this to return in a timely manner.
https://src.fedoraproject.org/api/0/projects?owner=orphan

Maybe the script could be adjust to query https://src.fedoraproject.org/extras/pagure_poc.json instead?

Or we could generate a new json file with just the list of package having orphan as main admin.

@pingou - that loads really quick. Is that a stable/guaranteed API? It looks like it lists packages & owners which is, as far as I know right now, all I need.

It's a cron job that generates these JSON files on a hourly basis iirc, so it should be pretty stable and independent of other breakage we could accidentally do in pagure.

Check the folder for a couple of other files, but I think this one if the most interesting one.

If needed, we could run the cron more frequently since it takes under a minute to generate all three files currently.

I'm inclined to close this one now, I think the cron job does a better job and is faster. If people have the issue with pagure.io then we might reconsider but I think for now it should be fine.

What do you think?

no updates / feedback since 4 years, closing.

Metadata Update from @wombelix:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

That remains an issue if you're part of groups that have hundreds of projects
(say go-sig, rust-sig, perl-sig or python-sig)

Sorry, wrong ticket, I was thinking of another one :(
Metadata
None

bug

None
None
None
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.