Assuming this is anubis stopping something. Maybe something we can tweak on the anubis side.

Also sourcegraph can probably tweak something to make anubis happier.

Is this now working again as expected? Or no?

And yes, if it's still happening we would love to have some contacts we could talk with about it.
In general if you are seeing anubis from a script, make sure you are setting user-agent and accept headers.
The new anubis version challenges things without those headers.

According to the example, yes it is still failing. I do not have the contacts to see what's going on in there, but last time @jflory7 mentioned he had a contact though.

Although it does not have to necessarily be sourcegraph, any indexer would do, but pagure doesn't have one, and forgejo's is at least existent, but not too much beyond that. I did notice in a thread though that there is some path on src.fp.o that had all spec files or something like that?

Yeah, if we could get contacts from @jflory7 that would be good.

Hi all. My contact for this is Justin Dorfman. I can reach out to him on LinkedIn for input. I will share the link to this issue with him, as well as a heads-up that we are in a slow but steady march to Forgejo.

For this specific issue though, is there more context I can provide to him about the issue at hand? Or more likely, is there some technical knowledge that I could give to Justin Dorfman for him to share with Sourcegraph engineers? Other than failure to index with our new AI scraping protection in place, I'm not sure what else to share with him.

I'd suggest:

Please check your crawling of src.fedoraproject.org and see why it's failing. If it's getting an anubis challenge, please check that you are sending a user-agent and a accept header (in the vast majority of cases this solves the issue and you won't get challenged). If it doesn't solve the issue, let us know and we can try and come up with another way to allow your access (by user-agent, by network/ips, etc).

Thanks Kevin, I passed this comment and a link to this ticket in a LinkedIn message this afternoon. I'll wait a bit for a follow-up before trying to get Sourcegraph engagement through other means.

Hey all!

This has been fixed on our end. Please let us know if there are any other issues.

@jdorfman Woohoo, thanks so much for the follow-up! :pray: Our team will look into this and confirm before closing the issue.

Hi @jdorfman glad to have you around here, and thanks for looking into it on your side. Could we bother you with some of the technical details?
- In the example of the description, I see that a couple of them no longer have the old indexing warning, but most of them still do. Will this recover eventually? And is it dependent on the searches that someone queries or it just have to churn through all of the repos?
- Could you share some of how the indexing is done, how it was fixed etc? A concern is that if you are using an anubis backdoor that would be patched up. Hopefully you are not using the http interface at all, and just git transactions, but then it is not quite clear what was causing the outage.
- Is there a monitoring end-point that we could also check? Maybe from where that screenshot was taken?

Hey @lecris

• In the example of the description, I see that a couple of them no longer have the old indexing warning, but most of them still do. Will this recover eventually? And is it dependent on the searches that someone queries or it just have to churn through all of the repos?

It's weird because I just ran a manual indexing job and didn't get any errors. With that said the permanent fix requires the admins of the Pagure instance to allowlist the User-Agent of the Sourcegraph instance (Sourcegraph-Bot) so it bypasses the Anubis challenge.

Could you share some of how the indexing is done, how it was fixed etc? A concern is that if you are using an anubis backdoor that would be patched up. Hopefully you are not using the http interface at all, and just git transactions, but then it is not quite clear what was causing the outage.

We've been using the same interface since day one. Nothing has changed in that regard. The client uses pure HTTP for API communication. Git operations happen later via the clone URLs returned in the API responses.

This is the same pattern for all code hosts Sourcegraph supports (GitHub, GitLab, Bitbucket, etc.) discover via API, mirror via Git.

This was the fix on our end: req.Header.Set("Accept", "application/json")

Is there a monitoring end-point that we could also check? Maybe from where that screenshot was taken?

Unfortunately, no, there's no monitoring endpoint. This is from our admin panel. It's super locked down. That said, I can manually get you any related data you need.

Let me know if you have any more questions. Thanks!

the permanent fix requires the admins of the Pagure instance to allowlist the User-Agent of the Sourcegraph instance (Sourcegraph-Bot) so it bypasses the Anubis challenge.

well, we can... but as you see, setting the accept header seems to make current versions of anubis happy.

If that doesn't work, we can allowlist, just let us know.

@kevin oh okay, my bad. If everything's working now, let's keep it the way it is. :)

Sure. Let us know if it's not or you run into anything... happy to adjust things if needed.