After merging https://pagure.io/fedora-infra/ansible/pull-request/456 our copr-frontend playbook started failing because of a task from fas_client.
TASK [run fas_client only if we just installed] *********************************************************************************************************************************************************************************************** Tuesday 16 March 2021 18:25:25 +0000 (0:00:00.078) 0:01:56.998 ********* Tuesday 16 March 2021 18:25:25 +0000 (0:00:00.078) 0:01:56.997 ********* fatal: [copr-fe-dev.aws.fedoraproject.org]: FAILED! => {"changed": true, "cmd": ["fasClient", "-if"], "delta": "0:00:00.446293", "end": "2021-03-16 18:25:26.028764", "msg": "non-zero return code", "rc": 1, "start": "2021-03-16 18:25:25.582 471", "stderr": "Traceback (most recent call last):\n File \"/usr/bin/fasClient\", line 911, in <module>\n users = fas.filter_users(valid_groups=valid_groups, restricted_groups=restricted_groups)\n File \"/usr/bin/fasClient\", line 37 2, in filter_users\n if group not in self.groups:\n File \"/usr/bin/fasClient\", line 252, in _refresh_groups\n group_data = self.group_data(force_refresh=self.force_refresh)\n File \"/usr/lib/python3.9/site-packages/fedora/client/ fas2.py\", line 927, in group_data\n request = self.send_request(\n File \"/usr/lib/python3.9/site-packages/fedora/client/baseclient.py\", line 365, in send_request\n session_id, data = super(BaseClient, self).send_request(\n File \"/usr/lib/python3.9/site-packages/fedora/client/proxyclient.py\", line 478, in send_request\n raise ServerError(url, http_status, msg)\nfedora.client.ServerError: ServerError(https://admin.stg.fedoraproject.org/accounts/json/fas_client /group_data, 404, Not Found)", "stderr_lines": ["Traceback (most recent call last):", " File \"/usr/bin/fasClient\", line 911, in <module>", " users = fas.filter_users(valid_groups=valid_groups, restricted_groups=restricted_groups)", " File \"/usr/bin/fasClient\", line 372, in filter_users", " if group not in self.groups:", " File \"/usr/bin/fasClient\", line 252, in _refresh_groups", " group_data = self.group_data(force_refresh=self.force_refresh)", " File \"/ usr/lib/python3.9/site-packages/fedora/client/fas2.py\", line 927, in group_data", " request = self.send_request(", " File \"/usr/lib/python3.9/site-packages/fedora/client/baseclient.py\", line 365, in send_request", " session_id, d ata = super(BaseClient, self).send_request(", " File \"/usr/lib/python3.9/site-packages/fedora/client/proxyclient.py\", line 478, in send_request", " raise ServerError(url, http_status, msg)", "fedora.client.ServerError: ServerError(ht tps://admin.stg.fedoraproject.org/accounts/json/fas_client/group_data, 404, Not Found)"], "stdout": "", "stdout_lines": []}
This happens only for copr-fe-dev.aws.fedoraproject.org, the production instance copr-fe.aws.fedoraproject.org works fine.
copr-fe-dev.aws.fedoraproject.org
copr-fe.aws.fedoraproject.org
Can you please fix the issue or give us advice how to do it ourselves?
We have an outage planned for tomorrow, please see #9734 If we cannot resolve the issue till then, we will have to temporarily disable to fas_client role again.
Our staging setup no longer uses fas. :) So, this is expected. :)
You will see all the other staging servers have fas disabled.
I would have mentioned it sooner, but I didn't realize the copr dev instances were using staging fas. :(
So, options:
I guess I'd just say to remove it, but open to other ideas based on your needs.
Thank you for the explanation @kevin,
I tried to disable it for staging instance and use it only for the production one
https://pagure.io/fedora-infra/ansible/c/700167c1fc67b198c9fee4f3250d0b7f8785280f?branch=main
but then the playbook failed for the production instance. So I decided to drop it entirely
https://pagure.io/fedora-infra/ansible/c/110786d513d1672da2511c3d09017dcaf956b9b8?branch=main
Metadata Update from @frostyx: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)