While attempting to follow SOP: https://docs.fedoraproject.org/en-US/infra/sysadmin_guide/registry/ seems a playbook needs updating: manual/oci-registry-prune.yml
manual/oci-registry-prune.yml
Preventing us running manual pruning jobs on the image registry, so sooner is good.
Task is failing at two locations:
TASK [Find and Delete 30 days old OCI images] ************************************************************************************************************************************************************************************************ Friday 14 April 2023 10:17:11 +0000 (0:00:00.260) 0:00:00.260 ********** Friday 14 April 2023 10:17:11 +0000 (0:00:00.259) 0:00:00.259 ********** skipping: [oci-candidate-registry01.stg.iad2.fedoraproject.org] An exception occurred during task execution. To see the full traceback, use -vvv. The error was: TypeError: 'NoneType' object is not subscriptable fatal: [oci-candidate-registry01.iad2.fedoraproject.org -> compose-x86-01.iad2.fedoraproject.org]: FAILED! => {"changed": false, "module_stderr": "Traceback (most recent call last):\n File \"<stdin>\", line 107, in <module>\n File \"<stdin>\", line 99, in _ansiballz_main\n File \"<stdin>\", line 47, in invoke_module\n File \"<frozen runpy>\", line 226, in run_module\n File \"<frozen runpy>\", line 98, in _run_module_code\n File \"<frozen runpy>\", line 88, in _run_code\n File \"/tmp/ansible_delete_old_oci_images_payload_vap0j3jb/ansible_delete_old_oci_images_payload.zip/ansible/modules/delete_old_oci_images.py\", line 166, in <module>\n File \"/tmp/ansible_delete_old_oci_images_payload_vap0j3jb/ansible_delete_old_oci_images_payload.zip/ansible/modules/delete_old_oci_images.py\", line 144, in main\nTypeError: 'NoneType' object is not subscriptable\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}
and
TASK [Run registry garbage collection to reclaim disk space (stg)] *************************************************************************************************************************************************************************** Friday 14 April 2023 10:17:43 +0000 (0:00:00.119) 0:00:32.301 ********** Friday 14 April 2023 10:17:43 +0000 (0:00:00.119) 0:00:32.301 ********** fatal: [oci-candidate-registry01.stg.iad2.fedoraproject.org]: FAILED! => {"changed": true, "cmd": ["registry", "garbage-collect", "/etc/docker-distribution/registry/config.yml"], "delta": "0:00:00.017506", "end": "2023-04-14 10:17:44.805204", "msg": "non-zero return code", "rc": 1, "start": "2023-04-14 10:17:44.787698", "stderr": "failed to garbage collect: failed to mark: filesystem: Path not found: /docker/registry/v2/repositories", "stderr_lines": ["failed to garbage collect: failed to mark: filesystem: Path not found: /docker/registry/v2/repositories"], "stdout": "", "stdout_lines": []}
Yep. This was noted in https://pagure.io/fedora-infrastructure/issue/11195 already. ;)
I'd love someone to dig in and figure out what broke in our ansible library there...
Metadata Update from @kevin: - Issue close_status updated to: Duplicate - Issue status updated to: Closed (was: Open)