taskotron / libtaskotron

#276 look at bodhi retry code whether it's working at intended
Closed: Fixed Opened by kparal.

Closed: Fixed
Reopen Issue

The Bodhi client in python-fedora has been heavily reworked lately. From our logs, it's not clear whether server error codes are correctly retried (at least in certain cases):
https://taskotron.fedoraproject.org/taskmaster/builders/x86_64/builds/123034/steps/runtask/logs/taskotron.log

[libtaskotron:bodhi_utils.py:113] 2015-09-02 04:24:23 INFO    Querying Bodhi to map 18 builds to their updates...
[libtaskotron:logger.py:98] 2015-09-02 04:28:17 CRITICAL Traceback (most recent call last):
  File "/usr/bin/runtask", line 9, in <module>
    load_entry_point('libtaskotron==0.3.23', 'console_scripts', 'runtask')()
  File "/usr/lib/python2.7/site-packages/libtaskotron/runner.py", line 272, in main
    task_runner.run()
  File "/usr/lib/python2.7/site-packages/libtaskotron/runner.py", line 65, in run
    self.do_actions()
  File "/usr/lib/python2.7/site-packages/libtaskotron/runner.py", line 156, in do_actions
    self.do_single_action(action)
  File "/usr/lib/python2.7/site-packages/libtaskotron/runner.py", line 127, in do_single_action
    self.envdata)
  File "/usr/lib/python2.7/site-packages/libtaskotron/directives/python_directive.py", line 214, in process
    output = self.execute(task, method_name, input_data)
  File "/usr/lib/python2.7/site-packages/libtaskotron/directives/python_directive.py", line 155, in execute
    output = task_method(**kwargs)
  File "run_depcheck.py", line 56, in taskotron_run
    return depcheck.run(rpms, arch, repositories, report_format, workdir, artifactsdir)
  File "/home/buildslave/slave/x86_64/build/depcheck/__init__.py", line 128, in run
    output = squash_results.format_output(run_result, arch, report_format, artifactsdir)
  File "/home/buildslave/slave/x86_64/build/depcheck/squash_results.py", line 140, in format_output
    update_results = _squash_builds_to_updates(build_results)
  File "/home/buildslave/slave/x86_64/build/depcheck/squash_results.py", line 57, in _squash_builds_to_updates
    pending_ok, pending_fail = bodhi_client.build2update(results_nvr.keys(), strict = True)
  File "/usr/lib/python2.7/site-packages/libtaskotron/bodhi_utils.py", line 119, in build2update
    res = self.client.query(builds=builds_chunk)
  File "/usr/lib/python2.7/site-packages/fedora/client/bodhi.py", line 96, in wrapper
    result = method(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/fedora/client/bodhi.py", line 234, in query
    return self.send_request('updates', verb='GET', params=kwargs)
  File "/usr/lib/python2.7/site-packages/fedora/client/openidbaseclient.py", line 374, in send_request
    {'url': to_bytes(method), 'err': to_bytes(e)})
ServerError: ServerError(https://bodhi.fedoraproject.org/updates, 500, Error returned from json module while processing https://bodhi.fedoraproject.org/updates: Expecting value: line 1 column 1 (char 0))

They might have broken their internal retry functionality. Please have a look at it and verify (e.g. with http://httpstat.us/) whether it correctly retries attempts or not. If it doesn't, either fix it or create a ticket for them to fix it. If it does, look at how they do logging and consider increasing logging level for python-fedora (or maybe just Bodhi loggers inside that) so that we see retries in our debug log. Alternatively, if having Bodhi library log level to DEBUG would flood our log too much, suggest to Bodhi devs to increase the logging level of the retry attempt (e.g. to INFO or WARNING), so that it's easier for us to include those messages in our logs.

We're getting a substantial amount of these errors every day. We might want to look at this sooner than later.

I filled an issue on github https://github.com/fedora-infra/python-fedora/issues/144

I think this can be closed, no?

Yes, the new python-fedora update seems to finally resolve this.

Metadata Update from @kparal:
- Issue tagged with: easyfix
Metadata
None

easyfix

None
None
Low
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.