#2891 [1.24.1] kojid will not reconnect to the hub after network problems.
Opened by frank-mdc. Modified

When the connection to the hub is lost, kojid will simplex exit.
I think it will be better, when it will retry to reconnect to the hub all seconds.


In which situation? Except initial login it should retry all calls.

Metadata Update from @tkopecek:
- Custom field Size adjusted to None

Yes when kojid starts it will retry to connect to the hub.

But it looks like, that when the connection was established and then the hub server is rebooted, then no reconnection will done. Only 2 times an reconnection is tried, but in some situations more try's with an delay between is needed. I have not found any option to set an reconnection delay and maxium retry count.
/var/log/kojid.log:
2021-05-26 08:33:39,707 [INFO] koji.TaskManager: Expiring subsession 841 (task 510)
2021-05-26 08:33:54,779 [INFO] koji.TaskManager: pids: {508: 3262299}
2021-05-26 08:33:54,787 [INFO] koji.TaskManager: Task 508 (pid 3262299) exited with status 0
2021-05-26 08:33:54,824 [INFO] koji.TaskManager: Expiring subsession 839 (task 508)
2021-05-27 07:32:36,086 [INFO] koji: Try #1 for call 253614 (listBuildroots) failed: HTTPSConnectionPool(host='koji-hub.foo.foo', port=443): Max retries exceeded with url: /kojihub?session-id=714&session-
key=2-JnRs8GeD6uM7mjKSPpu&callnum=253613 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused
',))
2021-05-27 07:32:36,088 [INFO] koji: Try #2 for call 253614 (listBuildroots) failed: HTTPSConnectionPool(host='koji-hub.foo.foo', port=443): Max retries exceeded with url: /kojihub?session-id=714&session-
key=2-JnRs8GeD6uM7mjKSPpu&callnum=253613 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused
',))
2021-05-27 07:34:53,308 [INFO] koji: Try #1 for call 253639 (listBuildroots) failed: HTTPSConnectionPool(host='koji-hub.foo.foo', port=443): Max retries exceeded with url: /kojihub?session-id=714&session-
key=2-JnRs8GeD6uM7mjKSPpu&callnum=253638 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused
',))
2021-05-27 07:34:53,311 [INFO] koji: Try #2 for call 253639 (listBuildroots) failed: HTTPSConnectionPool(host='koji-hub.foo.foo', port=443): Max retries exceeded with url: /kojihub?session-id=714&session-
key=2-JnRs8GeD6uM7mjKSPpu&callnum=253638 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused
',))
2021-05-27 07:35:02,570 [WARNING] koji.build: Exiting
2021-05-27 07:35:02,570 [WARNING] koji.build: Shutting down, please wait...

Weird, there is whole set of kojid options, for which max_retries + retry_interval should be more than 2/60 seconds by default. https://docs.pagure.org/koji/kojid_conf/

Very interesting by default it will be 120 retry's, but as you can see, only 2 are tried. I don't have changed any default settings about this.

Hmm, can you check value of max_retries in https://pagure.io/koji/blob/master/f/koji/init.py#_2798 ?

On the installed package(python3-koji-1.24.1-1.el8.noarch) I don't have this line.
I have added the file from the package.

init__.py

Is it 2763 in your case (if tries > max_retries)

Yes on line 2715 it is set to 30 by default.
But why are only 2 times the api call will tried?

Can't find anything suspicious - it normally works for me (trying up to max_retries). I assume, you've enabled debugging, so you can see all tracebacks around. I'm afraid that I can only encourage you to add some debugging log messages around that function (_callMethod) to see in which point it breaks.

I have found one line in the code which differs from the documentation.
The doc say offline_retry is try by default. But in the code it is false by default. (Line 2732)
After setting it in the configuration, restart kijod and then the apache on the hub, it looks better.
Now I see the Info messages about the failed command, but the daemon will live forward after it.
And on the status page of the hub, the builder is shown on-line with ongoing time stamp.

This issue has been migrated to Fedora Forge:
https://forge.fedoraproject.org/koji/koji/issues/2891

Please continue any further discussion there.

Metadata