taskotron

#112 buildmaster fails to start builds on taskotron staging
Closed: Fixed Opened by mkrizek.

Closed: Fixed
Reopen Issue 
    sqlalchemy.exc.IntegrityError: (IntegrityError) insert or update on table "buildsets" violates foreign key constraint "buildsets_sourcestampid_fkey"
            DETAIL:  Key (sourcestampsetid)=(209822) is not present in table "sourcestamps".
             'INSERT INTO buildsets (external_idstring, reason, submitted_at, complete, complete_at, results, sourcestampsetid) VALUES (%(external_idstring)s, %(reason)s, %(submitted_at)s, %(complete)s, %(complete_at)s, %(results)s, %(sourcestampsetid)s) RETURNING buildsets.id' {'complete_at': None, 'complete': 0, 'results': -1, 'reason': "The JobScheduler scheduler named 'jobsched-x86_64' triggered this build", 'external_idstring': None, 'sourcestampsetid': 209822, 'submitted_at': 1437037942.984963}

Tried restarting buildbot, didn't help.

I asked on #buildbot, let's see if I get any answer.

Looking through the logs, this all started after there was an unexplained poweroff of the virthost which is running db-qa01.qa.

I suspect that something went wrong with the sudden poweroff - either buildbot set something incorrectly when it gave up on connecting to the db or the db itself has consistency issues.

Looking into how to check consistency of the db before looking at buildbot

The constraint in question here is that the FKID for a sourcestampset row and a sourcestamps row must be valid and identical before a buildset row can be inserted. In the case of the buildmaster running on taskotron.stg, the latest id in the sourcestampsets and the sourcestamps are not equal, so when new rows of both are created, sourcestampsets.id == sourcestamps.id + 1 and viola! constraint violation

Since the rows for sourcestamps and sourcestampsets are not created in the same transaction, I think we had a case of really bad timing and the db server went down between the time where the new rows were added.

A possible but slightly qnd-hacky fix would be to create a new sourcestamps row so that the ids match up but I'm trying to see if there is a better, less hacky method for fixing this problem

I did the hacky-hacky fix on stg and it seems to have worked, new jobs are being scheduled now.

Closing under the assumption that this is fixed, reopen if that turns out not ot be the case

Thanks!

Metadata Update from @mkrizek:
- Issue tagged with: infrastructure
Metadata
None

infra

None
None
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.