koji

#3772 Scheduler part 1
Merged by tkopecek. Opened by mikem.
mikem/koji scheduler-work3  into  master

Download 3772.patch

This branch represents current work on centralized scheduling logic in koji. Posted here for discussion

Metadata Update from @mikem:
- Pull-request tagged with: discussion

The code currently works for me, but there is more to be done

rebased onto e172763e1cdd1b52c8bab80873d7307fb28f97a0

28 new commits added

  • outer joins for log query
  • basic scheduler log cli
  • use host.update_ts in kojiweb/hostinfo
  • fix ambiguous column ref
  • fakehub --exclusive option
  • fakehub --user option
  • query for scheduler logs
  • call host.refuseTask when host check fails
  • ...
  • rework host.refuseTask
  • honor refusals
  • UpsertProcessor
  • use QueryView for get_task_runs
  • more QueryView
  • ...
  • basic QueryView working
  • QueryView fragment
  • initial task refusal functions
  • ...
  • handle first time case for last_run_ts
  • scheduler check_ts fragment
  • drop unused var
  • fix convert_value refs
  • avoid upsert in db_lock
  • pull getHostData() from pr3631
  • flake8
  • delete old scheduler log messages
  • Revert "move convert_value to util lib"

1 new commit added

  • drop unused task states

Maybe worth adding constraint to db?
  • DeleteProcessor

I played wioth the idea that QueryView could construct PG VIEW and use it later. But there is probably zero gain in that and it just would obscure things.

add I/LIKE, IS, IS NOT? What about OR here - it is not possible to pass such clauses here. (IS NOT NULL). Pass clause as a callable - eventCondition with params or just the output?
Rest are probably corner-cases not worth implementing here. (e.g. ~* operator, nvr concatenation...)

I played wioth the idea that QueryView could construct PG VIEW

The naming of the class is something I've debated. It's kind of like a view, but I definitely don't want to actually construct a pg view.

It's really more of a query constructor. Granted I guess that term applies to QueryProcessor too.

There is certainly room to extend the types of conditions that the class can handle. I think a lot of that can be future work as long as the basic structure here looks ok. The most important thing is whether the overall model and api for it looks reasonable.

1 new commit added

  • fix import

rebased onto 2badf88883e677c8dcd666a54d3180e6d8ead3c1

rebased. no new commits, but skipped a few that were included in #3820 and #3819. conflicts were mainly with overlapping schema additions, the removal of get_ready_hosts, and minor variations in the commits from #3820 and #3819.

s/scheduler_task_run/scheduler_task_runs/

type='choice'

added two commits: https://pagure.io/fork/tkopecek/koji/commits/scheduler-work3-tk

4 new commits added

  • use Task interface for correct state hooks
  • use UpsertProcessor
  • fix scheduler-info
  • fix schema

5 new commits added

  • fix unit tests
  • drop unused code
  • unify getTasks and getLoadData code
  • drop some test code
  • allow admins to trigger scheduler run with an api call

1 new commit added

  • unit test

rebased onto 761711e1607b5a10133a07eee97d8c26c2bd0d91

(trivial rebase onto current master branch)

I would try some testing now. There are two commits:
https://pagure.io/fork/tkopecek/koji/c/a18b69904e2af45680e271b6df4cb6c773bac443 - schema migration
https://pagure.io/fork/tkopecek/koji/c/ea0ba521aa4b66ad3b37da3d1b43f928fdd0eeb7 - default values instead of Nones - not really needed, just my linter throws less errors in do_schedule with potential len() on None. Other options are to add asserts aka assert(isinstance(self.hosts), dict) or just ignore it.

I would try some testing now. There are two commits:
https://pagure.io/fork/tkopecek/koji/c/a18b69904e2af45680e271b6df4cb6c773bac443 - schema migration
https://pagure.io/fork/tkopecek/koji/c/ea0ba521aa4b66ad3b37da3d1b43f928fdd0eeb7 - default values instead of Nones - not really needed, just my linter throws less errors in do_schedule with potential len() on None. Other options are to add asserts aka assert(isinstance(self.hosts), dict) or just ignore it.

Metadata Update from @tkopecek:
- Pull-request untagged with: discussion
- Pull-request tagged with: testing-ready

2 new commits added

  • default values
  • schema upgrade

Missing lock creation in schem migration.

Metadata Update from @tkopecek:
- Pull-request tagged with: scheduler

few more fixes in https://pagure.io/fork/tkopecek/koji/commits/scheduler-work3-fix1

Metadata Update from @relias-redhat:
- Pull-request untagged with: testing-ready

6 new commits added

  • configurable scheduler parameters
  • flake8 fixes
  • wrong variable
  • missing lock in db
  • unify clauses handling
  • remove typo

Metadata Update from @tkopecek:
- Pull-request tagged with: testing-ready

Metadata Update from @relias-redhat:
- Pull-request tagged with: testing-done

rebased onto 5bca4905184a3d5d1bde821fb339f0e959901291

Commit b872c0dd fixes this pull-request

Pull-Request has been merged by tkopecek
Metadata
None

scheduler testing-done testing-ready

Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.