From 3f40d170c327660d3771832603e0ff775e5f880c Mon Sep 17 00:00:00 2001 From: William Moreno Reyes Date: Aug 10 2016 18:34:57 +0000 Subject: [PATCH 1/69] Add pydev project files to .gitignore Signed-off-by: William Moreno Reyes --- diff --git a/.gitignore b/.gitignore index da43781..6b91c70 100644 --- a/.gitignore +++ b/.gitignore @@ -29,3 +29,7 @@ tests/*.git # Don't track alembic.ini alembic.ini + +# Pydev specific files +.project +.pydevproject From 979c38e73cd3630e79b9ea130c897dfd447f733a Mon Sep 17 00:00:00 2001 From: William Moreno Reyes Date: Aug 10 2016 18:55:08 +0000 Subject: [PATCH 2/69] Add a screenshot of the ssh key settings Signed-off-by: William Moreno Reyes --- diff --git a/doc/usage/_static/pagure_add_ssh_key.png b/doc/usage/_static/pagure_add_ssh_key.png new file mode 100644 index 0000000..0e1ae06 Binary files /dev/null and b/doc/usage/_static/pagure_add_ssh_key.png differ From 8f1e3489cfa970cea7dbfed2c0560fbd374b6204 Mon Sep 17 00:00:00 2001 From: William Moreno Reyes Date: Aug 10 2016 18:55:49 +0000 Subject: [PATCH 3/69] Document support for multiple ssh key per user Signed-off-by: William Moreno Reyes --- diff --git a/doc/usage/first_steps.rst b/doc/usage/first_steps.rst index cdb42e4..d080673 100644 --- a/doc/usage/first_steps.rst +++ b/doc/usage/first_steps.rst @@ -48,13 +48,27 @@ These two files will be named (for example) ``id_rsa`` and ``id_rsa.pub``. The first one is the private key that must never be shared. The second is the public key that can be uploaded on pagure to give you ssh access. -To upload your public key onto pagure, login and click on the user icon on +To upload your public key onto pagure: + +1. Login into pagure and click on the user icon on the top right corner, there, select ``My settings``. .. image:: _static/pagure_my_settings.png :target: _static/pagure_my_settings.png +2. In the authentication section of your user settings copy the content of your +``id_rsa.pub`` file in the Public SSH key text box and save your ssh key settings. + +.. image:: _static/pagure_add_ssh_key.png + :target: _static/pagure_add_ssh_key.png + +.. note:: Pagure support multiple ssh keys per user, to add more than a ssh key + to your user account just add your new ssh key in your authentication + settings, this way you will be able to push commits to your repository + from a different computer. + + Configure the default email address ----------------------------------- From fbd865f645c3366aebf3ddd57c59a65e3f9ae724 Mon Sep 17 00:00:00 2001 From: William Moreno Reyes Date: Aug 10 2016 20:48:37 +0000 Subject: [PATCH 4/69] Revert add pydev files to .gitignore Signed-off-by: William Moreno Reyes --- diff --git a/.gitignore b/.gitignore index 6b91c70..cfcf6e7 100644 --- a/.gitignore +++ b/.gitignore @@ -30,6 +30,3 @@ tests/*.git # Don't track alembic.ini alembic.ini -# Pydev specific files -.project -.pydevproject diff --git a/.project b/.project new file mode 100644 index 0000000..0626a20 --- /dev/null +++ b/.project @@ -0,0 +1,17 @@ + + + pagure + + + + + + org.python.pydev.PyDevBuilder + + + + + + org.python.pydev.pythonNature + + diff --git a/.pydevproject b/.pydevproject new file mode 100644 index 0000000..0e3729e --- /dev/null +++ b/.pydevproject @@ -0,0 +1,8 @@ + + +Default +python 2.7 + +/${PROJECT_DIR_NAME} + + From 2ac45f99b8eca10fb80e7758e46b853de8b7e306 Mon Sep 17 00:00:00 2001 From: William Moreno Reyes Date: Aug 10 2016 20:53:37 +0000 Subject: [PATCH 5/69] Revert add pydev files to .gitignore This reverts commit fbd865f645c3366aebf3ddd57c59a65e3f9ae724. --- diff --git a/.gitignore b/.gitignore index cfcf6e7..6b91c70 100644 --- a/.gitignore +++ b/.gitignore @@ -30,3 +30,6 @@ tests/*.git # Don't track alembic.ini alembic.ini +# Pydev specific files +.project +.pydevproject diff --git a/.project b/.project deleted file mode 100644 index 0626a20..0000000 --- a/.project +++ /dev/null @@ -1,17 +0,0 @@ - - - pagure - - - - - - org.python.pydev.PyDevBuilder - - - - - - org.python.pydev.pythonNature - - diff --git a/.pydevproject b/.pydevproject deleted file mode 100644 index 0e3729e..0000000 --- a/.pydevproject +++ /dev/null @@ -1,8 +0,0 @@ - - -Default -python 2.7 - -/${PROJECT_DIR_NAME} - - From 311888a03585fc1a26f894928318f858f1f80fc0 Mon Sep 17 00:00:00 2001 From: William Moreno Reyes Date: Aug 10 2016 20:55:00 +0000 Subject: [PATCH 6/69] Revert "Add pydev project files to .gitignore" This reverts commit 3f40d170c327660d3771832603e0ff775e5f880c. --- diff --git a/.gitignore b/.gitignore index 6b91c70..da43781 100644 --- a/.gitignore +++ b/.gitignore @@ -29,7 +29,3 @@ tests/*.git # Don't track alembic.ini alembic.ini - -# Pydev specific files -.project -.pydevproject From a7aca9bc00f5262f05a4c5018bfbbfbc96664d2f Mon Sep 17 00:00:00 2001 From: William Moreno Reyes Date: Aug 10 2016 21:01:38 +0000 Subject: [PATCH 7/69] Improve multiple keys per user settings Signed-off-by: William Moreno Reyes --- diff --git a/doc/usage/first_steps.rst b/doc/usage/first_steps.rst index d080673..7c2b453 100644 --- a/doc/usage/first_steps.rst +++ b/doc/usage/first_steps.rst @@ -65,7 +65,7 @@ the top right corner, there, select ``My settings``. .. note:: Pagure support multiple ssh keys per user, to add more than a ssh key to your user account just add your new ssh key in your authentication - settings, this way you will be able to push commits to your repository + settings (one key per row), this way you will be able to push commits to your repository from a different computer. From a4d69f822826eb715eb6269694a5e7c8b1d78fca Mon Sep 17 00:00:00 2001 From: Pierre-Yves Chibon Date: Aug 10 2016 21:05:03 +0000 Subject: [PATCH 8/69] Replace the jenkins hook by a more generic CI one --- diff --git a/pagure/hooks/jenkins_hook.py b/pagure/hooks/jenkins_hook.py deleted file mode 100644 index ad2521c..0000000 --- a/pagure/hooks/jenkins_hook.py +++ /dev/null @@ -1,140 +0,0 @@ -# -*- coding: utf-8 -*- - - -import os -import uuid - -import sqlalchemy as sa -import pygit2 - -from wtforms import validators, TextField, BooleanField -from flask.ext import wtf -from sqlalchemy.orm import relation, backref -from sqlalchemy.ext.declarative import declarative_base - -from pagure.hooks import BaseHook, RequiredIf -from pagure.lib.model import BASE, Project -from pagure import get_repo_path -from pagure import APP, SESSION - - -class PagureCI(BASE): - - __tablename__ = 'hook_pagure_ci' - - id = sa.Column(sa.Integer, primary_key=True) - project_id = sa.Column( - sa.Integer, - sa.ForeignKey('projects.id', onupdate='CASCADE',ondelete='CASCADE'), - nullable=False, - unique=False, - index=True) - pagure_ci_token = sa.Column(sa.String(32), nullable=True, unique=True, - index=True) - - active = sa.Column(sa.Boolean, nullable=False, default=False) - display_name = sa.Column(sa.String(64), nullable=False, default='Jenkins') - pagure_name = sa.Column(sa.String(255)) - - jenkins_name = sa.Column(sa.String(255)) - jenkins_url = sa.Column(sa.String(255), nullable=False, - default='http://jenkins.fedorainfracloud.org/') - jenkins_token = sa.Column(sa.String(64)) - - project = relation( - 'Project', - foreign_keys=[project_id], - remote_side=[Project.id], - backref=backref( - 'hook_pagure_ci', cascade="delete, delete-orphan", - single_parent=True) - ) - - def __init__(self): - self.pagure_ci_token = uuid.uuid4().hex - - -class ConfigNotFound(Exception): - pass - - -class Service(object): - PAGURE = PagureCI.pagure_name - JENKINS = PagureCI.jenkins_name - - -def get_configs(project_name, service): - """Returns all configurations with given name on a service. - - :raises ConfigNotFound: when no configuration matches - """ - cfg = SESSION.query(PagureCI).filter( - service == project_name).all() - if not cfg: - raise ConfigNotFound(project_name) - return cfg - - -class JenkinsForm(wtf.Form): - - '''Form to configure Jenkins hook''' - - pagure_name = TextField('Name of project in Pagure', - [validators.Required(), - validators.Length(max=255)]) - - jenkins_name = TextField('Name of project in Jenkins', - [validators.Required(), - validators.Length(max=255)]) - - jenkins_url = TextField('Jenkins URL', - [validators.Required(), - validators.Length(max=255)], - default='http://jenkins.fedorainfracloud.org/') - - jenkins_token = TextField('Jenkins token', - [validators.Required()]) - - active = BooleanField('Active', [validators.Optional()]) - - -class PagureCiHook(BaseHook): - ''' Jenkins hooks. ''' - - name = 'Pagure CI' - description = 'This hook help to set up CI for the project'\ - ' the changes made by the pushes to the git repository.' - form = JenkinsForm - db_object = PagureCI - backref = 'hook_pagure_ci' - form_fields = [ - 'pagure_name', 'jenkins_name', - 'jenkins_url', 'jenkins_token', 'active' - ] - - @classmethod - def set_up(cls, project): - ''' Install the generic post-receive hook that allow us to call - multiple post-receive hooks as set per plugin. - ''' - pass - - @classmethod - def install(cls, project, dbobj): - ''' Method called to install the hook for a project. - - :arg project: a ``pagure.model.Project`` object to which the hook - should be installed - - ''' - pass - - @classmethod - def remove(cls, project): - ''' Method called to remove the hook of a project. - - :arg project: a ``pagure.model.Project`` object to which the hook - should be installed - - ''' - pass diff --git a/pagure/hooks/pagure_ci.py b/pagure/hooks/pagure_ci.py new file mode 100644 index 0000000..247e66b --- /dev/null +++ b/pagure/hooks/pagure_ci.py @@ -0,0 +1,134 @@ +# -*- coding: utf-8 -*- + +""" + (c) 2016 - Copyright Red Hat Inc + + Authors: + Pierre-Yves Chibon + +""" + +import os + +import sqlalchemy as sa +import wtforms +from flask.ext import wtf +from sqlalchemy.orm import relation +from sqlalchemy.orm import backref + +from pagure.hooks import BaseHook, RequiredIf +from pagure.lib.model import BASE, Project, TypeCi +from pagure import get_repo_path, SESSION + + +class PagureCITable(BASE): + """ Stores information about the CI linked to on a project. + + Table -- hook_pagure_ci + """ + + __tablename__ = 'hook_pagure_ci' + + id = sa.Column(sa.Integer, primary_key=True) + project_id = sa.Column( + sa.Integer, + sa.ForeignKey( + 'projects.id', onupdate='CASCADE', ondelete='CASCADE'), + nullable=False, + unique=True, + index=True) + pagure_ci_token = sa.Column( + sa.String(32), + nullable=True, + unique=True, + index=True) + type_id = sa.Column( + sa.Integer, + sa.ForeignKey( + 'type_ci.id', onupdate='CASCADE', ondelete='CASCADE'), + nullable=False, + unique=True) + active = sa.Column(sa.Boolean, nullable=False, default=False) + + project = relation( + 'Project', remote_side=[Project.id], + backref=backref( + 'ci_hook', cascade="delete, delete-orphan", + single_parent=True) + ) + + type_ci = relation( + 'TypeCi', remote_side=[TypeCi.id], + ) + + +class PagureCiForm(wtf.Form): + ''' Form to configure the CI hook. ''' + type_ci = wtforms.SelectField( + 'Type of CI service', + [RequiredIf('active')], + choices=[] + ) + jenkins_url = wtforms.TextField( + 'URL to the project on the CI service', + [RequiredIf('active'), wtforms.validators.Length(max=255)], + ) + jenkins_token = wtforms.TextField( + 'CI token', + [RequiredIf('active')], + ) + active = wtforms.BooleanField( + 'Active', + [wtforms.validators.Optional()] + ) + + def __init__(self, *args, **kwargs): + """ Calls the default constructor with the normal argument but + uses the list of collection provided to fill the choices of the + drop-down list. + """ + super(PagureCiForm, self).__init__(*args, **kwargs) + + types = SESSION.query(TypeCi).order_by(TypeCi.type).all() + self.type_ci.choices = [ + (ci_type.type, ci_type.type) for ci_type in types + ] + + +class PagureCi(BaseHook): + ''' Mail hooks. ''' + + name = 'Pagure CI' + description = 'Generate notification emails for pushes to a git repository. '\ + 'This hook sends emails describing changes introduced by pushes to a git repository.' + form = PagureCiForm + db_object = PagureCITable + backref = 'ci_hook' + form_fields = ['type_ci', 'jenkins_url', 'jenkins_token', 'active'] + + @classmethod + def set_up(cls, project): + ''' Install the generic post-receive hook that allow us to call + multiple post-receive hooks as set per plugin. + ''' + pass + + @classmethod + def install(cls, project, dbobj): + ''' Method called to install the hook for a project. + + :arg project: a ``pagure.model.Project`` object to which the hook + should be installed + + ''' + pass + + @classmethod + def remove(cls, project): + ''' Method called to remove the hook of a project. + + :arg project: a ``pagure.model.Project`` object to which the hook + should be installed + + ''' + pass From a0828745f59dbf44b63f6acf530af3b671152306 Mon Sep 17 00:00:00 2001 From: Pierre-Yves Chibon Date: Aug 10 2016 21:05:03 +0000 Subject: [PATCH 9/69] Make the list of CI service supported a configuration option This would allow admins to easily add/remove support for some CI service. --- diff --git a/pagure/default_config.py b/pagure/default_config.py index e171b19..4a7ffae 100644 --- a/pagure/default_config.py +++ b/pagure/default_config.py @@ -221,3 +221,6 @@ ACLS = { # Bootstrap URLS BOOTSTRAP_URLS_CSS = 'https://apps.fedoraproject.org/global/fedora-bootstrap-1.0.1/fedora-bootstrap.css' BOOTSTRAP_URLS_JS = 'https://apps.fedoraproject.org/global/fedora-bootstrap-1.0.1/fedora-bootstrap.js' + +# List of the type of CI service supported by this pagure instance +PAGURE_CI_SERVICES = ['jenkins'] diff --git a/pagure/hooks/pagure_ci.py b/pagure/hooks/pagure_ci.py index 247e66b..dba55c7 100644 --- a/pagure/hooks/pagure_ci.py +++ b/pagure/hooks/pagure_ci.py @@ -18,7 +18,7 @@ from sqlalchemy.orm import backref from pagure.hooks import BaseHook, RequiredIf from pagure.lib.model import BASE, Project, TypeCi -from pagure import get_repo_path, SESSION +from pagure import get_repo_path, SESSION, APP class PagureCITable(BASE): @@ -89,9 +89,9 @@ class PagureCiForm(wtf.Form): """ super(PagureCiForm, self).__init__(*args, **kwargs) - types = SESSION.query(TypeCi).order_by(TypeCi.type).all() + types = APP.config.get('PAGURE_CI_SERVICES', []) self.type_ci.choices = [ - (ci_type.type, ci_type.type) for ci_type in types + (ci_type, ci_type) for ci_type in types ] From e33a2e123c8dac7c5b8dd3a89b8365fd928652d2 Mon Sep 17 00:00:00 2001 From: Pierre-Yves Chibon Date: Aug 10 2016 21:05:03 +0000 Subject: [PATCH 10/69] Drop all the pagure_c specific code, we want to be generic in this code --- diff --git a/pagure/templates/plugin.html b/pagure/templates/plugin.html index ab48797..3cbeb75 100644 --- a/pagure/templates/plugin.html +++ b/pagure/templates/plugin.html @@ -17,16 +17,6 @@ ) }}" method="post"> {{ plugin.description | markdown | noJS | safe }} - {% if pagure_ci_token and (plugin.name == 'Pagure CI') %} -
- - -
- {% endif %} - {% for field in fields %} {{ render_field_in_row(field) }} diff --git a/pagure/ui/plugins.py b/pagure/ui/plugins.py index d7b0e56..d0eda4d 100644 --- a/pagure/ui/plugins.py +++ b/pagure/ui/plugins.py @@ -20,14 +20,8 @@ import pagure.forms from pagure import APP, SESSION, login_required, is_repo_admin from pagure.lib.model import BASE from pagure.exceptions import FileNotFoundException -from pagure.hooks import jenkins_hook from pagure.lib import model -try: - from pagure.lib import pagure_ci -except ImportError: - pagure_ci = None - import json from kitchen.text.converters import to_bytes from cryptography.hazmat.primitives import constant_time @@ -43,9 +37,6 @@ def get_plugin_names(blacklist=None): elif not isinstance(blacklist, list): blacklist = [blacklist] - if pagure_ci is None and 'Pagure CI' not in blacklist: - blacklist.append('Pagure CI') - output = [ plugin.name for plugin in plugins @@ -101,7 +92,6 @@ def view_plugin(repo, plugin, username=None, full=True): fields = [] new = True dbobj = plugin.db_object() - pagure_ci_token = None if hasattr(repo, plugin.backref): dbobj = getattr(repo, plugin.backref) @@ -110,9 +100,6 @@ def view_plugin(repo, plugin, username=None, full=True): if dbobj and len(dbobj) > 0: dbobj = dbobj[0] new = False - # To populate the pagure CI token if generated - if hasattr(dbobj, "pagure_ci_token") and plugin.backref == "hook_pagure_ci": - pagure_ci_token = dbobj.pagure_ci_token else: dbobj = plugin.db_object() @@ -144,7 +131,6 @@ def view_plugin(repo, plugin, username=None, full=True): username=username, plugin=plugin, form=form, - pagure_ci_token=pagure_ci_token, fields=fields) if form.active.data: @@ -178,29 +164,4 @@ def view_plugin(repo, plugin, username=None, full=True): username=username, plugin=plugin, form=form, - pagure_ci_token=pagure_ci_token, fields=fields) - -if pagure_ci is not None: - @APP.route('/hooks//build-finished', methods=['POST']) - def hook_finished(pagure_ci_token): - """ Flags the Pull-request after getting notification from Jenkins - """ - - try: - data = json.loads(flask.request.get_data()) - cfg = jenkins_hook.get_configs( - data['name'], jenkins_hook.Service.JENKINS)[0] - build_id = data['build']['number'] - - if not constant_time.bytes_eq( - to_bytes(pagure_ci_token), to_bytes(cfg.pagure_ci_token)): - return ('Token mismatch', 401) - - except (TypeError, ValueError, KeyError, jenkins_hook.ConfigNotFound) as exc: - APP.logger.error('Error processing jenkins notification', exc_info=exc) - flask.abort(400, "Bad Request") - - APP.logger.info('Received jenkins notification') - pagure_ci.process_build(APP.logger, cfg, build_id) - return ('', 204) From 585fe08f5bd1fc765f6cfe232aabe19afd357bfc Mon Sep 17 00:00:00 2001 From: Pierre-Yves Chibon Date: Aug 10 2016 21:05:03 +0000 Subject: [PATCH 11/69] Move the CI specific API endpoints to their own section of the API This should help us expand support for CI services --- diff --git a/pagure/api/__init__.py b/pagure/api/__init__.py index 7f7d838..1c97c29 100644 --- a/pagure/api/__init__.py +++ b/pagure/api/__init__.py @@ -202,6 +202,9 @@ from pagure.api import fork from pagure.api import project from pagure.api import user +if pagure.APP.config.get('PAGURE_CI', False): + from pagure.api.ci import jenkins + @API.route('/version/') @API.route('/version') diff --git a/pagure/api/ci/__init__.py b/pagure/api/ci/__init__.py new file mode 100644 index 0000000..ccbb2cb --- /dev/null +++ b/pagure/api/ci/__init__.py @@ -0,0 +1,9 @@ +# -*- coding: utf-8 -*- + +""" + (c) 2016 - Copyright Red Hat Inc + + Authors: + Pierre-Yves Chibon + +""" diff --git a/pagure/api/ci/jenkins.py b/pagure/api/ci/jenkins.py new file mode 100644 index 0000000..ab05532 --- /dev/null +++ b/pagure/api/ci/jenkins.py @@ -0,0 +1,43 @@ +# -*- coding: utf-8 -*- + +""" + (c) 2016 - Copyright Red Hat Inc + + Authors: + Pierre-Yves Chibon + +""" + +import flask + +from sqlalchemy.exc import SQLAlchemyError + +import pagure +import pagure.exceptions +import pagure.lib +from pagure import APP, SESSION +from pagure.api import API, APIERROR + + +@API.route('/ci/jenkins//build-finished', methods=['POST']) +def ci_notification(pagure_ci_token): + """ Flag a pull-request based on the info provided by the CI service. + """ + + try: + data = flask.request.json() + cfg = jenkins_hook.get_configs( + data['name'], jenkins_hook.Service.JENKINS)[0] + build_id = data['build']['number'] + + if not constant_time.bytes_eq( + to_bytes(pagure_ci_token), to_bytes(cfg.pagure_ci_token)): + return ('Token mismatch', 401) + + except (TypeError, ValueError, KeyError, jenkins_hook.ConfigNotFound) as exc: + APP.logger.error('Error processing jenkins notification', exc_info=exc) + flask.abort(400, "Bad Request") + + APP.logger.info('Received jenkins notification') + pagure_ci.process_build(APP.logger, cfg, build_id) + return ('', 204) From 47ed2bc87644fff9dcab31aa89f40f32355b90ce Mon Sep 17 00:00:00 2001 From: Pierre-Yves Chibon Date: Aug 10 2016 21:05:03 +0000 Subject: [PATCH 12/69] Rework the API endpoint receiving the notifications from jenkins --- diff --git a/pagure/api/ci/jenkins.py b/pagure/api/ci/jenkins.py index ab05532..2d01423 100644 --- a/pagure/api/ci/jenkins.py +++ b/pagure/api/ci/jenkins.py @@ -15,29 +15,33 @@ from sqlalchemy.exc import SQLAlchemyError import pagure import pagure.exceptions import pagure.lib +import pagure.lib.lib_ci as lib_ci from pagure import APP, SESSION from pagure.api import API, APIERROR @API.route('/ci/jenkins//build-finished', methods=['POST']) -def ci_notification(pagure_ci_token): +def jenkins_ci_notification(pagure_ci_token): """ Flag a pull-request based on the info provided by the CI service. """ - try: - data = flask.request.json() - cfg = jenkins_hook.get_configs( - data['name'], jenkins_hook.Service.JENKINS)[0] - build_id = data['build']['number'] + project = lib_ci.get_project_by_ci_token(SESSION, pagure_ci_token) + if not project: + flask.abort(404, 'No project corresponding to this CI token') + + data = flask.request.json() + if not data: + flask.abort(400, "Bad Request: No JSON retrived") - if not constant_time.bytes_eq( - to_bytes(pagure_ci_token), to_bytes(cfg.pagure_ci_token)): - return ('Token mismatch', 401) + build_id = data.get('build', {}).get('number') + if not build_it: + flask.abort(400, "Bad Request: No build ID retrived") - except (TypeError, ValueError, KeyError, jenkins_hook.ConfigNotFound) as exc: - APP.logger.error('Error processing jenkins notification', exc_info=exc) - flask.abort(400, "Bad Request") + try: + lib_ci.process_jenkins_build(project, build_id) + except pagure.exceptions.PagureException as err: + APP.logger.error('Error processing jenkins notification', exc_info=err) + flask.abort(400, "Bad Request: %s" % err) - APP.logger.info('Received jenkins notification') - pagure_ci.process_build(APP.logger, cfg, build_id) + APP.logger.info('Successfully proccessed jenkins notification') return ('', 204) From 5333b905d6f8526b8d33e2220c5530a4387f647d Mon Sep 17 00:00:00 2001 From: Pierre-Yves Chibon Date: Aug 10 2016 21:05:03 +0000 Subject: [PATCH 13/69] Rework the pagure CI plugin Adjust the name of the fields, adjust how it is installed and removed add a template to be displayed in the settings page --- diff --git a/pagure/hooks/pagure_ci.py b/pagure/hooks/pagure_ci.py index dba55c7..ceb0d6a 100644 --- a/pagure/hooks/pagure_ci.py +++ b/pagure/hooks/pagure_ci.py @@ -16,8 +16,9 @@ from flask.ext import wtf from sqlalchemy.orm import relation from sqlalchemy.orm import backref +import pagure.lib from pagure.hooks import BaseHook, RequiredIf -from pagure.lib.model import BASE, Project, TypeCi +from pagure.lib.model import BASE, Project from pagure import get_repo_path, SESSION, APP @@ -40,14 +41,14 @@ class PagureCITable(BASE): pagure_ci_token = sa.Column( sa.String(32), nullable=True, - unique=True, index=True) - type_id = sa.Column( - sa.Integer, - sa.ForeignKey( - 'type_ci.id', onupdate='CASCADE', ondelete='CASCADE'), - nullable=False, - unique=True) + ci_type = sa.Column( + sa.String(255), + nullable=True) + ci_url = sa.Column( + sa.String(255), + nullable=True, + unique=False) active = sa.Column(sa.Boolean, nullable=False, default=False) project = relation( @@ -57,26 +58,37 @@ class PagureCITable(BASE): single_parent=True) ) - type_ci = relation( - 'TypeCi', remote_side=[TypeCi.id], - ) + +tmpl = """ +{% if repo | hasattr('ci_hook') and repo.ci_hook and repo.ci_hook[0].pagure_ci_token %} +The URL to be used to POST the results of your build +is: + +
+{{ (config['APP_URL'][:-1] if config['APP_URL'].endswith('/')
+  else config['APP_URL'])
+  + url_for('api_ns.%s_ci_notification' % repo.ci_hook[0].ci_type,
+    pagure_ci_token=repo.ci_hook[0].pagure_ci_token) }}
+
+ +{% else %} +Once this plugin has been activated, reload this tab or this page to access +the URL to which your CI service should send its info. +{% endif %} +""" class PagureCiForm(wtf.Form): ''' Form to configure the CI hook. ''' - type_ci = wtforms.SelectField( + ci_type = wtforms.SelectField( 'Type of CI service', [RequiredIf('active')], choices=[] ) - jenkins_url = wtforms.TextField( + ci_url = wtforms.TextField( 'URL to the project on the CI service', [RequiredIf('active'), wtforms.validators.Length(max=255)], ) - jenkins_token = wtforms.TextField( - 'CI token', - [RequiredIf('active')], - ) active = wtforms.BooleanField( 'Active', [wtforms.validators.Optional()] @@ -90,7 +102,7 @@ class PagureCiForm(wtf.Form): super(PagureCiForm, self).__init__(*args, **kwargs) types = APP.config.get('PAGURE_CI_SERVICES', []) - self.type_ci.choices = [ + self.ci_type.choices = [ (ci_type, ci_type) for ci_type in types ] @@ -99,19 +111,25 @@ class PagureCi(BaseHook): ''' Mail hooks. ''' name = 'Pagure CI' - description = 'Generate notification emails for pushes to a git repository. '\ - 'This hook sends emails describing changes introduced by pushes to a git repository.' + description = 'Integrate continuous integration (CI) services into your '\ + 'pagure project, providing you notifications for every pull-request '\ + 'opened in the project.' + extra_info = tmpl form = PagureCiForm db_object = PagureCITable backref = 'ci_hook' - form_fields = ['type_ci', 'jenkins_url', 'jenkins_token', 'active'] + form_fields = ['ci_type', 'ci_url', 'active'] @classmethod def set_up(cls, project): ''' Install the generic post-receive hook that allow us to call multiple post-receive hooks as set per plugin. ''' - pass + hook = project.ci_hook[0] + if not hook.pagure_ci_token: + hook.pagure_ci_token = pagure.lib.login.id_generator(32) + SESSION.add(project) + SESSION.commit() @classmethod def install(cls, project, dbobj): @@ -131,4 +149,7 @@ class PagureCi(BaseHook): should be installed ''' - pass + for hook in project.ci_hook: + hook.pagure_ci_token = None + SESSION.add(project) + SESSION.commit() From a9347ad793f8f7113bbe402d7340f409cbd35175 Mon Sep 17 00:00:00 2001 From: Pierre-Yves Chibon Date: Aug 10 2016 21:05:03 +0000 Subject: [PATCH 14/69] Add a jinja filter allowing to render a given piece of template --- diff --git a/pagure/ui/filters.py b/pagure/ui/filters.py index a7a562d..ff5113f 100644 --- a/pagure/ui/filters.py +++ b/pagure/ui/filters.py @@ -36,6 +36,13 @@ def jinja_hasattr(obj, string): return hasattr(obj, string) +@APP.template_filter('render') +def jinja_render(tmpl, **kwargs): + """ Render the tiven template with the provided arguments + """ + return flask.render_template_string(tmpl, **kwargs) + + @APP.template_filter('humanize') def humanize_date(date): """ Template filter returning the last commit date of the provided repo. From b53e07a1b9e78c7f92f41944460528fc158254e0 Mon Sep 17 00:00:00 2001 From: Pierre-Yves Chibon Date: Aug 10 2016 21:05:03 +0000 Subject: [PATCH 15/69] Render the plugin's extra info if there is one --- diff --git a/pagure/templates/plugin.html b/pagure/templates/plugin.html index 3cbeb75..ed5c240 100644 --- a/pagure/templates/plugin.html +++ b/pagure/templates/plugin.html @@ -22,6 +22,10 @@ {{ render_field_in_row(field) }} {% endfor %}
+ + {% if plugin.extra_info %} + {{ plugin.extra_info | render(repo=repo) | safe }} + {% endif %}

From bf62ec1f591f517bb70e7f493f8446d0ec29335b Mon Sep 17 00:00:00 2001 From: Pierre-Yves Chibon Date: Aug 10 2016 21:05:04 +0000 Subject: [PATCH 16/69] Document in the code what the hook_token field of a project is for --- diff --git a/pagure/lib/model.py b/pagure/lib/model.py index 8766948..68085e6 100644 --- a/pagure/lib/model.py +++ b/pagure/lib/model.py @@ -299,6 +299,7 @@ class Project(BASE): description = sa.Column(sa.Text, nullable=True) url = sa.Column(sa.Text, nullable=True) _settings = sa.Column(sa.Text, nullable=True) + # The hook_token is used to sign the notification sent via web-hook hook_token = sa.Column(sa.String(40), nullable=False, unique=True) avatar_email = sa.Column(sa.Text, nullable=True) parent_id = sa.Column( From 0145a07acd780e623d2371efef2a268561b0ef24 Mon Sep 17 00:00:00 2001 From: Pierre-Yves Chibon Date: Aug 10 2016 21:05:04 +0000 Subject: [PATCH 17/69] Add the lib_ci module This lib is meant to be used by the api.ci modules when they want to flag a pull-request with some info. --- diff --git a/pagure/lib/lib_ci.py b/pagure/lib/lib_ci.py new file mode 100644 index 0000000..9853811 --- /dev/null +++ b/pagure/lib/lib_ci.py @@ -0,0 +1,82 @@ +# -*- coding: utf-8 -*- +import json +import logging + +import requests + +from sqlalchemy.orm import scoped_session, sessionmaker +from sqlalchemy.exc import SQLAlchemyError +from sqlalchemy import create_engine + +import pagure.exceptions +import pagure.lib +from pagure.lib import model +from pagure.hooks import pagure_ci + + +BUILD_STATS = { + 'SUCCESS': ('Build successful', 100), + 'FAILURE': ('Build failed', 0), +} + + +def get_project_by_ci_token(session, ci_token): + """ Return the project corresponding to the provided ci_token. """ + query = session.query( + model.Project + ).filter( + model.Project.id == pagure_ci.PagureCITable.project_id + ).filter( + pagure_ci.PagureCITable.pagure_ci_token == ci_token + ) + + return query.first() + + +def process_jenkins_build(project, build_id): + """ Gets the build info from jenkins and flags that particular + pull-request. + """ + import jenkins + jenk = jenkins.Jenkins(project.ci_hook.url) + jenkins_name = project.ci_hook.url.split('/job/', 1)[1].split('/', 1)[0] + build_info = jenk.get_build_info(jenkins_name, build_id) + result = build_info['result'] + url = build_info['url'] + + pr_id = None + for action in build_info['actions']: + for cause in action.get('causes', []): + try: + pr_id = int(cause['note']) + except (KeyError, ValueError): + continue + + if not pr_id: + raise pagure.exceptions.PagureException( + 'No PR found corresponding') + + if result not in BUILD_STATS: + pagure.exceptions.PagureException( + 'Unknown build status: %s' % result) + + request = pagure.lib.search_pull_requests( + session, project_id=project.id, requestid=requestid) + + if not request: + raise pagure.exceptions.PagureException('Request not found') + + comment, percent = BUILD_STATS[result] + + message = pagure.lib.add_pull_request_flag( + session, + request=request, + username=username, + percent=percent, + comment=comment, + url=url, + uid=None, + user=repo.user.username, + requestfolder=APP.config['REQUESTS_FOLDER'], + ) + SESSION.commit() From cd8c70fa801060b4c5621ba69add93e24e850430 Mon Sep 17 00:00:00 2001 From: Pierre-Yves Chibon Date: Aug 10 2016 21:05:04 +0000 Subject: [PATCH 18/69] Remove the old pagure_ci lib that is no longer used --- diff --git a/pagure/lib/pagure_ci.py b/pagure/lib/pagure_ci.py deleted file mode 100644 index 88ddcd7..0000000 --- a/pagure/lib/pagure_ci.py +++ /dev/null @@ -1,128 +0,0 @@ -# -*- coding: utf-8 -*- -import os -import flask -from sqlalchemy.orm import scoped_session, sessionmaker -from sqlalchemy.exc import SQLAlchemyError -from sqlalchemy import create_engine - -from pagure.hooks import jenkins_hook -import pagure.lib -from pagure.lib import model -from pagure import APP, SESSION -import pagure.exceptions - -import json -import logging - -import requests -import jenkins - -APP.logger.setLevel(logging.INFO) - -PAGURE_URL = '{base}api/0/{repo}/pull-request/{pr}/flag' -JENKINS_TRIGGER_URL = '{base}job/{project}/buildWithParameters' - - -def process_pr(logger, cfg, pr_id, repo, branch): - ''' Process the pull-request, it POST the - data to the Jenkins URL to trigger build. - ''' - if cfg.active: - post_data(logger, - JENKINS_TRIGGER_URL.format( - base=cfg.jenkins_url, project=cfg.jenkins_name), - {'token': cfg.jenkins_token, - 'cause': pr_id, - 'REPO': repo, - 'BRANCH': branch}) - else: - raise pagure.exceptions.HookInactiveException(cfg.pagure_name) - - -def process_build(logger, cfg, build_id): - ''' Gets the build info from jenkins - and flags that particular pull-request. - ''' - if cfg.active: - jenk = jenkins.Jenkins(cfg.jenkins_url) - build_info = jenk.get_build_info(cfg.jenkins_name, build_id) - result = build_info['result'] - url = build_info['url'] - - pr_id = None - - for action in build_info['actions']: - for cause in action.get('causes', []): - try: - pr_id = int(cause['note']) - except (KeyError, ValueError): - continue - - if not pr_id: - logger.info('Not a PR check') - return - - # Comment in Pagure - logger.info('Updating %s PR %d: %s', cfg.pagure_name, pr_id, result) - try: - pagure_ci_flag(logger, - username=cfg.display_name, - repo=cfg.pagure_name, - requestid=pr_id, - result=result, - url=url) - - except KeyError as exc: - logger.warning('Unknown build status', exc_info=exc) - else: - raise pagure.exceptions.HookInactiveException(cfg.pagure_name) - - -def post_data(logger, *args, **kwargs): - ''' POST data to jenkins and reports a response. ''' - resp = requests.post(*args, **kwargs) - logger.debug('Received response status %s', resp.status_code) - if resp.status_code < 200 or resp.status_code >= 300: - logger.error('Network request failed: %d: %s', - resp.status_code, resp.text) - - -def pagure_ci_flag(logger, repo, username, url, result, requestid): - ''' Flag the pull-request in pagure. ''' - - comment, percent = { - 'SUCCESS': ('Build successful', 100), - 'FAILURE': ('Build failed', 0), - }[result] - - repo = pagure.lib.get_project(SESSION, repo, user=None) - output = {} - - if repo is None: - raise pagure.exceptions.FileNotFoundException('Repo not found') - - request = pagure.lib.search_pull_requests( - SESSION, project_id=repo.id, requestid=requestid) - - if not request: - raise pagure.exceptions.FileNotFoundException('Request not found') - - try: - message = pagure.lib.add_pull_request_flag( - SESSION, - request=request, - username=username, - percent=percent, - comment=comment, - url=url, - uid=None, - user=repo.user.username, - requestfolder=APP.config['REQUESTS_FOLDER'], - ) - SESSION.commit() - logger.debug('Received response status: %s', message) - output['message'] = message - - except SQLAlchemyError as err: # pragma: no cover - logger.exception(err) - SESSION.rollback() From 0c3540b321cca0a1e7ca3b53e4030f1e91a906f5 Mon Sep 17 00:00:00 2001 From: Pierre-Yves Chibon Date: Aug 10 2016 21:05:04 +0000 Subject: [PATCH 19/69] Add the logic allowing to set the different types of CI services supported If None are provided, pagure is considered as if it had CI disabled. --- diff --git a/pagure/lib/__init__.py b/pagure/lib/__init__.py index b092f0a..293f3c8 100644 --- a/pagure/lib/__init__.py +++ b/pagure/lib/__init__.py @@ -48,7 +48,7 @@ from pagure.lib import model REDIS = None - +PAGURE_CI = None def set_redis(host, port, db): """ Set the redis connection with the specified information. """ @@ -56,6 +56,10 @@ def set_redis(host, port, db): pool = redis.ConnectionPool(host=host, port=port, db=db) REDIS = redis.StrictRedis(connection_pool=pool) +def set_pagure_ci(services): + """ Set the list of CI services supported by this pagure instance. """ + PAGURE_CI = services + def __get_user(session, key): """ Searches for a user in the database for a given username or email. From 6fec8ce86b8201bec4c38a99840b42ce90707d85 Mon Sep 17 00:00:00 2001 From: Pierre-Yves Chibon Date: Aug 10 2016 21:05:04 +0000 Subject: [PATCH 20/69] First work on the pagure-ci server/service The idea is that this service will listen for messages coming from pagure when a CI build should be triggered and will actually trigger that build. --- diff --git a/pagureCI/pagure_ci_server.py b/pagureCI/pagure_ci_server.py new file mode 100644 index 0000000..a951646 --- /dev/null +++ b/pagureCI/pagure_ci_server.py @@ -0,0 +1,150 @@ +#!/usr/bin/env python +# -*- coding: utf-8 -*- + +""" + (c) 2016 - Copyright Red Hat Inc + + Authors: + Pierre-Yves Chibon + + +This server listens to message sent via redis and send the corresponding +web-hook request. + +Using this mechanism, we no longer block the main application if the +receiving end is offline or so. + +""" + +import datetime +import hashlib +import hmac +import json +import logging +import os +import requests +import time +import uuid + +import six +import trollius +import trollius_redis + +from kitchen.text.converters import to_bytes + + +log = logging.getLogger(__name__) + +if 'PAGURE_CONFIG' not in os.environ \ + and os.path.exists('/etc/pagure/pagure.cfg'): + print 'Using configuration file `/etc/pagure/pagure.cfg`' + os.environ['PAGURE_CONFIG'] = '/etc/pagure/pagure.cfg' + + +import pagure +import pagure.lib +from pagure.exceptions import PagureEvException + +_i = 0 + + +@trollius.coroutine +def handle_messages(): + connection = yield trollius.From(trollius_redis.Connection.create( + host='0.0.0.0', port=6379, db=0)) + + # Create subscriber. + subscriber = yield trollius.From(connection.start_subscribe()) + + # Subscribe to channel. + yield trollius.From(subscriber.subscribe(['pagure.ci'])) + + # Inside a while loop, wait for incoming events. + while True: + reply = yield trollius.From(subscriber.next_published()) + log.info( + 'Received: %s on channel: %s', + repr(reply.value), reply.channel) + data = json.loads(reply.value) + + pr_id = data['pr']['id'] + project = data['pr']['project']['name'] + branch = data['pr']['branch_from'] + + username = None + projectname = data['pr']['project'] + if data['pr'].get('parent'): + username, data['pr']['project']['user']['user'] + + project = pagure.lib.get_project( + session=pagure.SESSION, name=projectname, user=username) + + if not project: + log.warning( + 'No project could be found from the message %s' % data) + continue + + repo = data['pr'].get('remote_git') + if not repo: + base = pagure.APP.config['APP_URL'] + if base.endswith('/'): + base[:-1] + base += '/%s' % project.path + + log.info("Trigger on %s PR #%s from %s: %s", + project.fullname, pr_id, repo, branch) + + url = project.ci_hook.ci_url + if url.endswith('/'): + url = url[:-1] + + if data['ci_type'] == 'jenkins': + url += '/buildWithParameters' + request.post( + url, + data={ + 'token': cfg.jenkins_token, + 'cause': pr_id, + 'REPO': repo.fullname, + 'BRANCH': branch + } + ) + else: + log.warning('Un-supported CI type') + + +def main(): + server = None + try: + loop = trollius.get_event_loop() + tasks = [ + trollius.async(handle_messages()), + ] + loop.run_until_complete(trollius.wait(tasks)) + loop.run_forever() + except KeyboardInterrupt: + pass + except trollius.ConnectionResetError: + pass + + log.info("End Connection") + loop.close() + log.info("End") + + +if __name__ == '__main__': + log = logging.getLogger("") + formatter = logging.Formatter( + "%(asctime)s %(levelname)s [%(module)s:%(lineno)d] %(message)s") + + # setup console logging + log.setLevel(logging.DEBUG) + ch = logging.StreamHandler() + ch.setLevel(logging.DEBUG) + + aslog = logging.getLogger("asyncio") + aslog.setLevel(logging.DEBUG) + + ch.setFormatter(formatter) + log.addHandler(ch) + main() From 92152a0e032f2aae629914f939d13e4e7674eefb Mon Sep 17 00:00:00 2001 From: Pierre-Yves Chibon Date: Aug 10 2016 21:05:04 +0000 Subject: [PATCH 21/69] Improve the documentation of the jenkins_ci_notification endpoint --- diff --git a/pagure/api/ci/jenkins.py b/pagure/api/ci/jenkins.py index 2d01423..bd70748 100644 --- a/pagure/api/ci/jenkins.py +++ b/pagure/api/ci/jenkins.py @@ -22,14 +22,23 @@ from pagure.api import API, APIERROR @API.route('/ci/jenkins//build-finished', methods=['POST']) def jenkins_ci_notification(pagure_ci_token): - """ Flag a pull-request based on the info provided by the CI service. + """ + Jenkins Build Notification + -------------------------- + At the end of a build on Jenkins, this URL is used (if the project is + rightly configured) to flag a pull-request with the result of the build. + + :: + + POST /api/0/ci/jenkins//build-finished + """ project = lib_ci.get_project_by_ci_token(SESSION, pagure_ci_token) if not project: flask.abort(404, 'No project corresponding to this CI token') - data = flask.request.json() + data = flask.request.get_json() if not data: flask.abort(400, "Bad Request: No JSON retrived") From 37a972a4777f3b79f328e68a329d607842100ef4 Mon Sep 17 00:00:00 2001 From: Pierre-Yves Chibon Date: Aug 10 2016 21:05:04 +0000 Subject: [PATCH 22/69] Document the CI API endpoints --- diff --git a/pagure/api/__init__.py b/pagure/api/__init__.py index 1c97c29..3c6bba1 100644 --- a/pagure/api/__init__.py +++ b/pagure/api/__init__.py @@ -202,7 +202,7 @@ from pagure.api import fork from pagure.api import project from pagure.api import user -if pagure.APP.config.get('PAGURE_CI', False): +if pagure.APP.config.get('PAGURE_CI_SERVICES', False): from pagure.api.ci import jenkins @@ -437,6 +437,11 @@ def api(): issues.append(load_doc(issue.api_view_issue_comment)) issues.append(load_doc(issue.api_comment_issue)) + ci = [] + if pagure.APP.config.get('PAGURE_CI_SERVICES', True): + if 'jenkins' in pagure.APP.config[('PAGURE_CI_SERVICES']: + ci.append(load_doc(jenkins.jenkins_ci_notification)) + api_pull_request_views_doc = load_doc(fork.api_pull_request_views) api_pull_request_view_doc = load_doc(fork.api_pull_request_view) api_pull_request_merge_doc = load_doc(fork.api_pull_request_merge) @@ -486,6 +491,7 @@ def api(): api_view_user_doc, api_groups_doc, ], + ci=ci, extras=extras, ) diff --git a/pagure/templates/api.html b/pagure/templates/api.html index b035f06..b34c151 100644 --- a/pagure/templates/api.html +++ b/pagure/templates/api.html @@ -76,6 +76,20 @@ {% endfor %} + {% if config.get('PAGURE_CI_SERVICES') %} +

+ Continous Integration Services + + + +

+ +
+ {% for html in ci %} + {{ html | InsertDiv |safe }} + {% endfor %} + {% endif %} +

Extras From ab380e9143fd90fa6f7c4505c8ff5eec91bb048e Mon Sep 17 00:00:00 2001 From: Pierre-Yves Chibon Date: Aug 10 2016 21:05:04 +0000 Subject: [PATCH 23/69] Fix small typo --- diff --git a/pagure/api/__init__.py b/pagure/api/__init__.py index 3c6bba1..1e1d55e 100644 --- a/pagure/api/__init__.py +++ b/pagure/api/__init__.py @@ -439,7 +439,7 @@ def api(): ci = [] if pagure.APP.config.get('PAGURE_CI_SERVICES', True): - if 'jenkins' in pagure.APP.config[('PAGURE_CI_SERVICES']: + if 'jenkins' in pagure.APP.config['PAGURE_CI_SERVICES']: ci.append(load_doc(jenkins.jenkins_ci_notification)) api_pull_request_views_doc = load_doc(fork.api_pull_request_views) From 2b83326e3f3b1e1717cbb37cd18d161945d8d764 Mon Sep 17 00:00:00 2001 From: Pierre-Yves Chibon Date: Aug 10 2016 21:05:04 +0000 Subject: [PATCH 24/69] Fix configuring and activating the Pagure CI plugin in one action Thanks to @skrzepto for finding the bug --- diff --git a/pagure/hooks/pagure_ci.py b/pagure/hooks/pagure_ci.py index ceb0d6a..689a0cb 100644 --- a/pagure/hooks/pagure_ci.py +++ b/pagure/hooks/pagure_ci.py @@ -125,11 +125,7 @@ class PagureCi(BaseHook): ''' Install the generic post-receive hook that allow us to call multiple post-receive hooks as set per plugin. ''' - hook = project.ci_hook[0] - if not hook.pagure_ci_token: - hook.pagure_ci_token = pagure.lib.login.id_generator(32) - SESSION.add(project) - SESSION.commit() + pass @classmethod def install(cls, project, dbobj): @@ -139,7 +135,10 @@ class PagureCi(BaseHook): should be installed ''' - pass + if not dbobj.pagure_ci_token: + dbobj.pagure_ci_token = pagure.lib.login.id_generator(32) + SESSION.add(dbobj) + SESSION.commit() @classmethod def remove(cls, project): From c48a709e940282c43a4c14595b7371b1b5a0de42 Mon Sep 17 00:00:00 2001 From: Pierre-Yves Chibon Date: Aug 10 2016 21:05:04 +0000 Subject: [PATCH 25/69] Display separately the jenkins token and the result URL --- diff --git a/pagure/hooks/pagure_ci.py b/pagure/hooks/pagure_ci.py index 689a0cb..37fbb6e 100644 --- a/pagure/hooks/pagure_ci.py +++ b/pagure/hooks/pagure_ci.py @@ -61,8 +61,13 @@ class PagureCITable(BASE): tmpl = """ {% if repo | hasattr('ci_hook') and repo.ci_hook and repo.ci_hook[0].pagure_ci_token %} -The URL to be used to POST the results of your build -is: + +The token to be used by jenkins to trigger the build is: +
+{{ repo.ci_hook[0].pagure_ci_token}}
+
+ +The URL to be used to POST the results of your build is: 
 {{ (config['APP_URL'][:-1] if config['APP_URL'].endswith('/')

From 8f2b558de0d27f625c176b448b7c84fd8cb66efc Mon Sep 17 00:00:00 2001
From: skrzepto 
Date: Aug 10 2016 21:05:04 +0000
Subject: [PATCH 26/69] fixing typo


---

diff --git a/pagure/api/ci/jenkins.py b/pagure/api/ci/jenkins.py
index bd70748..c2f5955 100644
--- a/pagure/api/ci/jenkins.py
+++ b/pagure/api/ci/jenkins.py
@@ -43,7 +43,7 @@ def jenkins_ci_notification(pagure_ci_token):
         flask.abort(400, "Bad Request: No JSON retrived")
 
     build_id = data.get('build', {}).get('number')
-    if not build_it:
+    if not build_id:
         flask.abort(400, "Bad Request: No build ID retrived")
 
     try:

From f5ba25dbf452dc35f0d3cfb719f462e49a8e0c1a Mon Sep 17 00:00:00 2001
From: Pierre-Yves Chibon 
Date: Aug 10 2016 21:05:04 +0000
Subject: [PATCH 27/69] Ensure that we are calling the proper ci_hook object


By default SQLAlchemy gives us a list so we need to access the first
element in it.

We could use `uselist=False` in the backref, but we'll need to adjust
all the plugins at one as otherwise we break pagure/ui/plugins.py.

---

diff --git a/pagure/lib/lib_ci.py b/pagure/lib/lib_ci.py
index 9853811..4e8f9ee 100644
--- a/pagure/lib/lib_ci.py
+++ b/pagure/lib/lib_ci.py
@@ -38,8 +38,8 @@ def process_jenkins_build(project, build_id):
     pull-request.
     """
     import jenkins
-    jenk = jenkins.Jenkins(project.ci_hook.url)
-    jenkins_name = project.ci_hook.url.split('/job/', 1)[1].split('/', 1)[0]
+    jenk = jenkins.Jenkins(project.ci_hook[0].url)
+    jenkins_name = project.ci_hook[0].url.split('/job/', 1)[1].split('/', 1)[0]
     build_info = jenk.get_build_info(jenkins_name, build_id)
     result = build_info['result']
     url = build_info['url']
diff --git a/pagureCI/pagure_ci_server.py b/pagureCI/pagure_ci_server.py
index a951646..8b43c65 100644
--- a/pagureCI/pagure_ci_server.py
+++ b/pagureCI/pagure_ci_server.py
@@ -94,7 +94,7 @@ def handle_messages():
         log.info("Trigger on %s PR #%s from %s: %s",
                       project.fullname, pr_id, repo, branch)
 
-        url = project.ci_hook.ci_url
+        url = project.ci_hook[0].ci_url
         if url.endswith('/'):
             url = url[:-1]
 

From 8cde7bfb5a8b6e5495a1cc1843699f08911fe5c2 Mon Sep 17 00:00:00 2001
From: Pierre-Yves Chibon 
Date: Aug 10 2016 21:05:04 +0000
Subject: [PATCH 28/69] Fix tests related to pagure-ci


---

diff --git a/tests/test_pagure_flask_ui_plugins_pagure_ci.py b/tests/test_pagure_flask_ui_plugins_pagure_ci.py
index 5db71c1..7783e76 100644
--- a/tests/test_pagure_flask_ui_plugins_pagure_ci.py
+++ b/tests/test_pagure_flask_ui_plugins_pagure_ci.py
@@ -57,9 +57,6 @@ class PagureFlaskPluginPagureCItests(tests.Modeltests):
                 'test project #1

', output.data) self.assertTrue('

Pagure CI settings

' in output.data) self.assertTrue( - '' - in output.data) - self.assertTrue( '' in output.data) self.assertTrue( From 168ef46860b778152d022318f879c97865303bd7 Mon Sep 17 00:00:00 2001 From: Pierre-Yves Chibon Date: Aug 10 2016 21:05:05 +0000 Subject: [PATCH 29/69] Rename pagureCI to pagure-ci --- diff --git a/pagure-ci/README.rst b/pagure-ci/README.rst new file mode 100644 index 0000000..9c88d0a --- /dev/null +++ b/pagure-ci/README.rst @@ -0,0 +1,118 @@ +Pagure CI +========= + +This is to setup Pagure CI for development. It is assumed that all the +dependencies are resolved. It is advised to use a virtual envivironment +for development. + + * Run:: + + python setup.py develop + + +Now in pagureCI/consumer.py add the following elements in `topic` list + +:: + + 'org.fedoraproject.dev.pagure.pull-request.new', + 'org.fedoraproject.dev.pagure.pull-request.comment.added', + + +Configuring Jenkins +=================== + +Jenkins configuration is the most important part of how the Pagure CI works, +after you login to your Jenkins Instance. + + +* Go to Manage Jenkins -> Configuire Global Security and under that select + 'Project-based Matrix Authorization Strategy' + +* Add a user and give all the permission to that user. + +* Download the following plugins: + +:: + + Build Authorization Root Plugin + Git Plugins + Notification Plugin + + +* Click on the New Item + +* Select Freestyle Project + +* Click OK and enter the name of the project, make sure the project name you + filled in the Pagure CI form should match the name you entered here. + +* Under 'Job Notification' click 'Add Endpoint' + +* Fields in Endpoint will be : + +:: + + FORMAT: JSON + PROTOCOL: HTTP + EVENT: Job Finalized + URL: + TIMEOUT: 3000 + LOG: 1 + +* Tick the build is parameterized + +* From the Add Parameter drop down select String Parameter + +* Two string parameters need to be created REPO and BRANCH + +* Source Code Management select Git and give the URL of the pagure project + +* Under Build Trigger click on Trigger build remotely and give the same token + that you gave in the Pagure CI form. + +* Under Build -> Add build step -> Execute Shell + +* In the box given enter the shell steps you want for testing your project. + + +Example Script + +:: + + if [ -n "$REPO" -a -n "$BRANCH" ]; then + git remote rm proposed || true + git remote add proposed "$REPO" + git fetch proposed + git checkout origin/master + git config --global user.email "you@example.com" + git config --global user.name "Your Name" + git merge --no-ff "proposed/$BRANCH" -m "Merge PR" + fi + +* After all the configuration done, go to the dev instance of pagure running + and under project settings in `Plugin` select Pagure CI and fill the appropriate + information. Which on submiting should give you a POST url. + +* Copy and paste the URL in the Notification section under the Jenkins project + you want the CI to work, + + +Get It Running: +=============== + +In one terminal window run: + +:: + + fedmsg-relay + +Another window: + +:: + + fedmsg-hub + +* Now clone the project locally and make a branch. make some changes and push it + to the repo and try to make a PR. You will notice if everything works fine a lot + of logs in the server console and `build fail` flag on the PR. + Build fail because there is no git server running. diff --git a/pagure-ci/__init__.py b/pagure-ci/__init__.py new file mode 100644 index 0000000..e69de29 --- /dev/null +++ b/pagure-ci/__init__.py diff --git a/pagure-ci/consumer.py b/pagure-ci/consumer.py new file mode 100644 index 0000000..ea97f30 --- /dev/null +++ b/pagure-ci/consumer.py @@ -0,0 +1,84 @@ +# -*- coding: utf-8 -*- +import fedmsg.consumers +from pagure.hooks import jenkins_hook +import pagure.lib +from pagure.lib import pagure_ci +from pagure.lib.model import BASE, Project, User +from pagure import APP, SESSION +import pagure.exceptions +PAGURE_MAIN_REPO = '{base}{name}.git' +PAGURE_FORK_REPO = '{base}forks/{user}/{name}.git' + + +class Integrator(fedmsg.consumers.FedmsgConsumer): + ''' Integrates Jenkins with Pagure. ''' + topic = [ + 'io.pagure.prod.pagure.pull-request.comment.added', + 'io.pagure.prod.pagure.pull-request.new', + 'org.fedoraproject.prod.jenkins.build', + ] + + config_key = 'integrator.enabled' + + + def __init__(self, hub): + super(Integrator, self).__init__(hub) + + def consume(self, msg): + ''' Pagure CI consumer which consumes message from + fedmsg and triggers Jenkins build. + ''' + topic, msg = msg['topic'], msg['body'] + self.log.info("Received %r, %r", topic, msg.get('msg_id', None)) + msg = msg['msg'] + try: + if topic.endswith('.pull-request.comment.added'): + if is_rebase(msg): + self.trigger_build(msg) + elif topic.endswith('.pull-request.new'): + self.trigger_build(msg) + else: + self.process_build(msg) + except jenkins_hook.ConfigNotFound as exc: + self.log.info('Unconfigured project %r', str(exc)) + except pagure.exceptions.HookInactiveException as exc: + self.log.info('Hook Inactive for project %r', str(exc)) + + def trigger_build(self, msg): + ''' Triggers or requests to start a build in Jenkins. ''' + pr_id = msg['pullrequest']['id'] + project = msg['pullrequest']['project']['name'] + branch = msg['pullrequest']['branch_from'] + + for cfg in jenkins_hook.get_configs(project, jenkins_hook.Service.PAGURE): + repo = msg['pullrequest'].get('remote_git') or get_repo(cfg, msg) + self.log.info("Trigger on %s PR #%s from %s: %s", + project, pr_id, repo, branch) + + pagure_ci.process_pr(self.log, cfg, pr_id, repo, branch) + + def process_build(self, msg): + ''' Extracts the information from the build and flag the pull-request. ''' + for cfg in jenkins_hook.get_configs(msg['project'], jenkins_hook.Service.JENKINS): + pagure_ci.process_build(self.log, cfg, msg['build']) + + +def get_repo(cfg, msg): + ''' Formats the URL for pagure repo. ''' + url = PAGURE_MAIN_REPO + if msg['pullrequest']['repo_from']['parent']: + url = PAGURE_FORK_REPO + return url.format( + base=APP.config['APP_URL'], + user=msg['pullrequest']['repo_from']['user']['name'], + name=msg['pullrequest']['repo_from']['name']) + + +def is_rebase(msg): + ''' Returns Rebase if the Pull-request is rebased. ''' + if msg['pullrequest']['status'] != 'Open': + return False + try: + return msg['pullrequest']['comments'][-1]['notification'] + except (IndexError, KeyError): + return False diff --git a/pagure-ci/pagure_ci_server.py b/pagure-ci/pagure_ci_server.py new file mode 100644 index 0000000..8b43c65 --- /dev/null +++ b/pagure-ci/pagure_ci_server.py @@ -0,0 +1,150 @@ +#!/usr/bin/env python +# -*- coding: utf-8 -*- + +""" + (c) 2016 - Copyright Red Hat Inc + + Authors: + Pierre-Yves Chibon + + +This server listens to message sent via redis and send the corresponding +web-hook request. + +Using this mechanism, we no longer block the main application if the +receiving end is offline or so. + +""" + +import datetime +import hashlib +import hmac +import json +import logging +import os +import requests +import time +import uuid + +import six +import trollius +import trollius_redis + +from kitchen.text.converters import to_bytes + + +log = logging.getLogger(__name__) + +if 'PAGURE_CONFIG' not in os.environ \ + and os.path.exists('/etc/pagure/pagure.cfg'): + print 'Using configuration file `/etc/pagure/pagure.cfg`' + os.environ['PAGURE_CONFIG'] = '/etc/pagure/pagure.cfg' + + +import pagure +import pagure.lib +from pagure.exceptions import PagureEvException + +_i = 0 + + +@trollius.coroutine +def handle_messages(): + connection = yield trollius.From(trollius_redis.Connection.create( + host='0.0.0.0', port=6379, db=0)) + + # Create subscriber. + subscriber = yield trollius.From(connection.start_subscribe()) + + # Subscribe to channel. + yield trollius.From(subscriber.subscribe(['pagure.ci'])) + + # Inside a while loop, wait for incoming events. + while True: + reply = yield trollius.From(subscriber.next_published()) + log.info( + 'Received: %s on channel: %s', + repr(reply.value), reply.channel) + data = json.loads(reply.value) + + pr_id = data['pr']['id'] + project = data['pr']['project']['name'] + branch = data['pr']['branch_from'] + + username = None + projectname = data['pr']['project'] + if data['pr'].get('parent'): + username, data['pr']['project']['user']['user'] + + project = pagure.lib.get_project( + session=pagure.SESSION, name=projectname, user=username) + + if not project: + log.warning( + 'No project could be found from the message %s' % data) + continue + + repo = data['pr'].get('remote_git') + if not repo: + base = pagure.APP.config['APP_URL'] + if base.endswith('/'): + base[:-1] + base += '/%s' % project.path + + log.info("Trigger on %s PR #%s from %s: %s", + project.fullname, pr_id, repo, branch) + + url = project.ci_hook[0].ci_url + if url.endswith('/'): + url = url[:-1] + + if data['ci_type'] == 'jenkins': + url += '/buildWithParameters' + request.post( + url, + data={ + 'token': cfg.jenkins_token, + 'cause': pr_id, + 'REPO': repo.fullname, + 'BRANCH': branch + } + ) + else: + log.warning('Un-supported CI type') + + +def main(): + server = None + try: + loop = trollius.get_event_loop() + tasks = [ + trollius.async(handle_messages()), + ] + loop.run_until_complete(trollius.wait(tasks)) + loop.run_forever() + except KeyboardInterrupt: + pass + except trollius.ConnectionResetError: + pass + + log.info("End Connection") + loop.close() + log.info("End") + + +if __name__ == '__main__': + log = logging.getLogger("") + formatter = logging.Formatter( + "%(asctime)s %(levelname)s [%(module)s:%(lineno)d] %(message)s") + + # setup console logging + log.setLevel(logging.DEBUG) + ch = logging.StreamHandler() + ch.setLevel(logging.DEBUG) + + aslog = logging.getLogger("asyncio") + aslog.setLevel(logging.DEBUG) + + ch.setFormatter(formatter) + log.addHandler(ch) + main() diff --git a/pagureCI/README.rst b/pagureCI/README.rst deleted file mode 100644 index 9c88d0a..0000000 --- a/pagureCI/README.rst +++ /dev/null @@ -1,118 +0,0 @@ -Pagure CI -========= - -This is to setup Pagure CI for development. It is assumed that all the -dependencies are resolved. It is advised to use a virtual envivironment -for development. - - * Run:: - - python setup.py develop - - -Now in pagureCI/consumer.py add the following elements in `topic` list - -:: - - 'org.fedoraproject.dev.pagure.pull-request.new', - 'org.fedoraproject.dev.pagure.pull-request.comment.added', - - -Configuring Jenkins -=================== - -Jenkins configuration is the most important part of how the Pagure CI works, -after you login to your Jenkins Instance. - - -* Go to Manage Jenkins -> Configuire Global Security and under that select - 'Project-based Matrix Authorization Strategy' - -* Add a user and give all the permission to that user. - -* Download the following plugins: - -:: - - Build Authorization Root Plugin - Git Plugins - Notification Plugin - - -* Click on the New Item - -* Select Freestyle Project - -* Click OK and enter the name of the project, make sure the project name you - filled in the Pagure CI form should match the name you entered here. - -* Under 'Job Notification' click 'Add Endpoint' - -* Fields in Endpoint will be : - -:: - - FORMAT: JSON - PROTOCOL: HTTP - EVENT: Job Finalized - URL: - TIMEOUT: 3000 - LOG: 1 - -* Tick the build is parameterized - -* From the Add Parameter drop down select String Parameter - -* Two string parameters need to be created REPO and BRANCH - -* Source Code Management select Git and give the URL of the pagure project - -* Under Build Trigger click on Trigger build remotely and give the same token - that you gave in the Pagure CI form. - -* Under Build -> Add build step -> Execute Shell - -* In the box given enter the shell steps you want for testing your project. - - -Example Script - -:: - - if [ -n "$REPO" -a -n "$BRANCH" ]; then - git remote rm proposed || true - git remote add proposed "$REPO" - git fetch proposed - git checkout origin/master - git config --global user.email "you@example.com" - git config --global user.name "Your Name" - git merge --no-ff "proposed/$BRANCH" -m "Merge PR" - fi - -* After all the configuration done, go to the dev instance of pagure running - and under project settings in `Plugin` select Pagure CI and fill the appropriate - information. Which on submiting should give you a POST url. - -* Copy and paste the URL in the Notification section under the Jenkins project - you want the CI to work, - - -Get It Running: -=============== - -In one terminal window run: - -:: - - fedmsg-relay - -Another window: - -:: - - fedmsg-hub - -* Now clone the project locally and make a branch. make some changes and push it - to the repo and try to make a PR. You will notice if everything works fine a lot - of logs in the server console and `build fail` flag on the PR. - Build fail because there is no git server running. diff --git a/pagureCI/__init__.py b/pagureCI/__init__.py deleted file mode 100644 index e69de29..0000000 --- a/pagureCI/__init__.py +++ /dev/null diff --git a/pagureCI/consumer.py b/pagureCI/consumer.py deleted file mode 100644 index ea97f30..0000000 --- a/pagureCI/consumer.py +++ /dev/null @@ -1,84 +0,0 @@ -# -*- coding: utf-8 -*- -import fedmsg.consumers -from pagure.hooks import jenkins_hook -import pagure.lib -from pagure.lib import pagure_ci -from pagure.lib.model import BASE, Project, User -from pagure import APP, SESSION -import pagure.exceptions -PAGURE_MAIN_REPO = '{base}{name}.git' -PAGURE_FORK_REPO = '{base}forks/{user}/{name}.git' - - -class Integrator(fedmsg.consumers.FedmsgConsumer): - ''' Integrates Jenkins with Pagure. ''' - topic = [ - 'io.pagure.prod.pagure.pull-request.comment.added', - 'io.pagure.prod.pagure.pull-request.new', - 'org.fedoraproject.prod.jenkins.build', - ] - - config_key = 'integrator.enabled' - - - def __init__(self, hub): - super(Integrator, self).__init__(hub) - - def consume(self, msg): - ''' Pagure CI consumer which consumes message from - fedmsg and triggers Jenkins build. - ''' - topic, msg = msg['topic'], msg['body'] - self.log.info("Received %r, %r", topic, msg.get('msg_id', None)) - msg = msg['msg'] - try: - if topic.endswith('.pull-request.comment.added'): - if is_rebase(msg): - self.trigger_build(msg) - elif topic.endswith('.pull-request.new'): - self.trigger_build(msg) - else: - self.process_build(msg) - except jenkins_hook.ConfigNotFound as exc: - self.log.info('Unconfigured project %r', str(exc)) - except pagure.exceptions.HookInactiveException as exc: - self.log.info('Hook Inactive for project %r', str(exc)) - - def trigger_build(self, msg): - ''' Triggers or requests to start a build in Jenkins. ''' - pr_id = msg['pullrequest']['id'] - project = msg['pullrequest']['project']['name'] - branch = msg['pullrequest']['branch_from'] - - for cfg in jenkins_hook.get_configs(project, jenkins_hook.Service.PAGURE): - repo = msg['pullrequest'].get('remote_git') or get_repo(cfg, msg) - self.log.info("Trigger on %s PR #%s from %s: %s", - project, pr_id, repo, branch) - - pagure_ci.process_pr(self.log, cfg, pr_id, repo, branch) - - def process_build(self, msg): - ''' Extracts the information from the build and flag the pull-request. ''' - for cfg in jenkins_hook.get_configs(msg['project'], jenkins_hook.Service.JENKINS): - pagure_ci.process_build(self.log, cfg, msg['build']) - - -def get_repo(cfg, msg): - ''' Formats the URL for pagure repo. ''' - url = PAGURE_MAIN_REPO - if msg['pullrequest']['repo_from']['parent']: - url = PAGURE_FORK_REPO - return url.format( - base=APP.config['APP_URL'], - user=msg['pullrequest']['repo_from']['user']['name'], - name=msg['pullrequest']['repo_from']['name']) - - -def is_rebase(msg): - ''' Returns Rebase if the Pull-request is rebased. ''' - if msg['pullrequest']['status'] != 'Open': - return False - try: - return msg['pullrequest']['comments'][-1]['notification'] - except (IndexError, KeyError): - return False diff --git a/pagureCI/pagure_ci_server.py b/pagureCI/pagure_ci_server.py deleted file mode 100644 index 8b43c65..0000000 --- a/pagureCI/pagure_ci_server.py +++ /dev/null @@ -1,150 +0,0 @@ -#!/usr/bin/env python -# -*- coding: utf-8 -*- - -""" - (c) 2016 - Copyright Red Hat Inc - - Authors: - Pierre-Yves Chibon - - -This server listens to message sent via redis and send the corresponding -web-hook request. - -Using this mechanism, we no longer block the main application if the -receiving end is offline or so. - -""" - -import datetime -import hashlib -import hmac -import json -import logging -import os -import requests -import time -import uuid - -import six -import trollius -import trollius_redis - -from kitchen.text.converters import to_bytes - - -log = logging.getLogger(__name__) - -if 'PAGURE_CONFIG' not in os.environ \ - and os.path.exists('/etc/pagure/pagure.cfg'): - print 'Using configuration file `/etc/pagure/pagure.cfg`' - os.environ['PAGURE_CONFIG'] = '/etc/pagure/pagure.cfg' - - -import pagure -import pagure.lib -from pagure.exceptions import PagureEvException - -_i = 0 - - -@trollius.coroutine -def handle_messages(): - connection = yield trollius.From(trollius_redis.Connection.create( - host='0.0.0.0', port=6379, db=0)) - - # Create subscriber. - subscriber = yield trollius.From(connection.start_subscribe()) - - # Subscribe to channel. - yield trollius.From(subscriber.subscribe(['pagure.ci'])) - - # Inside a while loop, wait for incoming events. - while True: - reply = yield trollius.From(subscriber.next_published()) - log.info( - 'Received: %s on channel: %s', - repr(reply.value), reply.channel) - data = json.loads(reply.value) - - pr_id = data['pr']['id'] - project = data['pr']['project']['name'] - branch = data['pr']['branch_from'] - - username = None - projectname = data['pr']['project'] - if data['pr'].get('parent'): - username, data['pr']['project']['user']['user'] - - project = pagure.lib.get_project( - session=pagure.SESSION, name=projectname, user=username) - - if not project: - log.warning( - 'No project could be found from the message %s' % data) - continue - - repo = data['pr'].get('remote_git') - if not repo: - base = pagure.APP.config['APP_URL'] - if base.endswith('/'): - base[:-1] - base += '/%s' % project.path - - log.info("Trigger on %s PR #%s from %s: %s", - project.fullname, pr_id, repo, branch) - - url = project.ci_hook[0].ci_url - if url.endswith('/'): - url = url[:-1] - - if data['ci_type'] == 'jenkins': - url += '/buildWithParameters' - request.post( - url, - data={ - 'token': cfg.jenkins_token, - 'cause': pr_id, - 'REPO': repo.fullname, - 'BRANCH': branch - } - ) - else: - log.warning('Un-supported CI type') - - -def main(): - server = None - try: - loop = trollius.get_event_loop() - tasks = [ - trollius.async(handle_messages()), - ] - loop.run_until_complete(trollius.wait(tasks)) - loop.run_forever() - except KeyboardInterrupt: - pass - except trollius.ConnectionResetError: - pass - - log.info("End Connection") - loop.close() - log.info("End") - - -if __name__ == '__main__': - log = logging.getLogger("") - formatter = logging.Formatter( - "%(asctime)s %(levelname)s [%(module)s:%(lineno)d] %(message)s") - - # setup console logging - log.setLevel(logging.DEBUG) - ch = logging.StreamHandler() - ch.setLevel(logging.DEBUG) - - aslog = logging.getLogger("asyncio") - aslog.setLevel(logging.DEBUG) - - ch.setFormatter(formatter) - log.addHandler(ch) - main() From 062a6cfb44d5fd16c61216907c827ce3d14ba6fd Mon Sep 17 00:00:00 2001 From: Pierre-Yves Chibon Date: Aug 10 2016 21:05:05 +0000 Subject: [PATCH 30/69] Include the pagure-ci module in the releases --- diff --git a/MANIFEST.in b/MANIFEST.in index f687a3c..d2ddd0d 100644 --- a/MANIFEST.in +++ b/MANIFEST.in @@ -1,6 +1,7 @@ include LICENSE README.rst requirements.txt UPGRADING.rst include createdb.py recursive-include pagure * +recursive-include pagure-ci * recursive-include files * recursive-include milters * recursive-include tests * From 41ae5708ad4070efdc9294ccec7cc91b60360ec8 Mon Sep 17 00:00:00 2001 From: skrzepto Date: Aug 10 2016 21:05:05 +0000 Subject: [PATCH 31/69] changing url to ci_url --- diff --git a/pagure/lib/lib_ci.py b/pagure/lib/lib_ci.py index 4e8f9ee..f9b4f92 100644 --- a/pagure/lib/lib_ci.py +++ b/pagure/lib/lib_ci.py @@ -38,8 +38,8 @@ def process_jenkins_build(project, build_id): pull-request. """ import jenkins - jenk = jenkins.Jenkins(project.ci_hook[0].url) - jenkins_name = project.ci_hook[0].url.split('/job/', 1)[1].split('/', 1)[0] + jenk = jenkins.Jenkins(project.ci_hook[0].ci_url) + jenkins_name = project.ci_hook[0].ci_url.split('/job/', 1)[1].split('/', 1)[0] build_info = jenk.get_build_info(jenkins_name, build_id) result = build_info['result'] url = build_info['url'] From 45417c572376240f91acec8ecd1eeea47c147b2f Mon Sep 17 00:00:00 2001 From: Pierre-Yves Chibon Date: Aug 10 2016 21:05:05 +0000 Subject: [PATCH 32/69] Clean-up/styling of the pagure spec file --- diff --git a/files/pagure.spec b/files/pagure.spec index 5a75a9d..6a5d6da 100644 --- a/files/pagure.spec +++ b/files/pagure.spec @@ -1,86 +1,86 @@ %{!?python_sitelib: %global python_sitelib %(%{__python} -c "from %distutils.sysconfig import get_python_lib; print (get_python_lib())")} -Name: pagure -Version: 2.3.3 -Release: 1%{?dist} -Summary: A git-centered forge - -License: GPLv2+ -URL: https://pagure.io/pagure -Source0: https://pagure.io/releases/pagure/%{name}-%{version}.tar.gz - -BuildArch: noarch - -BuildRequires: python2-devel -BuildRequires: python-setuptools -BuildRequires: python-nose - -BuildRequires: py-bcrypt -BuildRequires: python-alembic -BuildRequires: python-arrow -BuildRequires: python-binaryornot -BuildRequires: python-bleach -BuildRequires: python-blinker -BuildRequires: python-chardet -BuildRequires: python-cryptography -BuildRequires: python-docutils -BuildRequires: python-flask -BuildRequires: python-flask-wtf -BuildRequires: python-flask-multistatic -BuildRequires: python-markdown -BuildRequires: python-psutil -BuildRequires: python-pygit2 >= 0.20.1 -BuildRequires: python-pygments -BuildRequires: python-fedora -BuildRequires: python-openid -BuildRequires: python-openid-cla -BuildRequires: python-openid-teams -BuildRequires: python-straight-plugin -BuildRequires: python-wtforms -BuildRequires: python-munch -BuildRequires: python-enum34 -BuildRequires: python-redis +Name: pagure +Version: 2.3.3 +Release: 1%{?dist} +Summary: A git-centered forge + +License: GPLv2+ +URL: https://pagure.io/pagure +Source0: https://pagure.io/releases/pagure/%{name}-%{version}.tar.gz + +BuildArch: noarch + +BuildRequires: python2-devel +BuildRequires: python-setuptools +BuildRequires: python-nose + +BuildRequires: py-bcrypt +BuildRequires: python-alembic +BuildRequires: python-arrow +BuildRequires: python-binaryornot +BuildRequires: python-bleach +BuildRequires: python-blinker +BuildRequires: python-chardet +BuildRequires: python-cryptography +BuildRequires: python-docutils +BuildRequires: python-flask +BuildRequires: python-flask-wtf +BuildRequires: python-flask-multistatic +BuildRequires: python-markdown +BuildRequires: python-psutil +BuildRequires: python-pygit2 >= 0.20.1 +BuildRequires: python-pygments +BuildRequires: python-fedora +BuildRequires: python-openid +BuildRequires: python-openid-cla +BuildRequires: python-openid-teams +BuildRequires: python-straight-plugin +BuildRequires: python-wtforms +BuildRequires: python-munch +BuildRequires: python-enum34 +BuildRequires: python-redis # EPEL6 %if ( 0%{?rhel} && 0%{?rhel} == 6 ) -BuildRequires: python-sqlalchemy0.8 -Requires: python-sqlalchemy0.8 +BuildRequires: python-sqlalchemy0.8 +Requires: python-sqlalchemy0.8 %else -BuildRequires: python-sqlalchemy > 0.8 -Requires: python-sqlalchemy > 0.8 -BuildRequires: systemd +BuildRequires: python-sqlalchemy > 0.8 +Requires: python-sqlalchemy > 0.8 +BuildRequires: systemd %endif -Requires: py-bcrypt -Requires: python-alembic -Requires: python-arrow -Requires: python-binaryornot -Requires: python-bleach -Requires: python-blinker -Requires: python-chardet -Requires: python-cryptography -Requires: python-docutils -Requires: python-enum34 -Requires: python-flask -Requires: python-flask-wtf -Requires: python-flask-multistatic -Requires: python-markdown -Requires: python-psutil -Requires: python-pygit2 >= 0.20.1 -Requires: python-pygments -Requires: python-fedora -Requires: python-openid -Requires: python-openid-cla -Requires: python-openid-teams -Requires: python-straight-plugin -Requires: python-wtforms -Requires: python-munch -Requires: python-redis -Requires: mod_wsgi +Requires: py-bcrypt +Requires: python-alembic +Requires: python-arrow +Requires: python-binaryornot +Requires: python-bleach +Requires: python-blinker +Requires: python-chardet +Requires: python-cryptography +Requires: python-docutils +Requires: python-enum34 +Requires: python-flask +Requires: python-flask-wtf +Requires: python-flask-multistatic +Requires: python-markdown +Requires: python-psutil +Requires: python-pygit2 >= 0.20.1 +Requires: python-pygments +Requires: python-fedora +Requires: python-openid +Requires: python-openid-cla +Requires: python-openid-teams +Requires: python-straight-plugin +Requires: python-wtforms +Requires: python-munch +Requires: python-redis +Requires: mod_wsgi # No dependency of the app per se, but required to make it working. -Requires: gitolite3 +Requires: gitolite3 %description Pagure is a light-weight git-centered forge based on pygit2. @@ -90,7 +90,7 @@ system and possibilities to create new projects, fork existing ones and create/merge pull-requests across or within projects. -%package milters +%package milters Summary: Milter to integrate pagure with emails BuildArch: noarch BuildRequires: systemd-devel @@ -101,39 +101,39 @@ Requires(postun): systemd # It would work with sendmail but we configure things (like the tempfile) # to work with postfix Requires: postfix -%description milters +%description milters Milters (Mail filters) allowing the integration of pagure and emails. This is useful for example to allow commenting on a ticket by email. -%package ev -Summary: EventSource server for pagure -BuildArch: noarch +%package ev +Summary: EventSource server for pagure +BuildArch: noarch BuildRequires: systemd-devel -Requires: python-redis -Requires: python-trollius -Requires: python-trollius-redis -Requires(post): systemd -Requires(preun): systemd -Requires(postun): systemd -%description ev +Requires: python-redis +Requires: python-trollius +Requires: python-trollius-redis +Requires(post): systemd +Requires(preun): systemd +Requires(postun): systemd +%description ev Pagure comes with an eventsource server allowing live update of the pages supporting it. This package provides it. -%package webhook -Summary: Web-Hook server for pagure -BuildArch: noarch +%package webhook +Summary: Web-Hook server for pagure +BuildArch: noarch BuildRequires: systemd-devel -Requires: python-redis -Requires: python-trollius -Requires: python-trollius-redis -Requires(post): systemd -Requires(preun): systemd -Requires(postun): systemd -%description webhook +Requires: python-redis +Requires: python-trollius +Requires: python-trollius-redis +Requires(post): systemd +Requires(preun): systemd +Requires(postun): systemd +%description webhook Pagure comes with an webhook server allowing http callbacks for any action done on a project. This package provides it. @@ -505,7 +505,7 @@ install -m 644 webhook-server/pagure_webhook.service \ - Add a small padding at the bottom of the blockquote (Ryan Lerch) - In the list of closed PR, replace the column of the assignee with the date of closing (Ryan Lerch) -- Drop font awesome since we no longer use it and compress the png of the +- Drop font awesome since we no longer use it and compress the png of the current logo (Ryan Lerch) - Drop the svg of the old logo from the source (Ryan Lerch) - Add descriptions to the git hooks in the settings page (farhaanbukhsh) From f70306762b6812fc09d235af3d7162600074f3fd Mon Sep 17 00:00:00 2001 From: Pierre-Yves Chibon Date: Aug 10 2016 21:05:05 +0000 Subject: [PATCH 33/69] Drop the fedmsg consumer for pagure-ci Since we moved to use redis as a mean of communication between our services, the fedmsg consumer is no longer needed. --- diff --git a/pagure-ci/consumer.py b/pagure-ci/consumer.py deleted file mode 100644 index ea97f30..0000000 --- a/pagure-ci/consumer.py +++ /dev/null @@ -1,84 +0,0 @@ -# -*- coding: utf-8 -*- -import fedmsg.consumers -from pagure.hooks import jenkins_hook -import pagure.lib -from pagure.lib import pagure_ci -from pagure.lib.model import BASE, Project, User -from pagure import APP, SESSION -import pagure.exceptions -PAGURE_MAIN_REPO = '{base}{name}.git' -PAGURE_FORK_REPO = '{base}forks/{user}/{name}.git' - - -class Integrator(fedmsg.consumers.FedmsgConsumer): - ''' Integrates Jenkins with Pagure. ''' - topic = [ - 'io.pagure.prod.pagure.pull-request.comment.added', - 'io.pagure.prod.pagure.pull-request.new', - 'org.fedoraproject.prod.jenkins.build', - ] - - config_key = 'integrator.enabled' - - - def __init__(self, hub): - super(Integrator, self).__init__(hub) - - def consume(self, msg): - ''' Pagure CI consumer which consumes message from - fedmsg and triggers Jenkins build. - ''' - topic, msg = msg['topic'], msg['body'] - self.log.info("Received %r, %r", topic, msg.get('msg_id', None)) - msg = msg['msg'] - try: - if topic.endswith('.pull-request.comment.added'): - if is_rebase(msg): - self.trigger_build(msg) - elif topic.endswith('.pull-request.new'): - self.trigger_build(msg) - else: - self.process_build(msg) - except jenkins_hook.ConfigNotFound as exc: - self.log.info('Unconfigured project %r', str(exc)) - except pagure.exceptions.HookInactiveException as exc: - self.log.info('Hook Inactive for project %r', str(exc)) - - def trigger_build(self, msg): - ''' Triggers or requests to start a build in Jenkins. ''' - pr_id = msg['pullrequest']['id'] - project = msg['pullrequest']['project']['name'] - branch = msg['pullrequest']['branch_from'] - - for cfg in jenkins_hook.get_configs(project, jenkins_hook.Service.PAGURE): - repo = msg['pullrequest'].get('remote_git') or get_repo(cfg, msg) - self.log.info("Trigger on %s PR #%s from %s: %s", - project, pr_id, repo, branch) - - pagure_ci.process_pr(self.log, cfg, pr_id, repo, branch) - - def process_build(self, msg): - ''' Extracts the information from the build and flag the pull-request. ''' - for cfg in jenkins_hook.get_configs(msg['project'], jenkins_hook.Service.JENKINS): - pagure_ci.process_build(self.log, cfg, msg['build']) - - -def get_repo(cfg, msg): - ''' Formats the URL for pagure repo. ''' - url = PAGURE_MAIN_REPO - if msg['pullrequest']['repo_from']['parent']: - url = PAGURE_FORK_REPO - return url.format( - base=APP.config['APP_URL'], - user=msg['pullrequest']['repo_from']['user']['name'], - name=msg['pullrequest']['repo_from']['name']) - - -def is_rebase(msg): - ''' Returns Rebase if the Pull-request is rebased. ''' - if msg['pullrequest']['status'] != 'Open': - return False - try: - return msg['pullrequest']['comments'][-1]['notification'] - except (IndexError, KeyError): - return False From 0b25a797a87e78ec334ff27fcf3165dfc5cf2914 Mon Sep 17 00:00:00 2001 From: Pierre-Yves Chibon Date: Aug 10 2016 21:05:05 +0000 Subject: [PATCH 34/69] Drop empty file --- diff --git a/pagure-ci/__init__.py b/pagure-ci/__init__.py deleted file mode 100644 index e69de29..0000000 --- a/pagure-ci/__init__.py +++ /dev/null From 3bf8860aed9946f677b2db089ed8b19f0fbbcf0e Mon Sep 17 00:00:00 2001 From: Pierre-Yves Chibon Date: Aug 10 2016 21:05:05 +0000 Subject: [PATCH 35/69] Adjust the spec file to include pagure-ci sub-package --- diff --git a/files/pagure.spec b/files/pagure.spec index 6a5d6da..95baf02 100644 --- a/files/pagure.spec +++ b/files/pagure.spec @@ -138,6 +138,25 @@ Pagure comes with an webhook server allowing http callbacks for any action done on a project. This package provides it. +%package ci +Summary: A CI service for pagure +BuildArch: noarch + +BuildRequires: systemd-devel +Requires: python-redis +Requires: python-trollius +Requires: python-trollius-redis +Requires: python-jenkins +Requires(post): systemd +Requires(preun): systemd +Requires(postun): systemd +%description ci +Pagure comes with a continuous integration service, currently supporting +only jenkins but extendable to others. +With this service, your CI server will be able to report the results of the +build on the pull-requests opened to your project. + + %prep %setup -q @@ -200,12 +219,22 @@ install -m 755 webhook-server/pagure-webhook-server.py \ install -m 644 webhook-server/pagure_webhook.service \ $RPM_BUILD_ROOT/%{_unitdir}/pagure_webhook.service +# Install the ci service +mkdir -p $RPM_BUILD_ROOT/%{_libexecdir}/pagure-ci +install -m 755 pagure-ci/pagure_ci_server.py \ + $RPM_BUILD_ROOT/%{_libexecdir}/pagure-ci/pagure_ci_server.py +install -m 644 pagure-ci/pagure-ci.service \ + $RPM_BUILD_ROOT/%{_unitdir}/pagure-ci.service + + %post milters %systemd_post pagure_milter.service %post ev %systemd_post pagure_ev.service %post webhook %systemd_post pagure_webhook.service +%post ci +%systemd_post pagure_ci.service %preun milters %systemd_preun pagure_milter.service @@ -213,6 +242,8 @@ install -m 644 webhook-server/pagure_webhook.service \ %systemd_preun pagure_ev.service %preun webhook %systemd_preun pagure_webhook.service +%preun ci +%systemd_preun pagure_ci.service %postun milters %systemd_postun_with_restart pagure_milter.service @@ -220,6 +251,8 @@ install -m 644 webhook-server/pagure_webhook.service \ %systemd_postun_with_restart pagure_ev.service %postun webhook %systemd_postun_with_restart pagure_webhook.service +%postun ci +%systemd_postun_with_restart pagure_ci.service %files @@ -258,6 +291,12 @@ install -m 644 webhook-server/pagure_webhook.service \ %{_unitdir}/pagure_webhook.service +%files ci +%license LICENSE +%{_libexecdir}/pagure-ci/ +%{_unitdir}/pagure_ci.service + + %changelog * Fri Jul 15 2016 Pierre-Yves Chibon - 2.3.3-1 - Update to 2.3.3 From 2a0a83dab2dcde2e9cf938add7478d517632e758 Mon Sep 17 00:00:00 2001 From: Pierre-Yves Chibon Date: Aug 10 2016 21:05:05 +0000 Subject: [PATCH 36/69] Adjust a little the README.rst about pagure-ci --- diff --git a/pagure-ci/README.rst b/pagure-ci/README.rst index 9c88d0a..a6d51b6 100644 --- a/pagure-ci/README.rst +++ b/pagure-ci/README.rst @@ -2,21 +2,13 @@ Pagure CI ========= This is to setup Pagure CI for development. It is assumed that all the -dependencies are resolved. It is advised to use a virtual envivironment -for development. +dependencies are resolved. * Run:: - python setup.py develop + PAGURE_CONFIG=/path/to/config python pagure-ci/pagure_ci_server.py -Now in pagureCI/consumer.py add the following elements in `topic` list - -:: - - 'org.fedoraproject.dev.pagure.pull-request.new', - 'org.fedoraproject.dev.pagure.pull-request.comment.added', - Configuring Jenkins =================== @@ -52,12 +44,12 @@ after you login to your Jenkins Instance. :: - FORMAT: JSON - PROTOCOL: HTTP - EVENT: Job Finalized - URL: - TIMEOUT: 3000 - LOG: 1 + FORMAT: JSON + PROTOCOL: HTTP + EVENT: Job Finalized + URL: + TIMEOUT: 3000 + LOG: 1 * Tick the build is parameterized @@ -94,25 +86,4 @@ Example Script information. Which on submiting should give you a POST url. * Copy and paste the URL in the Notification section under the Jenkins project - you want the CI to work, - - -Get It Running: -=============== - -In one terminal window run: - -:: - - fedmsg-relay - -Another window: - -:: - - fedmsg-hub - -* Now clone the project locally and make a branch. make some changes and push it - to the repo and try to make a PR. You will notice if everything works fine a lot - of logs in the server console and `build fail` flag on the PR. - Build fail because there is no git server running. + you want the CI to work. From 2fee98892aef0d90ff43d2b8a3f52d3651009182 Mon Sep 17 00:00:00 2001 From: Pierre-Yves Chibon Date: Aug 10 2016 21:05:05 +0000 Subject: [PATCH 37/69] No need to add the object to the session, they are already tracked --- diff --git a/pagure/hooks/pagure_ci.py b/pagure/hooks/pagure_ci.py index 37fbb6e..da4fbdf 100644 --- a/pagure/hooks/pagure_ci.py +++ b/pagure/hooks/pagure_ci.py @@ -142,7 +142,6 @@ class PagureCi(BaseHook): ''' if not dbobj.pagure_ci_token: dbobj.pagure_ci_token = pagure.lib.login.id_generator(32) - SESSION.add(dbobj) SESSION.commit() @classmethod @@ -155,5 +154,4 @@ class PagureCi(BaseHook): ''' for hook in project.ci_hook: hook.pagure_ci_token = None - SESSION.add(project) SESSION.commit() From 6861a321593bb1eefc72eb5d858a2b4475c7f345 Mon Sep 17 00:00:00 2001 From: Pierre-Yves Chibon Date: Aug 10 2016 21:05:05 +0000 Subject: [PATCH 38/69] Fix unit-tests for the pagure-ci plugin --- diff --git a/tests/test_pagure_flask_ui_plugins_pagure_ci.py b/tests/test_pagure_flask_ui_plugins_pagure_ci.py index 7783e76..1305b3f 100644 --- a/tests/test_pagure_flask_ui_plugins_pagure_ci.py +++ b/tests/test_pagure_flask_ui_plugins_pagure_ci.py @@ -56,18 +56,12 @@ class PagureFlaskPluginPagureCItests(tests.Modeltests): '
\n' 'test project #1
', output.data) self.assertTrue('

Pagure CI settings

' in output.data) - self.assertTrue( - '' - in output.data) - self.assertTrue( - '' - in output.data) - self.assertTrue( - '' - in output.data) - self.assertTrue( - '' - in output.data) + self.assertIn( + '' , output.data) + self.assertIn( + '', + output.data) csrf_token = output.data.split( 'name="csrf_token" type="hidden" value="')[1].split('">')[0] @@ -80,56 +74,55 @@ class PagureFlaskPluginPagureCItests(tests.Modeltests): '
\n' 'test project #1
', output.data) self.assertTrue('

Pagure CI settings

' in output.data) - self.assertTrue( - '' - in output.data) - self.assertTrue( - '' - in output.data) - self.assertTrue( - '' - in output.data) - self.assertTrue( - '' - in output.data) - self.assertTrue( - '' - in output.data) + self.assertIn( + '' , output.data) + self.assertIn( + '', + output.data) # Activate hook data = { - 'csrf_token': csrf_token, 'active': 'y', - 'pagure_name': 'test', - 'jenkins_name': 'jenkins_test', - 'jenkins_url': 'https://jenkins.fedoraproject.org', - 'jenkins_token': 'BEEFCAFE' + 'ci_url': 'https://jenkins.fedoraproject.org', + 'ci_type': 'jenkins', } - # No git found + # CSRF Token missing output = self.app.post( '/test/settings/Pagure CI', data=data, follow_redirects=True) + self.assertEqual(output.status_code, 200) + self.assertIn( + '
\n' + 'test project #1
', output.data) + self.assertTrue('

Pagure CI settings

' in output.data) + self.assertIn( + '' , output.data) + self.assertIn( + '', output.data) + + data['csrf_token'] = csrf_token + + # No git found + output = self.app.post('/test/settings/Pagure', data=data) self.assertEqual(output.status_code, 404) tests.create_projects_git(tests.HERE) - data = {'csrf_token': csrf_token} - # With the git repo + # Activate hook output = self.app.post( '/test/settings/Pagure CI', data=data, follow_redirects=True) self.assertEqual(output.status_code, 200) - self.assertIn( '
\n' 'test project #1
', output.data) - self.assertTrue('

Pagure CI settings

' in output.data) - self.assertFalse( - '\n Hook activated' in output.data) - self.assertTrue( - '' - '\nThis field is required.' - in output.data) - self.assertTrue( - '' in output.data) + self.assertIn( + 'Settings - test - Pagure', output.data) + self.assertIn('

Settings for test

', output.data) + self.assertIn( + '\n Hook Pagure CI activated', + output.data) output = self.app.get('/test/settings/Pagure CI') self.assertEqual(output.status_code, 200) @@ -137,99 +130,60 @@ class PagureFlaskPluginPagureCItests(tests.Modeltests): '
\n' 'test project #1
', output.data) self.assertTrue('

Pagure CI settings

' in output.data) - self.assertTrue( - '' - in output.data) - self.assertTrue( - '' - in output.data) - self.assertTrue( - '' - in output.data) - self.assertTrue( - '' - in output.data) - self.assertTrue( - '' - in output.data) - - # Missing the required - data = {'csrf_token': csrf_token, 'active': 'y'} - - output = self.app.post( - '/test/settings/Pagure CI', data=data, follow_redirects=True) - self.assertEqual(output.status_code, 200) self.assertIn( - '
\n' - 'test project #1
', output.data) - self.assertTrue('

Pagure CI settings

' in output.data) - self.assertFalse( - '\n Hook activated' in output.data) + '' , output.data) self.assertTrue( - '' - '\nThis field is required.' + '' in output.data) - self.assertTrue( - '' in output.data) - # Activate hook + # De-activate the hook data = { 'csrf_token': csrf_token, - 'active': 'y', - 'pagure_name': 'test', - 'jenkins_name': 'jenkins_test', - 'jenkins_url': 'https://jenkins.fedoraproject.org', - 'jenkins_token': 'BEEFCAFE' } - output = self.app.post( '/test/settings/Pagure CI', data=data, follow_redirects=True) self.assertEqual(output.status_code, 200) self.assertIn( + '\n Hook Pagure CI inactived', + output.data) + self.assertIn( '
\n

Settings for test

', output.data) - self.assertTrue( - '\n Hook Pagure CI activated' in output.data) output = self.app.get('/test/settings/Pagure CI') + self.assertEqual(output.status_code, 200) self.assertIn( '
\n' 'test project #1
', output.data) self.assertTrue('

Pagure CI settings

' in output.data) - self.assertTrue( - '' - in output.data) - self.assertTrue( - '' - in output.data) - self.assertTrue( - '' - in output.data) - self.assertTrue( - '' - in output.data) - self.assertTrue( - '' - in output.data) + self.assertIn( + '' , output.data) + self.assertIn( + '', output.data) + + # Missing the required ci_url + data = {'csrf_token': csrf_token, 'active': 'y'} - # De-Activate hook - data = { - 'csrf_token': csrf_token, - 'pagure_name': 'test', - 'jenkins_name': 'jenkins_test', - 'jenkins_url': 'https://jenkins.fedoraproject.org', - 'jenkins_token': 'BEEFCAFE' - } output = self.app.post( '/test/settings/Pagure CI', data=data, follow_redirects=True) self.assertEqual(output.status_code, 200) - self.assertTrue( - '\n Hook Pagure CI inactived' in output.data) - self.assertIn( - '
\n

Settings for test

', + '
\n' + 'test project #1
', output.data) + self.assertIn('

Pagure CI settings

', output.data) + self.assertFalse( + '\n Hook activated' in output.data) + print output.data + self.assertIn( + '' + '\nThis field is required.', output.data) + self.assertIn( + '', output.data) if __name__ == '__main__': From b32b9496728175e1c521074ea702f8a1c7a35734 Mon Sep 17 00:00:00 2001 From: Pierre-Yves Chibon Date: Aug 10 2016 21:05:05 +0000 Subject: [PATCH 39/69] Get the session and the requestfolder from the api --- diff --git a/pagure/api/ci/jenkins.py b/pagure/api/ci/jenkins.py index c2f5955..53b41e6 100644 --- a/pagure/api/ci/jenkins.py +++ b/pagure/api/ci/jenkins.py @@ -47,7 +47,12 @@ def jenkins_ci_notification(pagure_ci_token): flask.abort(400, "Bad Request: No build ID retrived") try: - lib_ci.process_jenkins_build(project, build_id) + lib_ci.process_jenkins_build( + SESSION, + project, + build_id, + requestfolder=APP.config['REQUESTS_FOLDER'] + ) except pagure.exceptions.PagureException as err: APP.logger.error('Error processing jenkins notification', exc_info=err) flask.abort(400, "Bad Request: %s" % err) diff --git a/pagure/lib/lib_ci.py b/pagure/lib/lib_ci.py index f9b4f92..9d2b58e 100644 --- a/pagure/lib/lib_ci.py +++ b/pagure/lib/lib_ci.py @@ -33,7 +33,7 @@ def get_project_by_ci_token(session, ci_token): return query.first() -def process_jenkins_build(project, build_id): +def process_jenkins_build(session, project, build_id, requestfolder): """ Gets the build info from jenkins and flags that particular pull-request. """ @@ -77,6 +77,6 @@ def process_jenkins_build(project, build_id): url=url, uid=None, user=repo.user.username, - requestfolder=APP.config['REQUESTS_FOLDER'], + requestfolder=requestfolder, ) SESSION.commit() From 9bd98d07f4bd561a539b965189833bd6f93af4e0 Mon Sep 17 00:00:00 2001 From: Pierre-Yves Chibon Date: Aug 10 2016 21:05:05 +0000 Subject: [PATCH 40/69] Drop debugging line --- diff --git a/tests/test_pagure_flask_ui_plugins_pagure_ci.py b/tests/test_pagure_flask_ui_plugins_pagure_ci.py index 1305b3f..2becf79 100644 --- a/tests/test_pagure_flask_ui_plugins_pagure_ci.py +++ b/tests/test_pagure_flask_ui_plugins_pagure_ci.py @@ -176,7 +176,6 @@ class PagureFlaskPluginPagureCItests(tests.Modeltests): self.assertIn('

Pagure CI settings

', output.data) self.assertFalse( '\n Hook activated' in output.data) - print output.data self.assertIn( '' '\nThis field is required.', From c42414d359ae6e8ffdf4c54da74744c615410c2b Mon Sep 17 00:00:00 2001 From: Pierre-Yves Chibon Date: Aug 10 2016 21:05:05 +0000 Subject: [PATCH 41/69] Adjust the UI for the behavior of pagure.lib.__get_user pagure.lib.__get_user raises an exception if it doesn't find the user so the UI needs to catch it and handle it properly. --- diff --git a/pagure/ui/issues.py b/pagure/ui/issues.py index b954aac..98fd678 100644 --- a/pagure/ui/issues.py +++ b/pagure/ui/issues.py @@ -20,6 +20,7 @@ import kitchen.text.converters as ktc import mimetypes import pagure.doc_utils +import pagure.exceptions import pagure.lib import pagure.forms from pagure import (APP, SESSION, LOG, __get_file_in_tree, @@ -576,9 +577,10 @@ def new_issue(repo, username=None): content = form.issue_content.data private = form.private.data - user_obj = pagure.lib.__get_user( - SESSION, flask.g.fas_user.username) - if not user_obj: + try: + user_obj = pagure.lib.__get_user( + SESSION, flask.g.fas_user.username) + except pagure.exceptions.PagureException: flask.abort( 404, 'No such user found in the database: %s' % flask.g.fas_user.username) @@ -795,9 +797,10 @@ def edit_issue(repo, issueid, username=None): status = form.status.data private = form.private.data - user_obj = pagure.lib.__get_user( - SESSION, flask.g.fas_user.username) - if not user_obj: + try: + user_obj = pagure.lib.__get_user( + SESSION, flask.g.fas_user.username) + except pagure.exceptions.PagureException: flask.abort( 404, 'No such user found in the database: %s' % flask.g.fas_user.username) @@ -889,9 +892,10 @@ def upload_issue(repo, issueid, username=None): if issue is None or issue.project != repo: flask.abort(404, 'Issue not found') - user_obj = pagure.lib.__get_user( - SESSION, flask.g.fas_user.username) - if not user_obj: + try: + user_obj = pagure.lib.__get_user( + SESSION, flask.g.fas_user.username) + except pagure.exceptions.PagureException: flask.abort( 404, 'No such user found in the database: %s' % flask.g.fas_user.username) From f698993c71be66952205dd4c819a94ce62d290b2 Mon Sep 17 00:00:00 2001 From: Pierre-Yves Chibon Date: Aug 10 2016 21:05:06 +0000 Subject: [PATCH 42/69] Adjust unit-tests for the change in behavior when using an unknown user --- diff --git a/tests/test_pagure_flask_ui_issues.py b/tests/test_pagure_flask_ui_issues.py index 84f96ad..e9d5366 100644 --- a/tests/test_pagure_flask_ui_issues.py +++ b/tests/test_pagure_flask_ui_issues.py @@ -114,15 +114,10 @@ class PagureFlaskIssuestests(tests.Modeltests): # Invalid user data['csrf_token'] = csrf_token output = self.app.post('/test/new_issue', data=data) - self.assertEqual(output.status_code, 200) - self.assertTrue( - '
\n New issue' - in output.data) - self.assertEqual(output.data.count( - 'This field is required.'), 0) - self.assertTrue( - '\n No user "username" found' - in output.data) + self.assertEqual(output.status_code, 404) + self.assertIn( + '

No such user found in the database: username

', + output.data) user.username = 'pingou' with tests.user_set(pagure.APP, user): From cc97df3b2fd114932312e28d345cfef671e0d31f Mon Sep 17 00:00:00 2001 From: Farhaan Bukhsh Date: Aug 10 2016 21:05:06 +0000 Subject: [PATCH 43/69] Fix lib/lib_ci --- diff --git a/pagure/lib/lib_ci.py b/pagure/lib/lib_ci.py index 9d2b58e..7cf6082 100644 --- a/pagure/lib/lib_ci.py +++ b/pagure/lib/lib_ci.py @@ -38,7 +38,8 @@ def process_jenkins_build(session, project, build_id, requestfolder): pull-request. """ import jenkins - jenk = jenkins.Jenkins(project.ci_hook[0].ci_url) + # Jenkins Base URL + jenk = jenkins.Jenkins(project.ci_hook[0].ci_url.split('/job/')[0]) jenkins_name = project.ci_hook[0].ci_url.split('/job/', 1)[1].split('/', 1)[0] build_info = jenk.get_build_info(jenkins_name, build_id) result = build_info['result'] @@ -61,7 +62,7 @@ def process_jenkins_build(session, project, build_id, requestfolder): 'Unknown build status: %s' % result) request = pagure.lib.search_pull_requests( - session, project_id=project.id, requestid=requestid) + session, project_id=project.id, requestid=pr_id) if not request: raise pagure.exceptions.PagureException('Request not found') @@ -71,7 +72,7 @@ def process_jenkins_build(session, project, build_id, requestfolder): message = pagure.lib.add_pull_request_flag( session, request=request, - username=username, + username=project.ci_hook[0].ci_type, percent=percent, comment=comment, url=url, From 463ee4b18fe6c9d3f728024f89e02ebd90b9b6b5 Mon Sep 17 00:00:00 2001 From: Pierre-Yves Chibon Date: Aug 10 2016 21:05:06 +0000 Subject: [PATCH 44/69] Send notification to the CI service to kick off the build Let's kick the CI build when: * A new PR is created * An open PR gets a notification (rebase/new commits) And of course, only when there is a CI service configured for the project --- diff --git a/pagure/lib/__init__.py b/pagure/lib/__init__.py index 293f3c8..3a8193f 100644 --- a/pagure/lib/__init__.py +++ b/pagure/lib/__init__.py @@ -864,6 +864,15 @@ def add_pull_request_comment(session, request, commit, tree_id, filename, 'notification': notification, })) + # Send notification to the CI server, if the comment added was a + # notification and the PR is still open + if notification and request.status == 'Open' \ + and request.project.ci_hook and PAGURE_CI: + REDIS.publish('pagure.ci', json.dumps({ + 'ci_type': request.project.ci_hook.type_ci.type, + 'pr': request.to_json(public=True, with_comments=False) + })) + pagure.lib.notify.log( request.project, topic='pull-request.comment.added', @@ -1226,6 +1235,13 @@ def new_pull_request(session, branch_from, redis=REDIS, ) + # Send notification to the CI server + if request.project.ci_hook and PAGURE_CI: + REDIS.publish('pagure.ci', json.dumps({ + 'ci_type': request.project.ci_hook.type_ci.type, + 'pr': request.to_json(public=True, with_comments=False) + })) + return request From 2c6973e11d17cf62d0785e7977cf9e20553f36db Mon Sep 17 00:00:00 2001 From: Pierre-Yves Chibon Date: Aug 10 2016 21:05:06 +0000 Subject: [PATCH 45/69] Add the systemd service file for pagure_ci_server --- diff --git a/pagure-ci/pagure_ci.service b/pagure-ci/pagure_ci.service new file mode 100644 index 0000000..b93c675 --- /dev/null +++ b/pagure-ci/pagure_ci.service @@ -0,0 +1,14 @@ +[Unit] +Description=Pagure Continuous Integration service +After=redis.target +Documentation=https://pagure.io/pagure + +[Service] +ExecStart=/usr/libexec/pagure-ci/pagure_ci_server.py +Type=simple +User=git +Group=git +Restart=on-failure + +[Install] +WantedBy=multi-user.target From 5d94bb2ceebeb7d3af010a99b987473decb00519 Mon Sep 17 00:00:00 2001 From: Pierre-Yves Chibon Date: Aug 10 2016 21:05:06 +0000 Subject: [PATCH 46/69] Fix setting the different CI services supported --- diff --git a/pagure/__init__.py b/pagure/__init__.py index 8192ed9..cd09e14 100644 --- a/pagure/__init__.py +++ b/pagure/__init__.py @@ -174,6 +174,11 @@ if APP.config['EVENTSOURCE_SOURCE'] or APP.config['WEBHOOK']: db=APP.config['REDIS_DB'] ) + +if APP.config['PAGURE_CI_SERVICES']: + pagure.lib.set_pagure_ci(APP.config['PAGURE_CI_SERVICES']) + + if not APP.debug: APP.logger.addHandler(pagure.mail_logging.get_mail_handler( smtp_server=APP.config.get('SMTP_SERVER', '127.0.0.1'), diff --git a/pagure/lib/__init__.py b/pagure/lib/__init__.py index 3a8193f..0924fe9 100644 --- a/pagure/lib/__init__.py +++ b/pagure/lib/__init__.py @@ -58,6 +58,7 @@ def set_redis(host, port, db): def set_pagure_ci(services): """ Set the list of CI services supported by this pagure instance. """ + global PAGURE_CI PAGURE_CI = services From d6fb78ffabe0dd0298a535162d083819d59aba94 Mon Sep 17 00:00:00 2001 From: Pierre-Yves Chibon Date: Aug 10 2016 21:05:06 +0000 Subject: [PATCH 47/69] Fix the messages sent to the pagure-ci service via redis --- diff --git a/pagure/lib/__init__.py b/pagure/lib/__init__.py index 0924fe9..e46b636 100644 --- a/pagure/lib/__init__.py +++ b/pagure/lib/__init__.py @@ -870,7 +870,7 @@ def add_pull_request_comment(session, request, commit, tree_id, filename, if notification and request.status == 'Open' \ and request.project.ci_hook and PAGURE_CI: REDIS.publish('pagure.ci', json.dumps({ - 'ci_type': request.project.ci_hook.type_ci.type, + 'ci_type': request.project.ci_hook[0].ci_type, 'pr': request.to_json(public=True, with_comments=False) })) @@ -1239,7 +1239,7 @@ def new_pull_request(session, branch_from, # Send notification to the CI server if request.project.ci_hook and PAGURE_CI: REDIS.publish('pagure.ci', json.dumps({ - 'ci_type': request.project.ci_hook.type_ci.type, + 'ci_type': request.project.ci_hook[0].ci_type, 'pr': request.to_json(public=True, with_comments=False) })) From f790830001430f76539515e1adf27278dde0c22a Mon Sep 17 00:00:00 2001 From: Pierre-Yves Chibon Date: Aug 10 2016 21:05:06 +0000 Subject: [PATCH 48/69] Notify pagure-ci only if redis is configured --- diff --git a/pagure/lib/__init__.py b/pagure/lib/__init__.py index e46b636..8b1be61 100644 --- a/pagure/lib/__init__.py +++ b/pagure/lib/__init__.py @@ -1237,7 +1237,7 @@ def new_pull_request(session, branch_from, ) # Send notification to the CI server - if request.project.ci_hook and PAGURE_CI: + if REDIS and request.project.ci_hook and PAGURE_CI: REDIS.publish('pagure.ci', json.dumps({ 'ci_type': request.project.ci_hook[0].ci_type, 'pr': request.to_json(public=True, with_comments=False) From f240b64d66178048ac749f99cbd88cec489684d3 Mon Sep 17 00:00:00 2001 From: Pierre-Yves Chibon Date: Aug 10 2016 21:05:06 +0000 Subject: [PATCH 49/69] Also set-up redis if pagure-ci is configured --- diff --git a/pagure/__init__.py b/pagure/__init__.py index cd09e14..ceb0a06 100644 --- a/pagure/__init__.py +++ b/pagure/__init__.py @@ -167,7 +167,9 @@ if APP.config.get('PAGURE_AUTH', None) in ['fas', 'openid']: SESSION = pagure.lib.create_session(APP.config['DB_URL']) REDIS = None -if APP.config['EVENTSOURCE_SOURCE'] or APP.config['WEBHOOK']: +if APP.config['EVENTSOURCE_SOURCE'] \ + or APP.config['WEBHOOK'] \ + or APP.config['PAGURE_CI_SERVICES']: pagure.lib.set_redis( host=APP.config['REDIS_HOST'], port=APP.config['REDIS_PORT'], From 1edd4300fa6669ea2060c70892911a6f2d15d742 Mon Sep 17 00:00:00 2001 From: Pierre-Yves Chibon Date: Aug 10 2016 21:05:06 +0000 Subject: [PATCH 50/69] Add proper attribution of the code in lib_ci --- diff --git a/pagure/lib/lib_ci.py b/pagure/lib/lib_ci.py index 7cf6082..14afc9f 100644 --- a/pagure/lib/lib_ci.py +++ b/pagure/lib/lib_ci.py @@ -1,4 +1,15 @@ # -*- coding: utf-8 -*- + +""" + (c) 2016 - Copyright Red Hat Inc + + Authors: + Lubomír Sedlář + Farhaan Bukhsh + Pierre-Yves Chibon + +""" + import json import logging From fc54089a5efdafed1a72ad5f447a6651703b3f37 Mon Sep 17 00:00:00 2001 From: Pierre-Yves Chibon Date: Aug 10 2016 21:05:06 +0000 Subject: [PATCH 51/69] Fix retrieving the name of the project from the JSON blob --- diff --git a/pagure-ci/pagure_ci_server.py b/pagure-ci/pagure_ci_server.py index 8b43c65..4322bd6 100644 --- a/pagure-ci/pagure_ci_server.py +++ b/pagure-ci/pagure_ci_server.py @@ -72,7 +72,7 @@ def handle_messages(): branch = data['pr']['branch_from'] username = None - projectname = data['pr']['project'] + projectname = data['pr']['project']['name'] if data['pr'].get('parent'): username, data['pr']['project']['user']['user'] From fedc0ebbb5fbd0522dc852c15de8119b67f118a5 Mon Sep 17 00:00:00 2001 From: Pierre-Yves Chibon Date: Aug 10 2016 21:05:06 +0000 Subject: [PATCH 52/69] Log a little more info about where the build is triggered --- diff --git a/pagure-ci/pagure_ci_server.py b/pagure-ci/pagure_ci_server.py index 4322bd6..e9ce04d 100644 --- a/pagure-ci/pagure_ci_server.py +++ b/pagure-ci/pagure_ci_server.py @@ -100,6 +100,7 @@ def handle_messages(): if data['ci_type'] == 'jenkins': url += '/buildWithParameters' + log.info('Triggering the build at: %s', url) request.post( url, data={ From 0fce2961eb2970e55527da8eca9c0fac7c80d3e0 Mon Sep 17 00:00:00 2001 From: Pierre-Yves Chibon Date: Aug 10 2016 21:05:06 +0000 Subject: [PATCH 53/69] Disable pagure-ci by default --- diff --git a/pagure/default_config.py b/pagure/default_config.py index 4a7ffae..6fe64e6 100644 --- a/pagure/default_config.py +++ b/pagure/default_config.py @@ -223,4 +223,4 @@ BOOTSTRAP_URLS_CSS = 'https://apps.fedoraproject.org/global/fedora-bootstrap-1.0 BOOTSTRAP_URLS_JS = 'https://apps.fedoraproject.org/global/fedora-bootstrap-1.0.1/fedora-bootstrap.js' # List of the type of CI service supported by this pagure instance -PAGURE_CI_SERVICES = ['jenkins'] +PAGURE_CI_SERVICES = [] From e75d039887f7414dc661287e86046f704cb86e45 Mon Sep 17 00:00:00 2001 From: Pierre-Yves Chibon Date: Aug 10 2016 21:05:06 +0000 Subject: [PATCH 54/69] Fix invalid variable and module names and log when it finished to process --- diff --git a/pagure-ci/pagure_ci_server.py b/pagure-ci/pagure_ci_server.py index e9ce04d..d2bfe62 100644 --- a/pagure-ci/pagure_ci_server.py +++ b/pagure-ci/pagure_ci_server.py @@ -101,18 +101,20 @@ def handle_messages(): if data['ci_type'] == 'jenkins': url += '/buildWithParameters' log.info('Triggering the build at: %s', url) - request.post( + requests.post( url, data={ - 'token': cfg.jenkins_token, + 'token': project.ci_hook[0].pagure_ci_token, 'cause': pr_id, - 'REPO': repo.fullname, + 'REPO': project.fullname, 'BRANCH': branch } ) else: log.warning('Un-supported CI type') + log.info('Ready for another') + def main(): server = None From 5172a476cdff72b889d3bc9c5629b9d105a7a442 Mon Sep 17 00:00:00 2001 From: Pierre-Yves Chibon Date: Aug 10 2016 21:05:07 +0000 Subject: [PATCH 55/69] Enable the jenkins CI service for the tests --- diff --git a/runtests.sh b/runtests.sh index eb8063b..7be1368 100755 --- a/runtests.sh +++ b/runtests.sh @@ -1,3 +1,3 @@ #!/bin/bash -PYTHONPATH=pagure ./nosetests --with-coverage --cover-erase --cover-package=pagure $* +PAGURE_CONFIG=../tests/config PYTHONPATH=pagure ./nosetests --with-coverage --cover-erase --cover-package=pagure $* diff --git a/tests/config b/tests/config new file mode 100644 index 0000000..d52090c --- /dev/null +++ b/tests/config @@ -0,0 +1 @@ +PAGURE_CI_SERVICES = ['jenkins'] From ebc5250e7041257b11799ffe1764979fd1dbc417 Mon Sep 17 00:00:00 2001 From: Pierre-Yves Chibon Date: Aug 10 2016 21:05:07 +0000 Subject: [PATCH 56/69] Undo specifying a config file to run the tests and just bail on jenkins Basically, bail if there is no CI service enabled --- diff --git a/runtests.sh b/runtests.sh index 7be1368..eb8063b 100755 --- a/runtests.sh +++ b/runtests.sh @@ -1,3 +1,3 @@ #!/bin/bash -PAGURE_CONFIG=../tests/config PYTHONPATH=pagure ./nosetests --with-coverage --cover-erase --cover-package=pagure $* +PYTHONPATH=pagure ./nosetests --with-coverage --cover-erase --cover-package=pagure $* diff --git a/tests/config b/tests/config deleted file mode 100644 index d52090c..0000000 --- a/tests/config +++ /dev/null @@ -1 +0,0 @@ -PAGURE_CI_SERVICES = ['jenkins'] diff --git a/tests/test_pagure_flask_ui_plugins_pagure_ci.py b/tests/test_pagure_flask_ui_plugins_pagure_ci.py index 2becf79..aebe7b7 100644 --- a/tests/test_pagure_flask_ui_plugins_pagure_ci.py +++ b/tests/test_pagure_flask_ui_plugins_pagure_ci.py @@ -110,6 +110,9 @@ class PagureFlaskPluginPagureCItests(tests.Modeltests): tests.create_projects_git(tests.HERE) + if APP.config.get('PAGURE_CI_SERVICES', None) is None: + return + # Activate hook output = self.app.post( '/test/settings/Pagure CI', data=data, follow_redirects=True) From 508e20c9f98bd0ce05e5f0c6acc57ea9659f5ceb Mon Sep 17 00:00:00 2001 From: Farhaan Bukhsh Date: Aug 10 2016 21:05:07 +0000 Subject: [PATCH 57/69] Fix typos in lib_ci.py --- diff --git a/pagure/lib/lib_ci.py b/pagure/lib/lib_ci.py index 14afc9f..5eb8b05 100644 --- a/pagure/lib/lib_ci.py +++ b/pagure/lib/lib_ci.py @@ -88,7 +88,7 @@ def process_jenkins_build(session, project, build_id, requestfolder): comment=comment, url=url, uid=None, - user=repo.user.username, + user=project.user.username, requestfolder=requestfolder, ) - SESSION.commit() + session.commit() From 6259289a04b254d51017f06a38e06be7d9bda1dd Mon Sep 17 00:00:00 2001 From: Pierre-Yves Chibon Date: Aug 10 2016 21:05:07 +0000 Subject: [PATCH 58/69] Improve the README for pagure-ci --- diff --git a/pagure-ci/README.rst b/pagure-ci/README.rst index a6d51b6..0ab6362 100644 --- a/pagure-ci/README.rst +++ b/pagure-ci/README.rst @@ -6,12 +6,12 @@ dependencies are resolved. * Run:: - PAGURE_CONFIG=/path/to/config python pagure-ci/pagure_ci_server.py + PAGURE_CONFIG=/path/to/config PYTHONPATH=. python pagure-ci/pagure_ci_server.py -Configuring Jenkins -=================== +Configure Jenkins +================= Jenkins configuration is the most important part of how the Pagure CI works, after you login to your Jenkins Instance. @@ -24,43 +24,42 @@ after you login to your Jenkins Instance. * Download the following plugins: -:: - - Build Authorization Root Plugin - Git Plugins - Notification Plugin + * Build Authorization Root Plugin + * `Git Plugin `_ + * `Notification Plugin `_ -* Click on the New Item +Configure your project on Jenkins +================================= -* Select Freestyle Project +* Start by enabling the `Pagure CI` hook in the settings of your project on + pagure. This will provide you two values needed to configure your project + on jenkins: a token and an URL that jenkins calls to return the results + of a build. -* Click OK and enter the name of the project, make sure the project name you - filled in the Pagure CI form should match the name you entered here. +* Go to the `Configure` page of your project -* Under 'Job Notification' click 'Add Endpoint' +* Under `Job Notification` click `Add Endpoint` * Fields in Endpoint will be : :: - FORMAT: JSON - PROTOCOL: HTTP - EVENT: Job Finalized - URL: - TIMEOUT: 3000 - LOG: 1 - -* Tick the build is parameterized + FORMAT: JSON + PROTOCOL: HTTP + EVENT: Job Finalized + URL: + TIMEOUT: 3000 + LOG: 1 -* From the Add Parameter drop down select String Parameter +* Tick the checkbox `This build is parameterized` -* Two string parameters need to be created REPO and BRANCH +* Add two `String Parameters` named REPO and BRANCH * Source Code Management select Git and give the URL of the pagure project -* Under Build Trigger click on Trigger build remotely and give the same token - that you gave in the Pagure CI form. +* Under Build Trigger click on Trigger build remotely and specify the token + given by pagure. * Under Build -> Add build step -> Execute Shell @@ -71,19 +70,15 @@ Example Script :: - if [ -n "$REPO" -a -n "$BRANCH" ]; then - git remote rm proposed || true - git remote add proposed "$REPO" - git fetch proposed - git checkout origin/master - git config --global user.email "you@example.com" - git config --global user.name "Your Name" - git merge --no-ff "proposed/$BRANCH" -m "Merge PR" - fi - -* After all the configuration done, go to the dev instance of pagure running - and under project settings in `Plugin` select Pagure CI and fill the appropriate - information. Which on submiting should give you a POST url. - -* Copy and paste the URL in the Notification section under the Jenkins project - you want the CI to work. + # Script specific for Pull-Request build + if [ -n "$REPO" -a -n "$BRANCH" ]; then + git remote rm proposed || true + git remote add proposed "$REPO" + git fetch proposed + git checkout origin/master + git config --global user.email "you@example.com" + git config --global user.name "Your Name" + git merge --no-ff "proposed/$BRANCH" -m "Merge PR" + fi + + # Part of the script specific to how you run the tests on your project From 26a959fe6711cfe9a3f4dd8405821c7a93703f06 Mon Sep 17 00:00:00 2001 From: Pierre-Yves Chibon Date: Aug 10 2016 21:05:07 +0000 Subject: [PATCH 59/69] Fix typo in the pagure-ci tests --- diff --git a/tests/test_pagure_flask_ui_plugins_pagure_ci.py b/tests/test_pagure_flask_ui_plugins_pagure_ci.py index aebe7b7..2a2f2cd 100644 --- a/tests/test_pagure_flask_ui_plugins_pagure_ci.py +++ b/tests/test_pagure_flask_ui_plugins_pagure_ci.py @@ -110,7 +110,7 @@ class PagureFlaskPluginPagureCItests(tests.Modeltests): tests.create_projects_git(tests.HERE) - if APP.config.get('PAGURE_CI_SERVICES', None) is None: + if pagure.APP.config.get('PAGURE_CI_SERVICES', None) is None: return # Activate hook From 3e5bbc57856b66b55d9ebb5339c750b128e53e4b Mon Sep 17 00:00:00 2001 From: Pierre-Yves Chibon Date: Aug 10 2016 21:05:07 +0000 Subject: [PATCH 60/69] Use urlparse.urljoin to build the final url to call --- diff --git a/pagure-ci/pagure_ci_server.py b/pagure-ci/pagure_ci_server.py index d2bfe62..8ed72eb 100644 --- a/pagure-ci/pagure_ci_server.py +++ b/pagure-ci/pagure_ci_server.py @@ -24,6 +24,7 @@ import logging import os import requests import time +import urlparse import uuid import six @@ -95,11 +96,9 @@ def handle_messages(): project.fullname, pr_id, repo, branch) url = project.ci_hook[0].ci_url - if url.endswith('/'): - url = url[:-1] if data['ci_type'] == 'jenkins': - url += '/buildWithParameters' + url = urlparse.urljoin(url, '/buildWithParameters') log.info('Triggering the build at: %s', url) requests.post( url, From ec30fd1d34305eb6a98b0830ab2eb15ddcbcdf63 Mon Sep 17 00:00:00 2001 From: Pierre-Yves Chibon Date: Aug 10 2016 21:05:07 +0000 Subject: [PATCH 61/69] Drop un-used variable and retrieve the redis information from the configuration --- diff --git a/pagure-ci/pagure_ci_server.py b/pagure-ci/pagure_ci_server.py index 8ed72eb..fb43952 100644 --- a/pagure-ci/pagure_ci_server.py +++ b/pagure-ci/pagure_ci_server.py @@ -46,13 +46,14 @@ import pagure import pagure.lib from pagure.exceptions import PagureEvException -_i = 0 - @trollius.coroutine def handle_messages(): + host = pagure.APP.config.get('REDIS_HOST', '0.0.0.0') + port = pagure.APP.config.get('REDIS_PORT', 6379) + db = pagure.APP.config.get('REDIS_DB', 0) connection = yield trollius.From(trollius_redis.Connection.create( - host='0.0.0.0', port=6379, db=0)) + host=host, port=port, db=db)) # Create subscriber. subscriber = yield trollius.From(connection.start_subscribe()) From 1e6f32c4f18dfda82029ae294702a114715ad938 Mon Sep 17 00:00:00 2001 From: Pierre-Yves Chibon Date: Aug 10 2016 21:05:07 +0000 Subject: [PATCH 62/69] Fix wording in the exception message --- diff --git a/pagure/lib/lib_ci.py b/pagure/lib/lib_ci.py index 5eb8b05..d2ea609 100644 --- a/pagure/lib/lib_ci.py +++ b/pagure/lib/lib_ci.py @@ -66,7 +66,7 @@ def process_jenkins_build(session, project, build_id, requestfolder): if not pr_id: raise pagure.exceptions.PagureException( - 'No PR found corresponding') + 'No corresponding PR found') if result not in BUILD_STATS: pagure.exceptions.PagureException( From 9ae10961fa004f1561d8d9fedb66aa07dc70fb72 Mon Sep 17 00:00:00 2001 From: Pierre-Yves Chibon Date: Aug 10 2016 21:05:07 +0000 Subject: [PATCH 63/69] Fix typo in the docstring of the jinja_render filter --- diff --git a/pagure/ui/filters.py b/pagure/ui/filters.py index ff5113f..825c87f 100644 --- a/pagure/ui/filters.py +++ b/pagure/ui/filters.py @@ -38,7 +38,7 @@ def jinja_hasattr(obj, string): @APP.template_filter('render') def jinja_render(tmpl, **kwargs): - """ Render the tiven template with the provided arguments + """ Render the given template with the provided arguments """ return flask.render_template_string(tmpl, **kwargs) From d3caaaf46e81d4877dc69dcfbacd5e3cd291b8a4 Mon Sep 17 00:00:00 2001 From: Pierre-Yves Chibon Date: Aug 10 2016 21:05:07 +0000 Subject: [PATCH 64/69] Remove un-used import --- diff --git a/pagure/ui/plugins.py b/pagure/ui/plugins.py index d0eda4d..cf478c1 100644 --- a/pagure/ui/plugins.py +++ b/pagure/ui/plugins.py @@ -24,7 +24,6 @@ from pagure.lib import model import json from kitchen.text.converters import to_bytes -from cryptography.hazmat.primitives import constant_time # pylint: disable=E1101 From c5a776fa4a979c1c804e0925d75aed2cee4fa325 Mon Sep 17 00:00:00 2001 From: Pierre-Yves Chibon Date: Aug 10 2016 21:05:07 +0000 Subject: [PATCH 65/69] Change the endpoint called by jenkins to include the repo name and username We can then do a time constant check of the token provided making it harder to brute force the token since you now have to do it on project instead of trying until you find a project that works. --- diff --git a/pagure/api/ci/jenkins.py b/pagure/api/ci/jenkins.py index 53b41e6..74673a4 100644 --- a/pagure/api/ci/jenkins.py +++ b/pagure/api/ci/jenkins.py @@ -10,6 +10,8 @@ import flask +from cryptography.hazmat.primitives import constant_time +from kitchen.text.converters import to_bytes from sqlalchemy.exc import SQLAlchemyError import pagure @@ -20,8 +22,10 @@ from pagure import APP, SESSION from pagure.api import API, APIERROR -@API.route('/ci/jenkins//build-finished', methods=['POST']) -def jenkins_ci_notification(pagure_ci_token): + +@API.route('/ci/jenkins///build-finished', methods=['POST']) +@API.route('/ci/jenkins/forks////build-finished', methods=['POST']) +def jenkins_ci_notification(repo, pagure_ci_token, username=None): """ Jenkins Build Notification -------------------------- @@ -34,9 +38,14 @@ def jenkins_ci_notification(pagure_ci_token): """ - project = lib_ci.get_project_by_ci_token(SESSION, pagure_ci_token) - if not project: - flask.abort(404, 'No project corresponding to this CI token') + project = pagure.lib.get_project(SESSION, repo, user=username) + if repo is None: + flask.abort(404, 'Project not found') + + if not constant_time.bytes_eq( + to_bytes(pagure_ci_token), + to_bytes(project.ci_hook[0].pagure_ci_token)): + return ('Token mismatch', 401) data = flask.request.get_json() if not data: diff --git a/pagure/hooks/pagure_ci.py b/pagure/hooks/pagure_ci.py index da4fbdf..12c2576 100644 --- a/pagure/hooks/pagure_ci.py +++ b/pagure/hooks/pagure_ci.py @@ -68,11 +68,11 @@ The token to be used by jenkins to trigger the build is: The URL to be used to POST the results of your build is: - 
 {{ (config['APP_URL'][:-1] if config['APP_URL'].endswith('/')
   else config['APP_URL'])
   + url_for('api_ns.%s_ci_notification' % repo.ci_hook[0].ci_type,
+    repo=repo.name, username=username,
     pagure_ci_token=repo.ci_hook[0].pagure_ci_token) }}
diff --git a/pagure/templates/plugin.html b/pagure/templates/plugin.html index ed5c240..63ede4b 100644 --- a/pagure/templates/plugin.html +++ b/pagure/templates/plugin.html @@ -24,7 +24,7 @@ {% if plugin.extra_info %} - {{ plugin.extra_info | render(repo=repo) | safe }} + {{ plugin.extra_info | render(repo=repo, username=username) | safe }} {% endif %}

From 084b3d3095e56d9e9e51f65545b86d296dc798bb Mon Sep 17 00:00:00 2001 From: Pierre-Yves Chibon Date: Aug 10 2016 21:05:07 +0000 Subject: [PATCH 66/69] cryptography is also a dep for pagure-ci --- diff --git a/requirements.txt b/requirements.txt index de1cd79..355c34d 100644 --- a/requirements.txt +++ b/requirements.txt @@ -37,5 +37,6 @@ cryptography py-bcrypt #Required for Pagure CI +cryptography fedmsg python-jenkins From 5733f5dc6819d2dc569a9fd19693c60b86a80a10 Mon Sep 17 00:00:00 2001 From: Pierre-Yves Chibon Date: Aug 10 2016 21:05:07 +0000 Subject: [PATCH 67/69] fedmsg is no longer required for pagure-ci --- diff --git a/requirements.txt b/requirements.txt index 355c34d..7f97847 100644 --- a/requirements.txt +++ b/requirements.txt @@ -38,5 +38,4 @@ py-bcrypt #Required for Pagure CI cryptography -fedmsg python-jenkins From 8075d78446c0bbfbb7c23a89b9ec5889b16255d6 Mon Sep 17 00:00:00 2001 From: Patrick Uiterwijk Date: Aug 10 2016 21:05:08 +0000 Subject: [PATCH 68/69] Implement scanning of attached files for viruses Signed-off-by: Patrick Uiterwijk --- diff --git a/doc/configuration.rst b/doc/configuration.rst index eec82c7..42a36c0 100644 --- a/doc/configuration.rst +++ b/doc/configuration.rst @@ -171,6 +171,15 @@ when building the ``msg-id`` header of the emails sent. Defaults to: ``pagure.org`` +VIRUS_SCAN_ATTACHMENTS +~~~~~~~~~~~~~~~~~~~~~~ + +This setting configures whether attachments are scanned for viruses on +upload. For more information, see the install.rst guide. + +Defaults to: ``False`` + + Configure Gitolite ------------------ diff --git a/doc/install.rst b/doc/install.rst index 7490087..74ccb81 100644 --- a/doc/install.rst +++ b/doc/install.rst @@ -262,3 +262,20 @@ The ``alembic stamp`` command is the one actually saving the current revision into the database. This current revision is found using ``alembic heads`` which returns the most recent revision found by alembic, and since the database was just created, it is at the latest revision. + + +Set up virus scannining +----------------------- +Pagure can automatically scan uploaded attachments for viruses using Clam. +To set this up, first install clamav-data-empty, clamav-server, +clamav-server-systemd and clamav-update. + +Then edit /etc/freshclam.conf, removing the Example line and run freshclam once +to get an up to date database. + +Copy /usr/share/doc/clamav-server/clamd.conf to /etc/clamd.conf and edit that +too, again making sure to remove the Example line. Make sure to set LocalSocket +to a file in a directory that exists, and set User to an existing system user. + +Then start the clamd service and set VIRUS_SCAN_ATTACHMENTS = True in the +Pagure configuration. diff --git a/files/pagure.cfg.sample b/files/pagure.cfg.sample index a160c98..bd6d8f0 100644 --- a/files/pagure.cfg.sample +++ b/files/pagure.cfg.sample @@ -92,6 +92,9 @@ REMOTE_GIT_FOLDER = os.path.join( 'remotes' ) +### Whether to enable scanning for viruses in attachments +VIRUS_SCAN_ATTACHMENTS = False + ### Configuration file for gitolite GITOLITE_CONFIG = os.path.join( diff --git a/pagure/default_config.py b/pagure/default_config.py index 6fe64e6..d9a497c 100644 --- a/pagure/default_config.py +++ b/pagure/default_config.py @@ -123,6 +123,8 @@ REMOTE_GIT_FOLDER = os.path.join( 'remotes' ) +### Whether to enable scanning for viruses in attachments +VIRUS_SCAN_ATTACHMENTS = False # Configuration file for gitolite GITOLITE_CONFIG = os.path.join( diff --git a/pagure/forms.py b/pagure/forms.py index a9d1e0a..d6d0e24 100644 --- a/pagure/forms.py +++ b/pagure/forms.py @@ -10,9 +10,13 @@ import re from flask.ext import wtf +import flask import wtforms +import tempfile # pylint: disable=R0903,W0232,E1002 +import pagure + STRICT_REGEX = '^[a-zA-Z0-9-_]+$' TAGS_REGEX = '^[a-zA-Z0-9-_, .]+$' @@ -20,6 +24,32 @@ PROJECT_NAME_REGEX = \ '^[a-zA-z0-9_][a-zA-Z0-9-_]*(/?[a-zA-z0-9_][a-zA-Z0-9-_]+)?$' +def file_virus_validator(form, field): + if not pagure.APP.config['VIRUS_SCAN_ATTACHMENTS']: + return + from pyclamd import ClamdUnixSocket + + if not field.name in flask.request.files or \ + flask.request.files[field.name].filename == '': + # If no file was uploaded, this field is correct + return + uploaded = flask.request.files[field.name] + clam = ClamdUnixSocket() + if not clam.ping(): + raise wtforms.ValidationError('Unable to communicate with virus scanner') + results = clam.scan_stream(uploaded.stream.read()) + if results is None: + uploaded.stream.seek(0) + return + else: + result = results.values() + res_type, res_msg = result + if res_type == 'FOUND': + raise wtforms.ValidationError('Virus found: %s' % res_msg) + else: + raise wtforms.ValidationError('Error scanning uploaded file') + + class ProjectFormSimplified(wtf.Form): ''' Form to edit the description of a project. ''' description = wtforms.TextField( @@ -290,7 +320,7 @@ class UploadFileForm(wtf.Form): ''' Form to upload a file. ''' filestream = wtforms.FileField( 'File', - [wtforms.validators.Required()]) + [wtforms.validators.Required(), file_virus_validator]) class UserEmailForm(wtf.Form): @@ -326,7 +356,7 @@ class CommentForm(wtf.Form): ''' Form to upload a file. ''' comment = wtforms.FileField( 'Comment', - [wtforms.validators.Required()]) + [wtforms.validators.Required(), file_virus_validator]) class NewGroupForm(wtf.Form): diff --git a/requirements.txt b/requirements.txt index 7f97847..9e6ad4c 100644 --- a/requirements.txt +++ b/requirements.txt @@ -16,6 +16,7 @@ markdown munch Pillow psutil +pyclamd # This is only needed if VIRUS_SCAN_ATTACHMENTS is enabled pygit2 >= 0.20.1 pygments python-openid diff --git a/tests/test_pagure_flask_ui_issues.py b/tests/test_pagure_flask_ui_issues.py index e9d5366..b7c4b00 100644 --- a/tests/test_pagure_flask_ui_issues.py +++ b/tests/test_pagure_flask_ui_issues.py @@ -16,6 +16,8 @@ import unittest import shutil import sys import os +import pyclamd +import tempfile import pygit2 from mock import patch @@ -35,6 +37,7 @@ class PagureFlaskIssuestests(tests.Modeltests): super(PagureFlaskIssuestests, self).setUp() pagure.APP.config['TESTING'] = True + pagure.APP.config['VIRUS_SCAN_ATTACHMENTS'] = True pagure.SESSION = self.session pagure.ui.SESSION = self.session pagure.ui.app.SESSION = self.session @@ -1084,6 +1087,26 @@ class PagureFlaskIssuestests(tests.Modeltests): exp = {'output': 'notok'} self.assertDictEqual(json_data, exp) + # Try to attach a virus + with tempfile.NamedTemporaryFile() as eicarfile: + eicarfile.write(pyclamd.ClamdUnixSocket().EICAR()) + eicarfile.flush() + stream = open(eicarfile.name, 'rb') + data = { + 'csrf_token': csrf_token, + 'filestream': stream, + 'enctype': 'multipart/form-data', + } + output = self.app.post( + '/test/issue/1/upload', data=data, follow_redirects=True) + self.assertEqual(output.status_code, 200) + stream.close() + json_data = json.loads(output.data) + exp = { + 'output': 'notok', + } + self.assertDictEqual(json_data, exp) + # Attach a file to a ticket stream = open(os.path.join(tests.HERE, 'placebo.png'), 'rb') data = { From af5d13eb272a9476a667e5c2d747c93267c6f2f5 Mon Sep 17 00:00:00 2001 From: Pierre-Yves Chibon Date: Aug 10 2016 21:05:08 +0000 Subject: [PATCH 69/69] Misc fixes to the doc, formatting and links --- diff --git a/doc/install.rst b/doc/install.rst index 74ccb81..b5dfb38 100644 --- a/doc/install.rst +++ b/doc/install.rst @@ -254,9 +254,10 @@ where the file ``alembic.ini`` is. This file contains two important pieces of information: * ``sqlalchemy.url`` which is the URL used to connect to the database, likely -the same URL as the one in ``pagure.cfg``. + the same URL as the one in ``pagure.cfg``. + * ``script_location`` which is the path to the ``versions`` folder containing -all the alembic migration files. + all the alembic migration files. The ``alembic stamp`` command is the one actually saving the current revision into the database. This current revision is found using ``alembic heads`` diff --git a/doc/usage/roadmap.rst b/doc/usage/roadmap.rst index e1b7cd5..8f4c400 100644 --- a/doc/usage/roadmap.rst +++ b/doc/usage/roadmap.rst @@ -7,10 +7,12 @@ their tags. The principal is as follow: * All the ticket with the tag ``roadmap`` will show up on the roadmap page. + * For each milestones defined in the settings of the project, the roadmap -will group tickets with the corresponding tag. + will group tickets with the corresponding tag. + * Tickets with the tag ``roadmap`` that are not associated with any of the -milestones defined in the settings are group in an ``unplanned`` section. + milestones defined in the settings are group in an ``unplanned`` section. Example @@ -21,12 +23,12 @@ For a project named ``test`` on ``pagure.io``. * First, go to the settings page of the project, create the milestones you -like, for example: ``v1.0`` and ``v2.0``. + like, for example: ``v1.0`` and ``v2.0``. * For the tickets you want to be on these milestones, go through each of them -and add them the tags: ``roadmap`` in combination with the milestone you want -``v1.0`` or ``v2.0``, or none of them if the ticket is on the roadmap but -not assigned to any milestones. + and add them the tags: ``roadmap`` in combination with the milestone you want + ``v1.0`` or ``v2.0``, or none of them if the ticket is on the roadmap but + not assigned to any milestones. * And this is how it will look like diff --git a/doc/usage/theming.rst b/doc/usage/theming.rst index 961b4ab..3e7b78e 100644 --- a/doc/usage/theming.rst +++ b/doc/usage/theming.rst @@ -78,4 +78,4 @@ flask here, ie: search in one folder and if you don't find the file look in another one. `An example apache configuration `_ -is provided as part of the sources of `flask-multistatic `_. +is provided as part of the sources of `flask-multistatic`_.