Giter Club home page Giter Club logo

bugbot's Introduction

This tool is used by Mozilla release management to send emails to the Firefox developers. It will query the bugzilla.mozilla.org database and send emails to Mozilla developers and their managers (if Mozilla staff).

The tool will also notify release managers about potential issues in bugzilla and autofix some categories of issues.

The list of checkers is documented on the Mozilla wiki: https://wiki.mozilla.org/BugBot

This package currently uses Mozilla's Bugzilla REST API, and the Mozilla IAM phonebook (to access bug assignees' managers & Mozilla email addresses).

Installation

  1. Check out the code:

    git clone https://github.com/mozilla/bugbot.git
    
  2. (optional) Create your virtualenv using virtualenvwrapper:

    virtualenv -p python3 venv
    source venv/bin/activate
    
  3. Install the dependencies for Python 3 too:

    pip install -r requirements.txt
    

Auto-formatting with pre-commit

This project uses pre-commit.

  1. Install test dependencies, if not already installed:

    pip install -r requirements-test.txt
    
  2. Set up the git pre-commit hooks in your clone:

    pre-commit install
    

Every time you try to commit, pre-commit checks your files to ensure they follow our style standards and aren't affected by some simple issues. If the checks fail, pre-commit won't let you commit.

Running the Bot Rules

Before running:

  1. The LDAP + SMTP infos are used to send emails
  2. Need to generate an API key from bugzilla admin ( https://bugzilla.mozilla.org/userprefs.cgi?tab=apikey )
  3. Should generate an API key from Phabricator ( https://phabricator.services.mozilla.com/settings/user )
  4. The IAM secrets are used to generate a dump of phonebook, which is required for some scripts (employees can request them by filing a bug in the SSO: Requests component )
# in configs/config.json
{
  "ldap_username": "[email protected]",
  "ldap_password": "xxxxxxxxxxxxxx",
  "smtp_server": "smtp.xxx.xxx",
  "smtp_port": 314,
  "smtp_ssl": true,
  "bz_api_key": "xxxxxxxxxxxxxx",
  "bz_api_key_nomail": "xxxxxxxxxxxxxx",
  "phab_api_key": "xxxxxxxxxxxxxx",
  "iam_client_secret": "xxxxxxxxxxxxxx",
  "iam_client_id": "xxxxxxxxxxxxxx",
  "socorro_token": "xxxxxxxxxxxxxx"
}
Do a dryrun::
python -m bugbot.rules.stalled

There is a ton of rules in bugbot/rules/ so you should be able to find some good examples.

Setting up 'Round Robin' triage rotations

One use case for this tool is managing triage of multiple components across a team of multiple people.

To set up a new Round Robin rotation, a manager or team lead should create a calendar with the rotation of triagers and add a link to the rotation calendar in the triage rotations spreadsheet.

Running on a server

This needs to run on a private server because it will have login for smtp and bugzilla key so it can't currently be shared access.

Cronjob:

CRON_DIR=/path/to/repository
00 12  * * 1-5 cd $CRON_DIR ; ./cron_run_weekdays.sh &> /tmp/bugbot-weekdays.log
00 8   * * *   cd $CRON_DIR ; ./cron_run_daily.sh    &> /tmp/bugbot-daily.log
40 */1 * * *   cd $CRON_DIR ; ./cron_run_hourly.sh   &> /tmp/bugbot-hourly.log

We run hourly jobs at minute 40 past every hour to avoid overlap with daily jobs.

bugbot's People

Contributors

anoopvalluthadam avatar anubnair avatar archaeopteryx avatar bbajaj avatar calixteman avatar davehunt avatar daviddahl avatar dependabot-preview[bot] avatar dependabot[bot] avatar emceeaich avatar globau avatar hsinyi avatar jcristau avatar jensstutte avatar legneato avatar lothiraldan avatar lsblakk avatar marco-c avatar mozfreddyb avatar mozilla-github-standards avatar nt1m avatar padenot avatar pascalchevrel avatar promisefru avatar pyup-bot avatar suhaibmujahid avatar svoisen avatar sylvestre avatar tomrittervg avatar tublitzed avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

bugbot's Issues

auto_nag/scripts/rm_query_creator.py auto_nag/scripts/query_creator.py contain duplicate code

The following functions are duplicated.
We should move that into a common place:

def createQuery(queries_dir, title, short_title, url, show_summary):
    file_name = queries_dir + str(datetime.date.today()) + '_' + short_title
    if not os.path.exists(queries_dir):
        os.makedirs(queries_dir)
    qf = open(file_name, 'w')
    qf.write("query_name = \'" + title + "\'\n")
    qf.write("query_url = \'" + url + "\'\n")
    qf.write("show_summary = \'" + str(show_summary) + "\'\n")
    return file_name


def createQueriesList(queries_dir, weekday, print_all):
    queries = []
    for url in urls:
        if weekday >= 0 and weekday < 5 and url[0] == 5:
            queries.append(createQuery(queries_dir, title=url[1][0], short_title=url[1][1], url=url[1][2], show_summary=url[1][3]))
        if weekday == 0 and url[0] == 0:
            queries.append(createQuery(queries_dir, title=url[1][0], short_title=url[1][1], url=url[1][2], show_summary=url[1][3]))
        if weekday == 3 and url[0] == 3:
            queries.append(createQuery(queries_dir, title=url[1][0], short_title=url[1][1], url=url[1][2], show_summary=url[1][3]))
    print queries
    return queries


def cleanUp(queries_dir):
    try:
        for file in os.listdir(queries_dir):
            if file.startswith(str(datetime.date.today())):
                os.remove(os.path.join(queries_dir, file))
        return True
    except Exception as error:
        print "Error: ", str(error)
        return False

In query_creator.py, the last arg of urls = (0) seems useless

In all of the cases, the last value is 0, we should figure out if this is necessary or not.

urls = [
(5, ["Unlanded Beta " + beta_version + " Bugs", "unlanded_beta", unlanded_beta_url, 0]),
(5, ["Unlanded Aurora " + aurora_version + " Bugs", "unlanded_aurora", unlanded_aurora_url, 0]),
(5, ["Unlanded ESR38 Bugs", "unlanded_esr38", unlanded_esr38_url, 0]),
(5, ["Tracked or Nominated for Tracking with Need-Info? Beta " + beta_version + " Bugs", "needinfo_beta", needinfo_beta_url, 0]),

Port to Python Requests

We should consider porting the urllib and urllib2 code to python requests perhaps this could be a contribution opportunity or good first bug.

Cannot call setup.py without remoteobjects already installed

setup.py shouldn't depend on remoteobjects already being installed, but instead allow installing dependencies.

Traceback (most recent call last):
  File "setup.py", line 12, in <module>
    version=__import__('bugzilla').__version__,
  File "/Users/kewisch/mozilla/github/bztools/bugzilla/__init__.py", line 4, in <module>
    from remoteobjects import http
ImportError: No module named remoteobjects

Fix the flake8 warnings

Flake8 is finding a bunch of issues wrt the python code. We should fix them: (but we don't care about the max line length check)

$ flake8 --max-line-length=10800  scripts/ bugzilla/ templates/
scripts/attach.py:8:1: F401 'Bug' imported but unused
scripts/attach.py:10:1: F401 'qs' imported but unused
scripts/attach.py:14:1: E302 expected 2 blank lines, found 1
scripts/attach.py:35:16: E128 continuation line under-indented for visual indent
scripts/attach.py:91:1: E302 expected 2 blank lines, found 1
scripts/automated_release_emails.py:23:9: E128 continuation line under-indented for visual indent
scripts/automated_release_emails.py:24:9: E128 continuation line under-indented for visual indent
scripts/automated_release_emails.py:25:9: E128 continuation line under-indented for visual indent
scripts/automated_release_emails.py:26:9: E128 continuation line under-indented for visual indent
scripts/automated_release_emails.py:27:9: E128 continuation line under-indented for visual indent
scripts/automated_release_emails.py:28:9: E128 continuation line under-indented for visual indent
scripts/b2g_query_creator.py:35:22: E261 at least two spaces before inline comment
scripts/b2g_query_creator.py:87:127: E231 missing whitespace after ','
scripts/b2g_query_creator.py:88:125: E231 missing whitespace after ','
scripts/b2g_query_creator.py:89:118: E231 missing whitespace after ','
scripts/b2g_query_creator.py:90:116: E231 missing whitespace after ','
scripts/b2g_query_creator.py:91:119: E231 missing whitespace after ','
scripts/b2g_query_creator.py:92:117: E231 missing whitespace after ','
scripts/b2g_query_creator.py:93:134: E231 missing whitespace after ','
scripts/b2g_query_creator.py:94:132: E231 missing whitespace after ','
scripts/email_nag.py:77:35: W601 .has_key() is deprecated, use 'in'
scripts/email_nag.py:88:31: W601 .has_key() is deprecated, use 'in'
scripts/email_nag.py:92:34: W601 .has_key() is deprecated, use 'in'
scripts/email_nag.py:100:25: E126 continuation line over-indented for hanging indent
scripts/email_nag.py:126:5: E303 too many blank lines (2)
scripts/email_nag.py:142:9: E128 continuation line under-indented for visual indent
scripts/email_nag.py:143:9: E128 continuation line under-indented for visual indent
scripts/email_nag.py:144:9: E128 continuation line under-indented for visual indent
scripts/email_nag.py:145:9: E128 continuation line under-indented for visual indent
scripts/email_nag.py:146:9: E128 continuation line under-indented for visual indent
scripts/email_nag.py:150:19: E231 missing whitespace after ','
scripts/email_nag.py:153:21: E231 missing whitespace after ','
scripts/email_nag.py:153:25: E231 missing whitespace after ','
scripts/email_nag.py:153:34: E231 missing whitespace after ','
scripts/email_nag.py:153:43: E231 missing whitespace after ','
scripts/email_nag.py:162:33: E231 missing whitespace after ','
scripts/email_nag.py:183:13: E128 continuation line under-indented for visual indent
scripts/email_nag.py:185:13: E128 continuation line under-indented for visual indent
scripts/email_nag.py:187:13: E128 continuation line under-indented for visual indent
scripts/email_nag.py:189:13: E128 continuation line under-indented for visual indent
scripts/email_nag.py:190:13: E128 continuation line under-indented for visual indent
scripts/email_nag.py:192:13: E128 continuation line under-indented for visual indent
scripts/email_nag.py:193:13: E128 continuation line under-indented for visual indent
scripts/email_nag.py:195:13: E128 continuation line under-indented for visual indent
scripts/email_nag.py:196:13: E128 continuation line under-indented for visual indent
scripts/email_nag.py:197:13: E128 continuation line under-indented for visual indent
scripts/email_nag.py:199:13: E128 continuation line under-indented for visual indent
scripts/email_nag.py:200:13: E128 continuation line under-indented for visual indent
scripts/email_nag.py:202:13: E128 continuation line under-indented for visual indent
scripts/email_nag.py:203:13: E128 continuation line under-indented for visual indent
scripts/email_nag.py:205:13: E128 continuation line under-indented for visual indent
scripts/email_nag.py:207:13: E128 continuation line under-indented for visual indent
scripts/email_nag.py:209:13: E128 continuation line under-indented for visual indent
scripts/email_nag.py:211:13: E128 continuation line under-indented for visual indent
scripts/email_nag.py:213:13: E128 continuation line under-indented for visual indent
scripts/email_nag.py:242:37: W601 .has_key() is deprecated, use 'in'
scripts/email_nag.py:249:20: W601 .has_key() is deprecated, use 'in'
scripts/email_nag.py:252:20: W601 .has_key() is deprecated, use 'in'
scripts/email_nag.py:255:22: W601 .has_key() is deprecated, use 'in'
scripts/email_nag.py:258:17: E265 block comment should start with '# '
scripts/email_nag.py:277:24: W601 .has_key() is deprecated, use 'in'
scripts/email_nag.py:280:21: E126 continuation line over-indented for hanging indent
scripts/email_nag.py:280:26: E203 whitespace before ':'
scripts/email_nag.py:285:17: E121 continuation line under-indented for hanging indent
scripts/email_nag.py:287:35: W601 .has_key() is deprecated, use 'in'
scripts/email_nag.py:288:50: W601 .has_key() is deprecated, use 'in'
scripts/email_nag.py:296:64: E203 whitespace before ','
scripts/email_nag.py:304:21: E126 continuation line over-indented for hanging indent
scripts/email_nag.py:304:26: E203 whitespace before ':'
scripts/email_nag.py:308:22: E121 continuation line under-indented for hanging indent
scripts/email_nag.py:309:17: E121 continuation line under-indented for hanging indent
scripts/email_nag.py:323:43: W601 .has_key() is deprecated, use 'in'
scripts/email_nag.py:333:34: W601 .has_key() is deprecated, use 'in'
scripts/email_nag.py:368:44: W601 .has_key() is deprecated, use 'in'
scripts/email_nag.py:373:46: W601 .has_key() is deprecated, use 'in'
scripts/email_nag.py:375:52: W601 .has_key() is deprecated, use 'in'
scripts/email_nag.py:377:58: W601 .has_key() is deprecated, use 'in'
scripts/email_nag.py:379:56: W601 .has_key() is deprecated, use 'in'
scripts/email_nag.py:388:61: W601 .has_key() is deprecated, use 'in'
scripts/email_nag.py:391:60: W601 .has_key() is deprecated, use 'in'
scripts/email_nag.py:403:21: E126 continuation line over-indented for hanging indent
scripts/email_nag.py:407:27: E251 unexpected spaces around keyword / parameter equals
scripts/email_nag.py:407:29: E251 unexpected spaces around keyword / parameter equals
scripts/email_nag.py:408:32: E251 unexpected spaces around keyword / parameter equals
scripts/email_nag.py:408:34: E251 unexpected spaces around keyword / parameter equals
scripts/email_nag.py:419:20: W601 .has_key() is deprecated, use 'in'
scripts/email_nag.py:477:21: E128 continuation line under-indented for visual indent
scripts/email_nag.py:478:21: E128 continuation line under-indented for visual indent
scripts/email_nag.py:479:21: E128 continuation line under-indented for visual indent
scripts/email_nag.py:480:21: E128 continuation line under-indented for visual indent
bugzilla/__init__.py:1:20: E231 missing whitespace after ','
bugzilla/__init__.py:1:28: E231 missing whitespace after ','
bugzilla/agents.py:1:1: F403 'from bugzilla.models import *' used; unable to detect undefined names
bugzilla/agents.py:2:1: F403 'from bugzilla.utils import *' used; unable to detect undefined names
bugzilla/agents.py:4:1: E302 expected 2 blank lines, found 1
bugzilla/agents.py:10:1: E302 expected 2 blank lines, found 1
bugzilla/agents.py:12:1: W293 blank line contains whitespace
bugzilla/models.py:33:1: E302 expected 2 blank lines, found 1
bugzilla/models.py:233:1: W293 blank line contains whitespace
bugzilla/models.py:234:45: W292 no newline at end of file
bugzilla/utils.py:17:10: E231 missing whitespace after ','
bugzilla/utils.py:21:47: E231 missing whitespace after ','
bugzilla/utils.py:32:50: W291 trailing whitespace
bugzilla/utils.py:54:25: E221 multiple spaces before operator

Switch to the new travis infra

This job is running on container-based infrastructure, which does not allow use of 'sudo', setuid and setguid executables.
If you require sudo, add 'sudo: required' to your .travis.yml
See http://docs.travis-ci.com/user/workers/container-based-infrastructure/ for details.

As travis suggests, we should move to the new infra.

Check sec-approval status

When a bug has a patch and is affected the developer gets a nag email. Sometimes, that's a sec bug, and it is either waiting for sec-approval, or it has sec-approval but for a particular date in the future, often a couple of weeks after a release cycle starts. That means a developer might get daily nag emails on a security bug for several weeks, annoyingly and pointlessly.

Kind of a corner case here but for the devs who fix a lot of security issues, maybe we could fix this by parsing the whiteboard field and looking for something that looks like a check-in date. Example: [checkin on 6/26] . But it should not detect "checkin-needed".

Issue on the merge day time frame

Traceback (most recent call last):
  File "/home/sylvestre/dev/mozilla/relman-auto-nag/auto_nag/scripts/email_nag.py", line 420, in <module>
    cc_only=options.cc_only)
  File "/home/sylvestre/dev/mozilla/relman-auto-nag/auto_nag/scripts/email_nag.py", line 105, in generateEmailOutput
    'affected': bug.get_fx_affected_versions()
  File "/home/sylvestre/dev/mozilla/relman-auto-nag/auto_nag/bugzilla/models.py", line 124, in get_fx_affected_versions
    if self.api_data['cf_status_firefox' + version] == 'affected':
KeyError: u'cf_status_firefox59'

I guess it is because fx 59 flags have been removed too early
https://bugzilla.mozilla.org/show_bug.cgi?id=1459643

bugzilla/ & scripts/ should be moved into a common directory

To follow what is usually done in the Python world, we should create a new directory
auto-nag/
and move bugzilla/ & scripts/ into it.

It will probably require some changes in import and so.
Hopefully, this will also simplifies tests and code coverage executions.

Refactor the scripts to share the common code

In https://github.com/mozilla/relman-auto-nag/tree/master/auto_nag/scripts, we have a bunch of scripts.
Because @calixteman and myself are lazy, we have been duplicating the code.
Now that we are using it more and more, we shouldn't be doing that anymore.

For example, https://github.com/mozilla/relman-auto-nag/blob/master/auto_nag/scripts/no_assignee.py#L118
get_email is duplicated, ditto for send_email, get_login_info, etc

We should create a common file providing these functions with argument to manage the differences (mostly strings it seems)

Tracking nominations which need attention

Release management should have alerts for bugs which are nominated for tracking which don’t have a decision after N days. N=0 for release (immediate alerting) N=0 for beta when we’re close to a release, but N=5 when there is not a release imminent.

Improve the display of the "release manager feedback"

Currently, we have:

== Notify release managers when bugs are marked fixed in nightly but still affected for aurora, beta or release ==
* https://bugzil.la/1116036 -  - (assigned to Mark Finkle (:mfinkle))
* https://bugzil.la/1192910 -  - (assigned to Kartikaya Gupta (email:[email protected]))

We should have

* https://bugzil.la/1116036 -  - (assigned to Mark Finkle (:mfinkle))
Fixed 48, affected: 45, 46 and 47

So that release manager have a quick understanding of the issues affecting the releases.

BzAPI server on which your software depends is going away

Hi,

This bug has been filed because a code search has revealed that code in this repository is using "BzAPI", the Bugzilla API proxy server at https://api-dev.bugzilla.mozilla.org/.

This service is due to be decommissioned. Fortunately, Bugzilla has a new native REST API which works somewhat similarly that you could use instead:
https://wiki.mozilla.org/Bugzilla:REST_API

Moving to that new API would require some porting work. If that seems like too much effort, fear not, because there is also a BzAPI compatibility layer built on top of the new API. Because it's compatible, using that should simply require a change of API endpoint.

Change:
https://api-dev.bugzilla.mozilla.org/latest/
(or https://api-dev.bugzilla.mozilla.org/1.3/)
to
https://bugzilla.mozilla.org/bzapi/

You should not expect the api-dev server to continue to work after Friday 14th November. So please test and migrate your production instance(s) over before then, either to the BzAPI compatibility endpoint or to the new native REST API.

Thanks for using BzAPI. It's been a great ride :-)

Gerv

Sending mail is broken

Traceback (most recent call last):
  File "/home/sylvestre/dev/mozilla/relman-auto-nag/auto_nag/scripts/email_nag.py", line 451, in <module>
    dryrun=options.dryrun)
  File "/home/sylvestre/dev/mozilla/relman-auto-nag/auto_nag/mail.py", line 61, in sendMail
    mailserver = smtplib.SMTP(SMTP, PORT)
  File "/usr/lib/python2.7/smtplib.py", line 256, in __init__
    (code, msg) = self.connect(host, port)
  File "/usr/lib/python2.7/smtplib.py", line 318, in connect
    (code, msg) = self.getreply()
  File "/usr/lib/python2.7/smtplib.py", line 369, in getreply
    raise SMTPServerDisconnected("Connection unexpectedly closed")
smtplib.SMTPServerDisconnected: Connection unexpectedly closed

git revert 9314a38
fixed my issue

Port to new native REST API

Per announcement from Gerv below we should consider porting bztools to the new native REST API in the future but perhaps this would be a good first bug for a contributor.

"This service is due to be decommissioned. Fortunately, Bugzilla has a new native REST API which works somewhat similarly that you could use instead:
https://wiki.mozilla.org/Bugzilla:REST_API

Moving to that new API would require some porting work. If that seems like too much effort, fear not, because there is also a BzAPI compatibility layer built on top of the new API. Because it's compatible, using that should simply require a change of API endpoint.

Change:
https://api-dev.bugzilla.mozilla.org/latest/
(or https://api-dev.bugzilla.mozilla.org/1.3/)
to
https://bugzilla.mozilla.org/bzapi/"

Seems that bztools does not differenciate the two ESR releases

In bug 1180419, Marco was notified with this information:

== Bugs Tracked for ESR31 ==
* https://bugzil.la/1180419 -   - (assigned to Marco Bonardo [::mak])

However, the bug was only affecting ESR38 and never ESR31. I guess this tool does not handle correctly the period when we have two ESR releases running in parallel.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.