Repo in question is: https://github.com/mozilla-releng/pushsnapscript

Where to start:

Identify who are the core contributors of this project and get in touch with them via e-mail, Irc or GitHub.
Define a list of files from this repository which affect production.
Learn how to identify what has actually been deployed.

Repo in question is: https://github.com/mozilla-releng/bouncerscript

Where to start:

Identify who are the core contributors of this project and get in touch with them via e-mail, Irc or GitHub.
Define a list of files from this repository which affect production.
Learn how to identify what has actually been deployed.

Repo in question is: https://github.com/mozilla-releng/signingscript

Where to start:

Identify who are the core contributors of this project and get in touch with them via e-mail, Irc or GitHub.
Define a list of files from this repository which affect production.
Learn how to identify what has actually been deployed.

Currently the date in changelog.json is displayed like : 2018-10-29T22:40:41Z and commit numbers are nothing but "upside-down". We want date to be displayed without those "T" and "Z".

Repo in question is: https://github.com/mozilla-releng/signtool

Where to start:

Identify who are the core contributors of this project and get in touch with them via e-mail, Irc or GitHub.
Define a list of files from this repository which affect production.
Learn how to identify what has actually been deployed.

def filter_commit_data(commit) currently breaks generated markdown file.
Will modify the function for a workaround.

Example:

SHIP-IT MARKDOWN TABLE

The downloads page on thunderbird.net uses thunderbird_primary_builds.json

to generate the download list. There's some locales included there

that have been removed from Thunderbird in bug 1310221. This patch

brings thunderbirddetails.py (mostly) inline with all-locales in

comm/mail/locales.|2018-10-29T22:30:04Z
|3|Rail Aliiev|Use raw string for regexes (#242)|2018-10-29T22:22:50Z
|4|Pascal Chevrel|Nightly 65: Merge and deply when 65 builds are available for all locales (#238)|2018-10-23T09:27:10Z
|5|Nick Thomas|Use N most recent releases only (#237)|2018-10-17T20:10:15Z
|6|Pascal Chevrel|Update false Fennec instructions in template (#236)|2018-09-10T16:20:59Z
|7|Patrick Cloke|Bump Thunderbird nightly version to 64. (#234)|2018-09-05T14:40:36Z
|8|Pascal Chevrel|Update available firefox nightly version to 64. (#233)|2018-09-05T14:28:27Z
|9|Julien Cristau|Bug 1460752 - bump CURRENT_ESR to 60 (#235)|2018-09-05T13:27:57Z
|10|Rail Aliiev|Reformat, add more info (#232)|2018-07-13T20:49:08Z
|11|Rail Aliiev|Handle "legacy" releases properly, kill python2.6 (#231)

Repo in question is: https://github.com/mozilla-releng/shipitscript

Where to start:

Identify who are the core contributors of this project and get in touch with them via e-mail, Irc or GitHub.
Define a list of files from this repository which affect production.
Learn how to identify what has actually been deployed.

We will need to create a list of all mercurial repositories that are being used for firefox-ci and/or could influence the infra.

A good place to start would be our team's Google Drive Escalation Doc.

Please post in the following format:
<Name of repo> - <Link to mercurial repo>

After the merge in PR be7b8fc was done, the code brakes with "NameError: name 'url' is not defined"

To Reproduce
Simply run the script and something like
Traceback (most recent call last): File "E:/bcrisan/repos/fic2/client.py", line 187, in <module> create_md_table(repository_name) File "E:/bcrisan/repos/fic2/client.py", line 41, in create_md_table "|" + date + '\n' NameError: name 'url' is not defined
should pop out.

Expected behavior
The script should run and create the md/json files in repositories.

Desktop (please complete the following information):

• OS: Windows
• Browser: not applicable
• Version: master branch (https://github.com/Akhliskun/firefox-infra-changelog)

Additional context
Not required.

Short description:

Git and HG returns the date and time in different ways.
eg. Unix Timestamp from HG
To not be confusing we have to use a single date/time format.

Where to start:

We could start deciding what date/time format to use. For example the most popular date/time format is the following: YYYY-MM-DD Thh:mm:ss (eg 2018-11-02 T12:10:30) (follow the order of largest to smallest units)
-where:
YYYY = four-digit year
MM = two-digit month (01=January, etc.)
DD = two-digit day of month (01 through 31)
hh = two digits of hour (00 through 23) (am/pm NOT allowed)
mm = two digits of minute (00 through 59)
ss = two digits of second (00 through 59)
-also we can add:
TZD = time zone designator (Z or +hh:mm or -hh:mm)
Z = UTC (Coordinated Universal Time)

Requirements:

• Don't use classes
• Runs on Py3.X
• Document the code

This issue will track all the progress/bugs/requests and ToDos.
Fell free to use this Issue to discuses about steps, plans and ideas on what should go into the first release of FF-Infra

Deadline:

5th November 2018

What needs to be done: JIRA-like Board View

• Initial issues, linked to milestone and with tags.
• Github Login/Auth integration + examples + documentation.
• HG wrapper/auth integration
• JSON changelog, iteration 1 completed.
• MarkDown changelog, iteration 1 completed.
• Start to End, working prototype. (We get 2 changelogs, 1 .json 1 .md)
• 5th Meeting with Jordan + Notes completed.

Repo in question is: https://github.com/mozilla-releng/treescript

Where to start:

Identify who are the core contributors of this project and get in touch with them via e-mail, Irc or GitHub.
Define a list of files from this repository which affect production.
Learn how to identify what has actually been deployed.

Short description
We can figure out if a change affects CI by checking if this change happened in one of the CI files.

First solution
After we have a list with all the CI files we can check if the affected file is in our list, if it is we can save those changes if not we can ignore them.

First step
We can start to create a list with all the CI files

Proposal

1. Should we make a list for each repository separately or all in one? My thought is to have one separately for each repository, so we'll can compare the affected files from current repository with his CI-files list instead with a very large list.

Noteworthy:

Keeping the code clean and well document. This will allow us to easier read and understand the code, but also will help any new or returning people who are going to work on this to better understand what has been done thus far.

Good Practice:

I'm suggesting to attach this issue to the milestone with the closest deadline so we always do a tidiness-check before each deadline.

We want to use the last time when we run the script, and then to use it for getting the changes since that date-time.

Where to start:

We have to figure out how to save the date-time format into a file and then to use it to get the changes since that date.

Other Information

I have tried to save it as a string but it does not work. The script returns an error at the following line:

.get_commits(since=lastRun):

As the title suggest , I've added a new column named Commit Url. That column refers to hyperlink to each commit stored in "URL" string.

Describe the bug
https://github.com/Akhliskun/firefox-infra-changelog/blob/zfay_dev/hg_files/autoland.md
As shown in the link , at the 14 and 3 commits the table is broken, because the commit message contains the char " | "

What to do:

Bring the formatting of Roland's MD files to the master branch. These MD files should contain commit information such as: repo, commit message, ,link of the commit, and date.

Proposal:

Should we add the name/e-mail of the committer?

Repo in question is: https://github.com/mozilla-releng/pushapkscript

Where to start:

Identify who are the core contributors of this project and get in touch with them via e-mail, Irc or GitHub.
Define a list of files from this repository which affect production.
Learn how to identify what has actually been deployed.

Using tests/json_changelog.py create a test script that will read the data stored in the following two json files: git_changelog.json , hg_changelog.json and add the aggregated and formatted data to changelog.json

Where to start:

I have read this article JSON crash course that explains short words how json looks and how we can access the data inside using Python. I think this is a very good starting point, as it will give us a good basic understanding of basic concepts, which will help us come up with more implementation specific questions that Moz devs can help us with.

How data should be formatted:

Using a copy/paste of an example from the article linked above:

{
    "text key": "string",
    "integer key": 123,
    "boolean key": true,
    "simple list": [
        "value 1",
        "value 2"
    ],
    "dict": {
        "element 1": "value 1",
        "element 2": "value 2"
    },
    "dict list": [
        {
            "key 1.1": "value 1.1",
            "key 1.2": "value 1.2"
        },
        {
            "key 2.1": "value 2.1",
            "key 2.2": "value 2.2"
        }
    ]
}

Requirements

• Don't use classes
• Runs on Py3.X
• Document the code
• Data needs to be easily to work with, clear and concise keys names.

Challenges:

The difficult part, as I see it, is that we need to be able to sort the data with expressions like:

• list all changes made by user X
• list all changes that landed on both M-C and M-B
• list all changes in the last 24 hours for M-C

While most of this will be done via filter logic, to be able to work with clean code, we need a good json structure that is intuitive to anyone, including devs that might not know all the fine details.

Please ask questions or post findings/ideas/theories/proofs of concept in this issue.

Using tests/md_changelog.py create a test script that will read the data stored in the following two json files: git_changelog.json , hg_changelog.json and add the aggregated and formatted data to changelog.md

How data should be formatted:

Github

Mercurial

Requirements

• Don't use classes
• Runs on Py3.X
• Document the code

As PullRequest #28 was already merged, and it has a few issues, we will need to come back to this and fix all the issues listed in the PR.

Please push the fixed in a stand alone PR, without other content, just the fixes.

After the PR #85 the json files format for git repositories has been changed and introduces new issues regarding the generation of the main markdown table.

The issues is introduced by the first json element that now stores the last checked date.

Steps to reproduce the behavior:
Run the script
Check the main markdown table file.

Actual Result
Only the changes from the mercurial files are updated.

Expected behavior
The main markdown table should be updated with the latest changes on the git files and hg files.

Additional context
This appeared after structural modification of the git json files.

Using tests/git_api.py create a test script that is able to pull changelog (commits) from any of the Mozilla RelEng repositories.

A good place to start

https://developer.github.com/webhooks/

Requirements

• Don't use classes
• Runs on Py3.X
• Stores the data pulled from github in a JSON file named github_changelog.json
• Document the code

Using tests/hg_api.py create a test script that is able to pull changelog (commits) from any of the Mozilla HG repositories.

A good place to start

Talk to devs in #vcs about how we can grab the data that we need.
Don't forget to ask how we should handle private repos.

Requirements

• Don't use classes
• Runs on Py3.X
• Stores the data pulled from hgmo in a JSON file named hg_changelog.json
• Document the code

Repo in question is: https://github.com/mozilla-releng/beetmoverscript

Where to start:

Identify who are the core contributors of this project and get in touch with them via e-mail, Irc or GitHub.
Define a list of files from this repository which affect production.
Learn how to identify what has actually been deployed.

Repo in question is: https://github.com/mozilla-releng/addonscript

Where to start:

Identify who are the core contributors of this project and get in touch with them via e-mail, Irc or GitHub.
Define a list of files from this repository which affect production.
Learn how to identify what has actually been deployed.

Repo in question is: https://github.com/mozilla-releng/mozapkpublisher

Where to start:

Identify who are the core contributors of this project and get in touch with them via e-mail, Irc or GitHub.
Define a list of files from this repository which affect production.
Learn how to identify what has actually been deployed.

General idea:

Idea to have a list with all the information we need to save from each changes/commits.

Where to start:

We can start to create a few templates which contain the information about changes.
We have to find out which information would be useful to save.

Requirements

• Don't use classes
• Runs on Py3.X
• Document the code

As per Jordan's request, I will sync up with him on 6th and get the project moved.

Everyone will have to update the upstream link on your local repository clones.

Define a json file that stores all of the git and hg repositories in a single place

My solution to this problem would be a json file that would look something like this:

{
  "HG": {
    "Repo_HG1": "link1",
    "Repo_HG2": "link2",
    "Repo_HG3": "link3"
  },
  "Git": {
    "Repo_GIT1": "link1",
    "Repo_GIT2": "link2",
    "Repo_GIT3": "link3"
  }
}

Another option would be using a convertor, but most of them needs to be installed and kept up to date.

Also a big advantage would be the simplicity that would give in case we want to store another data in those repositories, e.g specific files to check.

{
  "HG": {
    "Repo_HG1": {
      "link": "https://example.com",
      "important_file": "https://example.com/important",
      "file_to_ignore": "https://example.com/ignore"
    },
    "Repo_HG2": "link2",
    "Repo_HG3": "link3"
  },
  "Git": {
    "Repo_GIT1": "link1",
    "Repo_GIT2": "link2",
    "Repo_GIT3": "link3"
  }
}

Added two repos, which have the highest potential to cause bustage:
taskcluster-auth : https://github.com/taskcluster/taskcluster-auth
taskcluster-queue : https://github.com/taskcluster/taskcluster-queue

To-do

Still to confirm if we should track all files/folders or not.

Describe the bug
After adding all of the github repositories to repositories.json, the script started throwing the following error
Traceback (most recent call last): File "E:/bcrisan/repos/fic/client.py", line 186, in <module> useful_data = filter_commit_data(commit_data) File "E:/bcrisan/repos/fic/client.py", line 151, in filter_commit_data commit_message = item['commit']['message'] TypeError: string indices must be integers

To Reproduce
Steps to reproduce the behavior:

1. Open the project in pycharm
2. Clone the project
3. Make sure you have the GIT_TOKEN installed
4. Run the script
5. See error

Expected behavior
The creation of multiple .md that would contain the commits table for each repository newly added under github json.

Desktop (please complete the following information):

• OS: Windows
• Browser: Not applicable
• Version: any version that contains commit 8d1b167

Now that we have the list of repositories we should start by asking each of the developers/contributors (that maintain these repositories) which files/folders (if that's the case) affects our infrastructure and how bad in case of an issue.

As discussed in the meeting from November 5'th, 2018 we should start asking every shift with the contributors.

Where to start:

List of repositories:

GitHub repositories:

• Ship it - https://github.com/mozilla-releng/ship-it
• services - https://github.com/mozilla/release-services
• beetmoverscript - https://github.com/mozilla-releng/beetmoverscript
• addonscript - https://github.com/mozilla-releng/addonscript
• ship it V2 - https://github.com/mozilla-releng/shipit-v2
• build-cloud-tools - https://github.com/mozilla-releng/build-cloud-tools
• build-puppet - https://github.com/mozilla-releng/build-puppet
• shipitscript - https://github.com/mozilla-releng/shipitscript
• bouncerscript - https://github.com/mozilla-releng/bouncerscript
• treescript - https://github.com/mozilla-releng/treescript
• mozapkpublisher - https://github.com/mozilla-releng/mozapkpublisher
• OpenCloudConfig - https://github.com/mozilla-releng/OpenCloudConfig
• scriptworker - https://github.com/mozilla-releng/scriptworker
• pushsnapscript - https://github.com/mozilla-releng/pushsnapscript
• signingscript - https://github.com/mozilla-releng/signingscript
• pushapkscript - https://github.com/mozilla-releng/pushapkscript
• balrogscript - https://github.com/mozilla-releng/balrogscript
• funsize - https://github.com/mozilla-releng/funsize
• signtool - https://github.com/mozilla-releng/signtool
• shipitapi - https://github.com/mozilla-releng/shipitapi
• ami-builder - https://github.com/mozilla-releng/ami-builder

HG repositories:

• Mozilla Build - https://hg.mozilla.org/mozilla-build/
• Mozilla Central - https://hg.mozilla.org/mozilla-central/
• Try - https://hg.mozilla.org/try/
• Autoland - https://hg.mozilla.org/integration/autoland/
• Inbound - https://hg.mozilla.org/integration/mozilla-inbound/
• comm-esr60 - https://hg.mozilla.org/releases/comm-esr60/
• Beta - https://hg.mozilla.org/releases/mozilla-beta/
• mozilla-esr60 - https://hg.mozilla.org/releases/mozilla-esr60/
• mozilla-release - https://hg.mozilla.org/releases/mozilla-release/
• braindump - https://hg.mozilla.org/build/braindump/
• CI Configuration - https://hg.mozilla.org/build/ci-configuration/
• CI Admin - https://hg.mozilla.org/build/ci-admin/

Ignored repositories:

• Version Control Tools - https://hg.mozilla.org/hgcustom/version-control-tools/
• mozharness - https://hg.mozilla.org/build/mozharness/
• cot-gpg-keys - https://github.com/mozilla-releng/cot-gpg-keys
• binary-transparency - https://github.com/mozilla-releng/binary-transparency
• pulse-notify - https://github.com/mozilla-releng/pulse-notify
• mozilla-version - https://github.com/mozilla-releng/mozilla-version

Action Items

• Identify who are the contributors of the project
• Get in touch with them
• Explain to them what we are trying to accomplish
• Take notes and make sure to update this issue
• Implement changes (that would mean updating the repositories.json file)

What to keep in mind:

Make sure that you only have 1 main dictionary in which you have nested dictionaries containing all the information.

Requirements

• When using the json.dump command, make sure you use indent=2
• Runs on Py3.X

Repo in question is: https://github.com/mozilla-releng/balrogscript

Where to start:

Identify who are the core contributors of this project and get in touch with them via e-mail, Irc or GitHub.
Define a list of files from this repository which affect production.
Learn how to identify what has actually been deployed.

Repo in question is: https://github.com/mozilla-releng/scriptworker

Where to start:

Identify who are the core contributors of this project and get in touch with them via e-mail, Irc or GitHub.
Define a list of files from this repository which affect production.
Learn how to identify what has actually been deployed.

Repo in question is: https://github.com/mozilla-releng/build-puppet

Where to start:

Identify who are the core contributors of this project and get in touch with them via e-mail, Irc or GitHub.
Define a list of files from this repository which affect production.
Learn how to identify what has actually been deployed.

Create separate GItHub and Mercurial handler definitions, outside the main function.

I would prefer a specific definition of each handler that has proper documentation.

The current definitions are buggy and are missing features.
Both functions should contain/do the following:

• a way of filtering data
• should write both .md and .json files for each repository.

We will need to create a list of all github repositories that are being used for firefox-ci and/or could influence the infra.

A good place to start would be our team's Google Drive Escalation Doc.

Please post in the following format:
<Name of repo> - <Link to github repo>

Repo in question is: https://github.com/mozilla-releng/build-cloud-tools

Where to start:

Identify who are the core contributors of this project and get in touch with them via e-mail, Irc or GitHub.
Define a list of files from this repository which affect production.
Learn how to identify what has actually been deployed.

Short description:

If a commit message contain a special character like '\n' for new line or '\r' carriage return or '\t' tab ...( #, """, , %, etc), it will affect our files (Json and MD)

Where to start:

We have two option to fix this:
Option 1: We have to encode the information using UTF-8 witch will interpret those characters as a string so our .json and .md will not be affected
Option 2: We have to remove those characters from text to make it more clean and easy to read

Requirements

• Don't use classes
• Runs on Py3.X
• Document the code

What we have until now:

First option: we can encode each information using ".encode("utf-8")" after we get it from API
Option two: we can import 're' library and use "re.sub" function to remove the characters from the string

Any better idea will be welcome.

As Dustin suggested, we should distinguish between things that deploy automatically and things that are deployed via some separate action.
This would help avoid the confusion when looking for a particular change or when it took effect.

Define a list of files from this Repository which affects our infrastructure.

Where to start:

Identify who are the contributors of the project
Get in touch with them ( Email, Irc, GitHub etc)
Explain to them what we are trying to accomplish
Take notes and make sure to update this issue

Repo in question is: https://github.com/mozilla-releng/OpenCloudConfig

Where to start:

Identify who are the core contributors of this project and get in touch with them via e-mail, Irc or GitHub.
Define a list of files from this repository which affect production.
Learn how to identify what has actually been deployed.

Opening this feature request as I would like to suggest creating a main markdown table containing information such as one listed below:
What's your opinion on this?

MARKDOWN MAIN TABLE

Describe the bug
While running the script the following error appears :
Traceback (most recent call last): File "C:/repo/firefox-infra-changelog/client.py", line 351, in <module> create_files_for_hg() File "C:/repo/firefox-infra-changelog/client.py", line 221, in create_files_for_hg create_hg_md_table(repository_name) File "C:/repo/firefox-infra-changelog/client.py", line 112, in create_hg_md_table md_file.write(value + "\n\n") File "C:\Users\adrian.pop\AppData\Local\Programs\Python\Python36-32\lib\encodings\cp1252.py", line 19, in encode return codecs.charmap_encode(input,self.errors,encoding_table)[0] UnicodeEncodeError: 'charmap' codec can't encode character '\u0131' in position 2583: character maps to <undefined>

We have to create a function to generate a markdown table based on data that's written in changelog.json. I'm going to self assign for this one.

Repo in question is: https://github.com/mozilla-releng/funsize

Where to start:

Identify who are the core contributors of this project and get in touch with them via e-mail, Irc or GitHub.
Define a list of files from this repository which affect production.
Learn how to identify what has actually been deployed.

Note: This repository has been archived so it now read-only. Its last activity was a year ago.

We have a few branches that came as PRs that shouldn't.
After @mutterroland 's next shift, I will remove them, in preparetion of moving this repository under mozilla org.