Giter Club home page Giter Club logo

cortx-re's Introduction

License: AGPL v3 YouTube Badge Slack GitHub contributors

Disclaimer: This project is not maintained anymore

CORTX DevOps and Release Engineering

The purpose of this repository is to keep the scripts, tools, and other configuration used in the CORTX DevOps and Release Engineering.

DevOps and Release Engineering team is responsible for the followings

  • CORTX Project CI/CD Build Process.
  • Automation to support CORTX project.
  • Manage DevOps Tools.

Repo Structure

An overview of folder structure in cortx-re repo

├───docker
├───jenkins
└───scripts
└───solutions

Docker

  • We have containerized Release engineering infrastructure to eliminate host and OS dependency. The dockerfiles, docker-compose files used to build this containers are available under this folder.

Jenkins

  • Jenkins job configurations, groovy scripts and template used in the Jenkins are available under this folder.

Scripts

  • Shell, python scripts used in the RE process automation are available under this folder.
  • Scripts like changelog generation, build manifest generation, rpm validation..etc are available under this folder

Solutions

  • Solution specific scripts, configuration files.

Documents

All RE related documents are available in the below locations

cortx-re's People

Contributors

aayushisharma88 avatar abhijitpatil1992 avatar abhirup-a-g avatar balajiramachandran-seagate avatar gauravchaudhari02 avatar gowthamchinna avatar kapiljinna avatar mend-bolt-for-github[bot] avatar mend-for-github-com[bot] avatar mukul-seagate11 avatar naveen-kumar-seagate avatar nikhilpatil2995 avatar nitisdev avatar pankajgborole avatar parikshit-dharmale avatar pujamudaliar avatar rahul27kumar avatar sachitanands avatar sanket292001 avatar saumitrak531 avatar shailesh-vaidya avatar subhalaxmisa avatar swanand-gadre avatar swatid-seagate avatar swatiseagate avatar udayan-y21 avatar vaibhavparatwar avatar venkuppu-chn avatar venkyos avatar vijwani-seagate avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

cortx-re's Issues

Generic Workflow to create and Publish Docker images.

We can use base-docker-image workflow to generate Docker Images. It takes most of the parameters as variables so can be easily scaled to build Docker images for other components. 

Workflow details are as below.

steps:
   - uses: actions/checkout@v2
   - name: Set ENV
     run: |
       echo ::set-env name=INPUT_REPOSITORY::$( echo $GITHUB_REPOSITORY | tr '[:upper:]' '[:lower:]')
       echo ::set-env name=IMAGE_NAME::cortx_centos
       echo ::set-env name=TAG::7.7.1908
   - name: Build and push Docker images
     uses: docker/[email protected]
     env:
         DEFAULT_PASSWORD: ${{ secrets.DEFAULT_PASSWORD }}
         REPO_NAME: ${{ github.event.repository.name }}
     with:
       # Path to the Dockerfile (Default is '{path}/Dockerfile')
       dockerfile: docker/cortx_components/centos/Dockerfile 
       # Build Arguments for docker image build. 
       build_args: 'CENTOS_RELEASE=7.7.1908,USER_PASS=root:"$DEFAULT_PASSWORD"'
       path: docker/cortx_components/centos/
       username: ${{ github.actor }}
       password: ${{ secrets.GITHUB_TOKEN }}
       registry: docker.pkg.github.com
       repository: ${{ env.INPUT_REPOSITORY }}/${{ env.IMAGE_NAME }}
       tags: ${{ env.TAG }}

Github : Update Branch protection rules for dev/release/main/cortx-1.0/Cortx-v1.0.0_Beta branch's

Seagate Github Repo Branch protection rules are not up to date with DCO/ SLA and other mandatory checks. Please create Seagate standard for branch protection rule and apply it for all component repo branch's. also please document this.

Github Repo's:

  • All Seagate Repos

Branch's:

  • dev
  • release
  • main
  • cortx-1.0
  • Cortx-v1.0.0_Beta

Document:

  • Create a internal document with applied branch protection rule info

[ Troubleshooting ] : Build failure on ssc-vm-c-0266 ssc-vm-0497 VM

Due to VM infra Issue 2 of our Jenkins Node causes build failures. please find more info below

Affected Node

Dev Build Node = ssc-vm-c-0266.colo.seagate.com
Release Build Node = ssc-vm-0497.colo.seagate.com

Jenkins Build Failure Log

image

subscription-manager status

[root@ssc-vm-0497 ~]# subscription-manager list
+-------------------------------------------+
Installed Product Status
+-------------------------------------------+
Product Name: Red Hat Enterprise Linux Server
Product ID: 69
Version: 7.7
Arch: x86_64
Status: Not Subscribed
Status Details: Not supported by a valid subscription.
Starts:
Ends:

[root@ssc-vm-c-0266 736373]# subscription-manager list
+-------------------------------------------+
Installed Product Status
+-------------------------------------------+
Product Name: Red Hat Enterprise Linux Server
Product ID: 69
Version: 7.7
Arch: x86_64
Status: Not Subscribed
Status Details: Not supported by a valid subscription.
Starts:
Ends:

Host VM images in github releases?

Hello,

Is it possible to use github releases (as @mukul-seagate11 suggested it maybe be) to host VM images for the opensource community? For Seagate-only people, here is an OVA image that we would like to test whether it can be hosted by github: LINK.

Thanks!

Reduce docker image size.

We should reduce Component Docker images sizes particularly for RHEL builds. Observed that docker images size for S3Server is 8.18 GB . This seems to be causing increased build time for S3Server. For RHEL builds, S3Server takes around 45 minutes which was around 20 mins in Cent-OS.

[root@ssc-vm-0497 ~]# docker images
REPOSITORY                                    TAG                 IMAGE ID            CREATED             SIZE
rhel_csm                                      7.7.1908            d47dd4dc784f        3 days ago          3.27 GB
ssc-vm-0279.colo.seagate.com:5000/rhel_csm    7.7.1908            d47dd4dc784f        3 days ago          3.27 GB
ssc-vm-0279.colo.seagate.com:5000/rhel_hare   7.7.1908            48c93d80b1d1        6 weeks ago         7.83 GB
ssc-vm-0279.colo.seagate.com:5000/rhel_s3     7.7.1908            cc3a485e6d7a        7 weeks ago         8.18 GB
ssc-vm-0279.colo.seagate.com:5000/rhel_nfs    7.7.1908            0f709a90d6ae        7 weeks ago         8.01 GB
ssc-vm-0279.colo.seagate.com:5000/rhel_mero   7.7.1908            e8d510ef2f5b        7 weeks ago         6.62 GB
ssc-vm-0279.colo.seagate.com:5000/rhel_eos    7.7.1908            ba9f80c644b5        7 weeks ago         2.03 GB
registry.access.redhat.com/rhel7-init         7.7                 fe040a67fd36        4 months ago        205 MB

CVE-2020-14422 (Medium) detected in ipaddress-1.0.23-py2.py3-none-any.whl

CVE-2020-14422 - Medium Severity Vulnerability

Vulnerable Library - ipaddress-1.0.23-py2.py3-none-any.whl

IPv4/IPv6 manipulation library

Library home page: https://files.pythonhosted.org/packages/c2/f8/49697181b1651d8347d24c095ce46c7346c37335ddc7d255833e7cde674d/ipaddress-1.0.23-py2.py3-none-any.whl

Path to dependency file: cortx-re/scripts/release_support/ova-release-notes/requirements.txt

Path to vulnerable library: cortx-re/scripts/release_support/ova-release-notes/requirements.txt

Dependency Hierarchy:

  • cryptography-3.3.2-cp27-cp27mu-manylinux2010_x86_64.whl (Root Library)
    • ipaddress-1.0.23-py2.py3-none-any.whl (Vulnerable Library)

Found in HEAD commit: 07ac8101f14a36d963102c94ebaea9129a7e3c55

Vulnerability Details

Lib/ipaddress.py in Python through 3.8.3 improperly computes hash values in the IPv4Interface and IPv6Interface classes, which might allow a remote attacker to cause a denial of service if an application is affected by the performance of a dictionary containing IPv4Interface or IPv6Interface objects, and this attacker can cause many dictionary entries to be created. This is fixed in: v3.5.10, v3.5.10rc1; v3.6.12; v3.7.9; v3.8.4, v3.8.4rc1, v3.8.5, v3.8.6, v3.8.6rc1; v3.9.0, v3.9.0b4, v3.9.0b5, v3.9.0rc1, v3.9.0rc2.

Publish Date: 2020-06-18

URL: CVE-2020-14422

CVSS 3 Score Details (5.9)

Base Score Metrics:

  • Exploitability Metrics:
    • Attack Vector: Network
    • Attack Complexity: High
    • Privileges Required: None
    • User Interaction: None
    • Scope: Unchanged
  • Impact Metrics:
    • Confidentiality Impact: None
    • Integrity Impact: None
    • Availability Impact: High

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Origin: https://security-tracker.debian.org/tracker/CVE-2020-14422

Release Date: 2020-06-18

Fix Resolution: 3.5.3-1+deb9u2, 3.7.3-2+deb10u2, 3.8.4~rc1-1

Need a VM for development

The release VM (OVA) that we have currently is awesome for quick deployment and testing.

However we also need a release VM (OVA) that is setup for development. Such a VM should come bundled with all the build dependencies.

Anything else @andriytk ?

Cortx stack deployment using ISO files.

We should be able to provision and deploy Cortx Stack from ISO file. We have generated two iso files.
Release ISO - This contains packages from all them component similar as release build.
Pre-requisite ISO - This contains required third party packages.

ISO files are placed at http://cortx-storage.colo.seagate.com/releases/cortx-iso/
Release ISO - http://cortx-storage.colo.seagate.com/releases/cortx-iso/releases/Cortx-1.0.0-Rel-2795.iso
Pre-requisite ISO - http://cortx-storage.colo.seagate.com/releases/cortx-iso/prereqs/Cortx-1.0.2-prereqs.iso

Release ISO content -

.
├── CHANGESET.txt
├── cortx-csm_agent-1.0.0-700_ebe8cb6.x86_64.rpm
├── cortx-csm_web-1.0.0-700_bf9d05f.x86_64.rpm
├── cortx-dsal-1.0.0-148_2045f17.el7.x86_64.rpm
├── cortx-dsal-devel-1.0.0-148_2045f17.el7.x86_64.rpm
├── cortx-fs-1.0.0-148_2045f17.el7.x86_64.rpm
├── cortx-fs-devel-1.0.0-148_2045f17.el7.x86_64.rpm
├── cortx-fs-ganesha-1.0.0-148_2045f17.el7.x86_64.rpm
├── cortx-fs-ganesha-test-1.0.0-148_2045f17.el7.x86_64.rpm
├── cortx-ha-1.0.0-30_ba5d410.x86_64.rpm
├── cortx-hare-1.0.0-744_git02842fa.el7.x86_64.rpm
├── cortx-libsspl_sec-1.0.0-520_gitf110da8.el7.x86_64.rpm
├── cortx-libsspl_sec-devel-1.0.0-520_gitf110da8.el7.x86_64.rpm
├── cortx-libsspl_sec-method_none-1.0.0-520_gitf110da8.el7.x86_64.rpm
├── cortx-libsspl_sec-method_pki-1.0.0-520_gitf110da8.el7.x86_64.rpm
├── cortx-motr-1.0.0-430_git6d9ef31_3.10.0_1062.el7.x86_64.rpm
├── cortx-motr-devel-1.0.0-430_git6d9ef31_3.10.0_1062.el7.x86_64.rpm
├── cortx-motr-tests-ut-1.0.0-430_git6d9ef31_3.10.0_1062.el7.x86_64.rpm
├── cortx-nsal-1.0.0-148_2045f17.el7.x86_64.rpm
├── cortx-nsal-devel-1.0.0-148_2045f17.el7.x86_64.rpm
├── cortx-prvsnr-1.0.0-380_git0d7a721_el7.x86_64.rpm
├── cortx-prvsnr-cli-1.0.0-380_git0d7a721_el7.x86_64.rpm
├── cortx-s3iamcli-1.0.0-1012_git138b0f4.noarch.rpm
├── cortx-s3iamcli-devel-1.0.0-1012_git138b0f4.noarch.rpm
├── cortx-s3server-1.0.0-1012_git138b0f4_el7.x86_64.rpm
├── cortx-sspl-1.0.0-520_gitf110da8.el7.noarch.rpm
├── cortx-sspl-cli-1.0.0-520_gitf110da8.el7.noarch.rpm
├── cortx-sspl-test-1.0.0-520_gitf110da8.el7.noarch.rpm
├── cortx-utils-1.0.0-148_2045f17.el7.x86_64.rpm
├── cortx-utils-devel-1.0.0-148_2045f17.el7.x86_64.rpm
├── python36-cortx-prvsnr-0.31.0-1.0.0.x86_64.rpm
├── RELEASE.INFO
├── RPM-GPG-KEY-Seagate
├── systemd-python36-1.0.0-520_gitf110da8.el7.x86_64.rpm
├── uds-pyi-1.0.0-1.r2.el7.x86_64.rpm
└── udx-discovery-0.1.2-3.el7.x86_64.rpm

0 directories, 36 files

Pre-requisite ISO content -

.
├── commons
│   ├── consul
│   │   └── consul_1.6.2
│   │       ├── consul
│   │       ├── consul_1.6.2_linux_amd64.zip
│   │       └── consul_1.6.2_SHA256SUMS
│   ├── elasticsearch
│   │   ├── elasticsearch-6.8.8.rpm
│   │   └── elasticsearch-oss-6.8.8.rpm
│   ├── ha
│   ├── kibana
│   │   ├── kibana-6.8.8-x86_64.rpm
│   │   └── kibana-oss-6.8.8-x86_64.rpm
│   ├── node
│   │   ├── node-v12.13.0-linux-x64.tar.xz
│   │   └── SHASUMS256.txt.asc
│   ├── rsyslog-8.40
│   │   ├── libestr-0.1.11-1.el7.x86_64.rpm
│   │   ├── libfastjson4-0.99.8-1.el7.centos.x86_64.rpm
│   │   ├── liblognorm5-2.0.6-1.el7.x86_64.rpm
│   │   ├── rsyslog-8.40.0-1.el7.x86_64.rpm
│   │   ├── rsyslog-elasticsearch-8.40.0-1.el7.x86_64.rpm
│   │   └── rsyslog-mmjsonparse-8.40.0-1.el7.x86_64.rpm
│   └── saltstack-2019.2.0
│       ├── m2crypto-0.31.0-3.el7.x86_64.rpm
│       ├── m2crypto-debuginfo-0.31.0-3.el7.x86_64.rpm
│       ├── python2-ioflo-1.3.8-4.el7.noarch.rpm
│       ├── python2-libcloud-2.0.0-3.el7.noarch.rpm
│       ├── python2-mock-1.0.1-12.el7.noarch.rpm
│       ├── python2-raet-0.6.6-6.el7.noarch.rpm
│       ├── python2-timelib-0.2.4-4.el7.noarch.rpm
│       ├── python2-zmq-15.3.0-6.el7.x86_64.rpm
│       ├── python2-zmq-tests-15.3.0-6.el7.x86_64.rpm
│       ├── python36-cherrypy-5.6.0-5.el7.noarch.rpm
│       ├── python36-ioflo-1.3.8-4.el7.noarch.rpm
│       ├── python36-libcloud-2.0.0-3.el7.noarch.rpm
│       ├── python36-libnacl-1.6.1-2.el7.noarch.rpm
│       ├── python36-m2crypto-0.31.0-3.el7.x86_64.rpm
│       ├── python36-mock-1.0.1-12.el7.noarch.rpm
│       ├── python36-raet-0.6.6-6.el7.noarch.rpm
│       ├── python36-timelib-0.2.4-4.el7.noarch.rpm
│       ├── python36-zmq-15.3.0-6.el7.x86_64.rpm
│       ├── python36-zmq-tests-15.3.0-6.el7.x86_64.rpm
│       ├── python-cherrypy-5.6.0-5.el7.noarch.rpm
│       ├── python-libnacl-1.6.1-2.el7.noarch.rpm
│       ├── python-zmq-debuginfo-15.3.0-6.el7.x86_64.rpm
│       ├── repodata
│       │   ├── 13ef8c6573196e779ae69d0b3618a12c65da5b4c922ccecf243bcf87ac364ba2-other.sqlite.bz2
│       │   ├── 188abf84ba145252326260190e9148696ea793d3ef6e5add2b22b5a539078ea2-other.xml.gz
│       │   ├── 78f19de94fbf1a2f20c73e4bfe44369c33bf49a55c6b4ac37a793c302a8c413b-filelists.sqlite.bz2
│       │   ├── 8256c9f089a94bf31f424655dd4096078464e09eeb11f907a41cf7fa89c0fbd5-primary.xml.gz
│       │   ├── 91416d25ec47b60e4e3dc05158a238447b91af44a937b4fda5e559153a9f504e-primary.sqlite.bz2
│       │   ├── d3a1ab9e67a811555f7dda62038573a69eed0e38b9be3ffb95e442dd390a5f03-filelists.xml.gz
│       │   └── repomd.xml
│       ├── salt-2019.2.0-2.el7.noarch.rpm
│       ├── salt-api-2019.2.0-2.el7.noarch.rpm
│       ├── salt-cloud-2019.2.0-2.el7.noarch.rpm
│       ├── salt-master-2019.2.0-2.el7.noarch.rpm
│       ├── salt-minion-2019.2.0-2.el7.noarch.rpm
│       ├── salt-ssh-2019.2.0-2.el7.noarch.rpm
│       ├── SALTSTACK-GPG-KEY.pub
│       ├── salt-syndic-2019.2.0-2.el7.noarch.rpm
│       └── SRPMS
│           ├── m2crypto-0.31.0-3.el7.src.rpm
│           ├── python-cherrypy-5.6.0-5.el7.src.rpm
│           ├── python-ioflo-1.3.8-4.el7.src.rpm
│           ├── python-libcloud-2.0.0-3.el7.src.rpm
│           ├── python-libnacl-1.6.1-2.el7.src.rpm
│           ├── python-mock-1.0.1-12.el7.src.rpm
│           ├── python-raet-0.6.6-6.el7.src.rpm
│           ├── python-timelib-0.2.4-4.el7.src.rpm
│           ├── python-zmq-15.3.0-6.el7.src.rpm
│           └── salt-2019.2.0-2.el7.src.rpm
├── csm_uploads
│   ├── eos-py-utils-1.0.0-14_0a04f73.noarch.rpm
│   ├── eos-py-utils-1.0.0-21_ec2f7ce.noarch.rpm
│   ├── eos-py-utils-1.0.0-26_56e5515.noarch.rpm
│   └── stats_utils-1.0.0-12_12726bf.x86_64.rpm
├── EPEL-7
│   ├── csnappy-0-6.20150729gitd7bc683.el7.x86_64.rpm
│   ├── erlang-appmon-R16B-03.18.el7.x86_64.rpm
│   ├── erlang-asn1-R16B-03.18.el7.x86_64.rpm
│   ├── erlang-common_test-R16B-03.18.el7.x86_64.rpm
│   ├── erlang-compiler-R16B-03.18.el7.x86_64.rpm
│   ├── erlang-cosEventDomain-R16B-03.18.el7.x86_64.rpm
│   ├── erlang-cosEvent-R16B-03.18.el7.x86_64.rpm
│   ├── erlang-cosFileTransfer-R16B-03.18.el7.x86_64.rpm
│   ├── erlang-cosNotification-R16B-03.18.el7.x86_64.rpm
│   ├── erlang-cosProperty-R16B-03.18.el7.x86_64.rpm
│   ├── erlang-cosTime-R16B-03.18.el7.x86_64.rpm
│   ├── erlang-cosTransactions-R16B-03.18.el7.x86_64.rpm
│   ├── erlang-crypto-R16B-03.18.el7.x86_64.rpm
│   ├── erlang-debugger-R16B-03.18.el7.x86_64.rpm
│   ├── erlang-dialyzer-R16B-03.18.el7.x86_64.rpm
│   ├── erlang-diameter-R16B-03.18.el7.x86_64.rpm
│   ├── erlang-edoc-R16B-03.18.el7.x86_64.rpm
│   ├── erlang-eldap-R16B-03.18.el7.x86_64.rpm
│   ├── erlang-erl_docgen-R16B-03.18.el7.x86_64.rpm
│   ├── erlang-erl_interface-R16B-03.18.el7.x86_64.rpm
│   ├── erlang-erts-R16B-03.18.el7.x86_64.rpm
│   ├── erlang-et-R16B-03.18.el7.x86_64.rpm
│   ├── erlang-eunit-R16B-03.18.el7.x86_64.rpm
│   ├── erlang-examples-R16B-03.18.el7.x86_64.rpm
│   ├── erlang-gs-R16B-03.18.el7.x86_64.rpm
│   ├── erlang-hipe-R16B-03.18.el7.x86_64.rpm
│   ├── erlang-ic-R16B-03.18.el7.x86_64.rpm
│   ├── erlang-inets-R16B-03.18.el7.x86_64.rpm
│   ├── erlang-jinterface-R16B-03.18.el7.x86_64.rpm
│   ├── erlang-kernel-R16B-03.18.el7.x86_64.rpm
│   ├── erlang-megaco-R16B-03.18.el7.x86_64.rpm
│   ├── erlang-mnesia-R16B-03.18.el7.x86_64.rpm
│   ├── erlang-observer-R16B-03.18.el7.x86_64.rpm
│   ├── erlang-odbc-R16B-03.18.el7.x86_64.rpm
│   ├── erlang-orber-R16B-03.18.el7.x86_64.rpm
│   ├── erlang-os_mon-R16B-03.18.el7.x86_64.rpm
│   ├── erlang-otp_mibs-R16B-03.18.el7.x86_64.rpm
│   ├── erlang-parsetools-R16B-03.18.el7.x86_64.rpm
│   ├── erlang-percept-R16B-03.18.el7.x86_64.rpm
│   ├── erlang-pman-R16B-03.18.el7.x86_64.rpm
│   ├── erlang-public_key-R16B-03.18.el7.x86_64.rpm
│   ├── erlang-R16B-03.18.el7.x86_64.rpm
│   ├── erlang-reltool-R16B-03.18.el7.x86_64.rpm
│   ├── erlang-runtime_tools-R16B-03.18.el7.x86_64.rpm
│   ├── erlang-sasl-R16B-03.18.el7.x86_64.rpm
│   ├── erlang-sd_notify-0.1-1.el7.x86_64.rpm
│   ├── erlang-snmp-R16B-03.18.el7.x86_64.rpm
│   ├── erlang-ssh-R16B-03.18.el7.x86_64.rpm
│   ├── erlang-ssl-R16B-03.18.el7.x86_64.rpm
│   ├── erlang-stdlib-R16B-03.18.el7.x86_64.rpm
│   ├── erlang-syntax_tools-R16B-03.18.el7.x86_64.rpm
│   ├── erlang-test_server-R16B-03.18.el7.x86_64.rpm
│   ├── erlang-toolbar-R16B-03.18.el7.x86_64.rpm
│   ├── erlang-tools-R16B-03.18.el7.x86_64.rpm
│   ├── erlang-tv-R16B-03.18.el7.x86_64.rpm
│   ├── erlang-typer-R16B-03.18.el7.x86_64.rpm
│   ├── erlang-webtool-R16B-03.18.el7.x86_64.rpm
│   ├── erlang-wx-R16B-03.18.el7.x86_64.rpm
│   ├── erlang-xmerl-R16B-03.18.el7.x86_64.rpm
│   ├── genders-1.22-2.el7.x86_64.rpm
│   ├── gflags-2.1.1-6.el7.x86_64.rpm
│   ├── glog-0.3.3-8.el7.x86_64.rpm
│   ├── hiredis-0.12.1-1.el7.x86_64.rpm
│   ├── jq-1.6-1.el7.x86_64.rpm
│   ├── libgenders-1.22-2.el7.x86_64.rpm
│   ├── libsodium-1.0.18-1.el7.x86_64.rpm
│   ├── libuv-1.34.0-1.el7.x86_64.rpm
│   ├── miniz-1.15-4.r4.el7.x86_64.rpm
│   ├── nodejs-6.17.1-1.el7.x86_64.rpm
│   ├── npm-3.10.10-1.6.17.1.1.el7.x86_64.rpm
│   ├── oniguruma-5.9.5-3.el7.x86_64.rpm
│   ├── openpgm-5.2.122-2.el7.x86_64.rpm
│   ├── perl-common-sense-3.6-4.el7.noarch.rpm
│   ├── perl-Config-Any-0.24-1.el7.noarch.rpm
│   ├── perl-Config-General-2.61-1.el7.noarch.rpm
│   ├── perl-JSON-XS-3.01-2.el7.x86_64.rpm
│   ├── perl-MCE-1.862-1.el7.noarch.rpm
│   ├── perl-Sereal-3.015-1.el7.noarch.rpm
│   ├── perl-Sereal-Decoder-3.015-1.el7.x86_64.rpm
│   ├── perl-Sereal-Encoder-3.015-1.el7.x86_64.rpm
│   ├── perl-Types-Serialiser-1.0-1.el7.noarch.rpm
│   ├── perl-YAML-LibYAML-0.54-1.el7.x86_64.rpm
│   ├── python2-keyring-5.0-4.el7.noarch.rpm
│   ├── python2-pip-8.1.2-10.el7.noarch.rpm
│   ├── python2-qpid-proton-0.29.0-1.el7.x86_64.rpm
│   ├── python36-dateutil-2.4.2-5.el7.noarch.rpm
│   ├── python36-docutils-0.14-1.el7.noarch.rpm
│   ├── python36-jinja2-2.8.1-2.el7.noarch.rpm
│   ├── python36-m2crypto-0.35.2-5.el7.x86_64.rpm
│   ├── python36-pika-0.10.0-10.el7.noarch.rpm
│   ├── python36-ply-3.9-2.el7.noarch.rpm
│   ├── python36-psutil-2.2.1-5.el7.x86_64.rpm
│   ├── python36-pysocks-1.6.8-6.el7.noarch.rpm
│   ├── python36-PyYAML-3.12-1.el7.x86_64.rpm
│   ├── python36-requests-2.14.2-2.el7.noarch.rpm
│   ├── python36-six-1.11.0-3.el7.noarch.rpm
│   ├── python36-urllib3-1.25.1-1.el7.noarch.rpm
│   ├── qpid-proton-c-0.29.0-1.el7.x86_64.rpm
│   ├── qpid-proton-c-docs-0.29.0-1.el7.noarch.rpm
│   ├── rabbitmq-server-3.3.5-34.el7.noarch.rpm
│   ├── rubygem-net-ssh-2.9.1-1.el7.noarch.rpm
│   ├── s3cmd-2.0.2-1.el7.noarch.rpm
│   ├── statsd-0.7.2-10.el7.noarch.rpm
│   ├── wxBase-2.8.12-20.el7.x86_64.rpm
│   ├── wxGTK-2.8.12-20.el7.x86_64.rpm
│   ├── wxGTK-gl-2.8.12-20.el7.x86_64.rpm
│   └── yaml-cpp-0.5.1-2.el7.x86_64.rpm
├── lustre
│   └── custom
│       ├── o2ib
│       │   ├── kmod-lustre-client-2.12.0.5_357_g0c95a37-1.el7.x86_64.rpm
│       │   ├── kmod-lustre-client-tests-2.12.0.5_357_g0c95a37-1.el7.x86_64.rpm
│       │   ├── lustre-client-2.12.0.5_357_g0c95a37-1.el7.src.rpm
│       │   ├── lustre-client-2.12.0.5_357_g0c95a37-1.el7.x86_64.rpm
│       │   ├── lustre-client-debuginfo-2.12.0.5_357_g0c95a37-1.el7.x86_64.rpm
│       │   ├── lustre-client-devel-2.12.0.5_357_g0c95a37-1.el7.x86_64.rpm
│       │   ├── lustre-client-tests-2.12.0.5_357_g0c95a37-1.el7.x86_64.rpm
│       │   ├── lustre-iokit-2.12.0.5_357_g0c95a37-1.el7.x86_64.rpm
│       │   └── repodata
│       │       ├── 2aece93708f688cecddd729693c030222c4fd46497c0ddcd34d78126a013332c-other.sqlite.bz2
│       │       ├── 7aac29b1ff0985942a940c00e7a7c7407dffad61ed70c60cc146ecae5e8c1f5e-other.xml.gz
│       │       ├── 8b8cf2981380893327a9d665265751706bf5f1e8c66a25a779ee8563014dc6a6-filelists.xml.gz
│       │       ├── 9fa844fb4c1604ec3a3c70f8b1cf716993df5708be176520e2eb654bdbef2bff-primary.sqlite.bz2
│       │       ├── d19ed9631312aa2aac7e51497a02a06cdad613b5305ed573e41f6df934a92e5b-primary.xml.gz
│       │       ├── db99ef7b255331dfde6d30998cd60eb0391fe182795c66661db0a4deb0466be6-filelists.sqlite.bz2
│       │       └── repomd.xml
│       └── tcp
│           ├── kmod-lustre-client-2.12.0.5_357_g0c95a37-1.el7.x86_64.rpm
│           ├── kmod-lustre-client-tests-2.12.0.5_357_g0c95a37-1.el7.x86_64.rpm
│           ├── lustre-client-2.12.0.5_357_g0c95a37-1.el7.src.rpm
│           ├── lustre-client-2.12.0.5_357_g0c95a37-1.el7.x86_64.rpm
│           ├── lustre-client-debuginfo-2.12.0.5_357_g0c95a37-1.el7.x86_64.rpm
│           ├── lustre-client-devel-2.12.0.5_357_g0c95a37-1.el7.x86_64.rpm
│           ├── lustre-client-tests-2.12.0.5_357_g0c95a37-1.el7.x86_64.rpm
│           ├── lustre-iokit-2.12.0.5_357_g0c95a37-1.el7.x86_64.rpm
│           └── repodata
│               ├── 4cb5f2fc4cc0a94bd5bdb90e1c3ade5af36790a9ba9829505dcfa80e638693a8-primary.xml.gz
│               ├── 5e0fd7e1d5025ebd2b834f12b8660459d2549b670ae298437410324e6957bf84-other.sqlite.bz2
│               ├── 75f6f7ddd1c4bb648ad43c63010ae59d975fcdb4baab72d9a0c9024e76c23d01-primary.sqlite.bz2
│               ├── 854b716d80c8b561ee8e8503ba931df84c442fa6039b06857262fe8e28012c6f-filelists.xml.gz
│               ├── c63312811e1e964e6379b5db7180bad8fefd6634d674341cbeef969002fa0e71-other.xml.gz
│               ├── cdab7080b901697c8599d7d57ed577b79e2ad03e33d8388a0696e3456b078d31-filelists.sqlite.bz2
│               └── repomd.xml
├── misc
│   └── scsi-network-relay-1.1.0-47.x86_64.rpm
├── performance
│   ├── kmod-lustre-client-2.12.3-1.el7.x86_64.rpm
│   ├── linux.mellanox.com
│   │   └── public
│   │       └── repo
│   │           └── mlnx_ofed
│   │               └── 4.7-3.2.9.0
│   │                   └── rhel7.7
│   │                       └── x86_64
│   │                           └── MLNX_LIBS
│   │                               ├── ar_mgr-1.0-0.47.MLNX20191120.gf30d03a.47329.x86_64.rpm
│   │                               ├── cc_mgr-1.0-0.46.MLNX20191120.gf30d03a.47329.x86_64.rpm
│   │                               ├── dapl-2.1.10mlnx-OFED.3.4.2.1.0.47329.x86_64.rpm
│   │                               ├── dapl-devel-2.1.10mlnx-OFED.3.4.2.1.0.47329.x86_64.rpm
│   │                               ├── dapl-devel-static-2.1.10mlnx-OFED.3.4.2.1.0.47329.x86_64.rpm
│   │                               ├── dapl-utils-2.1.10mlnx-OFED.3.4.2.1.0.47329.x86_64.rpm
│   │                               ├── dump_pr-1.0-0.42.MLNX20191120.gf30d03a.47329.x86_64.rpm
│   │                               ├── fabric-collector-1.1.0.MLNX20170103.89bb2aa-0.1.47329.x86_64.rpm
│   │                               ├── hcoll-4.4.2938-1.47329.x86_64.rpm
│   │                               ├── ibacm-41mlnx1-OFED.4.3.3.0.0.47329.x86_64.rpm
│   │                               ├── ibacm-devel-41mlnx1-OFED.4.3.3.0.0.47329.x86_64.rpm
│   │                               ├── ibdump-5.0.0-3.47329.x86_64.rpm
│   │                               ├── ibsim-0.7mlnx1-0.11.g85c342b.47329.x86_64.rpm
│   │                               ├── ibutils-1.5.7.1-0.12.gdcaeae2.47329.x86_64.rpm
│   │                               ├── ibutils2-2.1.1-0.113.MLNX20191121.g1c29603.47329.x86_64.rpm
│   │                               ├── infiniband-diags-5.4.0.MLNX20190908.5f40e4f-0.1.47329.x86_64.rpm
│   │                               ├── infiniband-diags-compat-5.4.0.MLNX20190908.5f40e4f-0.1.47329.x86_64.rpm
│   │                               ├── infiniband-diags-guest-5.4.0.MLNX20190908.5f40e4f-0.1.47329.x86_64.rpm
│   │                               ├── kmod-iser-4.7-OFED.4.7.3.2.9.1.g457f064.rhel7u7.x86_64.rpm
│   │                               ├── kmod-isert-4.7-OFED.4.7.3.2.9.1.g457f064.rhel7u7.x86_64.rpm
│   │                               ├── kmod-kernel-mft-mlnx-4.13.3-1.rhel7u7.x86_64.rpm
│   │                               ├── kmod-knem-1.1.3.90mlnx1-OFED.4.7.2.0.7.1.gec1f2f8.rhel7u7.x86_64.rpm
│   │                               ├── kmod-mlnx-en-4.7-3.2.9.0.g457f064.rhel7u7.x86_64.rpm
│   │                               ├── kmod-mlnx-nvme-4.7-OFED.4.7.3.2.9.1.g457f064.rhel7u7.x86_64.rpm
│   │                               ├── kmod-mlnx-ofa_kernel-4.7-OFED.4.7.3.2.9.1.g457f064.rhel7u7.x86_64.rpm
│   │                               ├── kmod-rshim-1.16-0.ga7ad4e6.rhel7u7.x86_64.rpm
│   │                               ├── kmod-srp-4.7-OFED.4.7.3.2.9.1.g457f064.rhel7u7.x86_64.rpm
│   │                               ├── knem-1.1.3.90mlnx1-OFED.4.7.2.0.7.1.gec1f2f8.rhel7u7.x86_64.rpm
│   │                               ├── libdisni-1.9-OFED.4.5.0.0.6.47329.x86_64.rpm
│   │                               ├── libdisni-devel-1.9-OFED.4.5.0.0.6.47329.x86_64.rpm
│   │                               ├── libibcm-41mlnx1-OFED.4.1.0.1.0.47329.i686.rpm
│   │                               ├── libibcm-41mlnx1-OFED.4.1.0.1.0.47329.x86_64.rpm
│   │                               ├── libibcm-devel-41mlnx1-OFED.4.1.0.1.0.47329.i686.rpm
│   │                               ├── libibcm-devel-41mlnx1-OFED.4.1.0.1.0.47329.x86_64.rpm
│   │                               ├── libibmad-5.4.0.MLNX20190423.1d917ae-0.1.47329.i686.rpm
│   │                               ├── libibmad-5.4.0.MLNX20190423.1d917ae-0.1.47329.x86_64.rpm
│   │                               ├── libibmad-devel-5.4.0.MLNX20190423.1d917ae-0.1.47329.i686.rpm
│   │                               ├── libibmad-devel-5.4.0.MLNX20190423.1d917ae-0.1.47329.x86_64.rpm
│   │                               ├── libibmad-static-5.4.0.MLNX20190423.1d917ae-0.1.47329.i686.rpm
│   │                               ├── libibmad-static-5.4.0.MLNX20190423.1d917ae-0.1.47329.x86_64.rpm
│   │                               ├── libibumad-43.1.1.MLNX20190905.1080879-0.1.47329.i686.rpm
│   │                               ├── libibumad-43.1.1.MLNX20190905.1080879-0.1.47329.x86_64.rpm
│   │                               ├── libibumad-devel-43.1.1.MLNX20190905.1080879-0.1.47329.i686.rpm
│   │                               ├── libibumad-devel-43.1.1.MLNX20190905.1080879-0.1.47329.x86_64.rpm
│   │                               ├── libibumad-static-43.1.1.MLNX20190905.1080879-0.1.47329.i686.rpm
│   │                               ├── libibumad-static-43.1.1.MLNX20190905.1080879-0.1.47329.x86_64.rpm
│   │                               ├── libibverbs-41mlnx1-OFED.4.7.0.0.2.47329.i686.rpm
│   │                               ├── libibverbs-41mlnx1-OFED.4.7.0.0.2.47329.x86_64.rpm
│   │                               ├── libibverbs-devel-41mlnx1-OFED.4.7.0.0.2.47329.i686.rpm
│   │                               ├── libibverbs-devel-41mlnx1-OFED.4.7.0.0.2.47329.x86_64.rpm
│   │                               ├── libibverbs-devel-static-41mlnx1-OFED.4.7.0.0.2.47329.i686.rpm
│   │                               ├── libibverbs-devel-static-41mlnx1-OFED.4.7.0.0.2.47329.x86_64.rpm
│   │                               ├── libibverbs-utils-41mlnx1-OFED.4.7.0.0.2.47329.i686.rpm
│   │                               ├── libibverbs-utils-41mlnx1-OFED.4.7.0.0.2.47329.x86_64.rpm
│   │                               ├── libmlx4-41mlnx1-OFED.4.7.3.0.3.47329.i686.rpm
│   │                               ├── libmlx4-41mlnx1-OFED.4.7.3.0.3.47329.x86_64.rpm
│   │                               ├── libmlx4-devel-41mlnx1-OFED.4.7.3.0.3.47329.i686.rpm
│   │                               ├── libmlx4-devel-41mlnx1-OFED.4.7.3.0.3.47329.x86_64.rpm
│   │                               ├── libmlx5-41mlnx1-OFED.4.7.0.3.3.47329.i686.rpm
│   │                               ├── libmlx5-41mlnx1-OFED.4.7.0.3.3.47329.x86_64.rpm
│   │                               ├── libmlx5-devel-41mlnx1-OFED.4.7.0.3.3.47329.i686.rpm
│   │                               ├── libmlx5-devel-41mlnx1-OFED.4.7.0.3.3.47329.x86_64.rpm
│   │                               ├── librdmacm-41mlnx1-OFED.4.7.3.0.6.47329.i686.rpm
│   │                               ├── librdmacm-41mlnx1-OFED.4.7.3.0.6.47329.x86_64.rpm
│   │                               ├── librdmacm-devel-41mlnx1-OFED.4.7.3.0.6.47329.i686.rpm
│   │                               ├── librdmacm-devel-41mlnx1-OFED.4.7.3.0.6.47329.x86_64.rpm
│   │                               ├── librdmacm-utils-41mlnx1-OFED.4.7.3.0.6.47329.i686.rpm
│   │                               ├── librdmacm-utils-41mlnx1-OFED.4.7.3.0.6.47329.x86_64.rpm
│   │                               ├── librxe-41mlnx1-OFED.4.4.2.4.6.47329.i686.rpm
│   │                               ├── librxe-41mlnx1-OFED.4.4.2.4.6.47329.x86_64.rpm
│   │                               ├── librxe-devel-static-41mlnx1-OFED.4.4.2.4.6.47329.i686.rpm
│   │                               ├── librxe-devel-static-41mlnx1-OFED.4.4.2.4.6.47329.x86_64.rpm
│   │                               ├── libvma-8.9.5-1.x86_64.rpm
│   │                               ├── libvma-devel-8.9.5-1.x86_64.rpm
│   │                               ├── libvma-utils-8.9.5-1.x86_64.rpm
│   │                               ├── mft-4.13.3-6.x86_64.rpm
│   │                               ├── mlnx-en-doc-4.7-3.2.9.0.g457f064.rhel7u7.x86_64.rpm
│   │                               ├── mlnx-en-sources-4.7-3.2.9.0.g457f064.rhel7u7.x86_64.rpm
│   │                               ├── mlnx-en-utils-4.7-3.2.9.0.g457f064.rhel7u7.x86_64.rpm
│   │                               ├── mlnx-ethtool-5.1-1.47329.x86_64.rpm
│   │                               ├── mlnx-fw-updater-4.7-3.2.9.0.x86_64.rpm
│   │                               ├── mlnx-iproute2-5.2.0-1.47329.x86_64.rpm
│   │                               ├── mlnx-ofa_kernel-4.7-OFED.4.7.3.2.9.1.g457f064.rhel7u7.x86_64.rpm
│   │                               ├── mlnx-ofa_kernel-devel-4.7-OFED.4.7.3.2.9.1.g457f064.rhel7u7.x86_64.rpm
│   │                               ├── mlnx-ofed-all-4.7-3.2.9.0.rhel7.7.noarch.rpm
│   │                               ├── mlnx-ofed-all-user-only-4.7-3.2.9.0.rhel7.7.noarch.rpm
│   │                               ├── mlnx-ofed-basic-4.7-3.2.9.0.rhel7.7.noarch.rpm
│   │                               ├── mlnx-ofed-basic-user-only-4.7-3.2.9.0.rhel7.7.noarch.rpm
│   │                               ├── mlnx-ofed-bluefield-4.7-3.2.9.0.rhel7.7.noarch.rpm
│   │                               ├── mlnx-ofed-bluefield-user-only-4.7-3.2.9.0.rhel7.7.noarch.rpm
│   │                               ├── mlnxofed-docs-4.7-3.2.9.0.noarch.rpm
│   │                               ├── mlnx-ofed-dpdk-4.7-3.2.9.0.rhel7.7.noarch.rpm
│   │                               ├── mlnx-ofed-dpdk-user-only-4.7-3.2.9.0.rhel7.7.noarch.rpm
│   │                               ├── mlnx-ofed-eth-only-user-only-4.7-3.2.9.0.rhel7.7.noarch.rpm
│   │                               ├── mlnx-ofed-guest-4.7-3.2.9.0.rhel7.7.noarch.rpm
│   │                               ├── mlnx-ofed-guest-user-only-4.7-3.2.9.0.rhel7.7.noarch.rpm
│   │                               ├── mlnx-ofed-hpc-4.7-3.2.9.0.rhel7.7.noarch.rpm
│   │                               ├── mlnx-ofed-hpc-user-only-4.7-3.2.9.0.rhel7.7.noarch.rpm
│   │                               ├── mlnx-ofed-hypervisor-4.7-3.2.9.0.rhel7.7.noarch.rpm
│   │                               ├── mlnx-ofed-hypervisor-user-only-4.7-3.2.9.0.rhel7.7.noarch.rpm
│   │                               ├── mlnx-ofed-kernel-only-4.7-3.2.9.0.rhel7.7.noarch.rpm
│   │                               ├── mlnx-ofed-vma-4.7-3.2.9.0.rhel7.7.noarch.rpm
│   │                               ├── mlnx-ofed-vma-eth-4.7-3.2.9.0.rhel7.7.noarch.rpm
│   │                               ├── mlnx-ofed-vma-eth-user-only-4.7-3.2.9.0.rhel7.7.noarch.rpm
│   │                               ├── mlnx-ofed-vma-user-only-4.7-3.2.9.0.rhel7.7.noarch.rpm
│   │                               ├── mlnx-ofed-vma-vpi-4.7-3.2.9.0.rhel7.7.noarch.rpm
│   │                               ├── mlnx-ofed-vma-vpi-user-only-4.7-3.2.9.0.rhel7.7.noarch.rpm
│   │                               ├── mpi-selector-1.0.3-1.47329.x86_64.rpm
│   │                               ├── mpitests_openmpi-3.2.20-e1a0676.47329.x86_64.rpm
│   │                               ├── mstflint-4.13.0-1.41.g4e8819c.47329.x86_64.rpm
│   │                               ├── mxm-3.7.3112-1.47329.x86_64.rpm
│   │                               ├── neohost-backend-1.4.0-14.x86_64.rpm
│   │                               ├── ofed-scripts-4.7-OFED.4.7.3.2.9.x86_64.rpm
│   │                               ├── openmpi-4.0.2rc3-1.47329.x86_64.rpm
│   │                               ├── opensm-5.5.1.MLNX20191120.0c8dde0-0.1.47329.x86_64.rpm
│   │                               ├── opensm-devel-5.5.1.MLNX20191120.0c8dde0-0.1.47329.x86_64.rpm
│   │                               ├── opensm-libs-5.5.1.MLNX20191120.0c8dde0-0.1.47329.x86_64.rpm
│   │                               ├── opensm-static-5.5.1.MLNX20191120.0c8dde0-0.1.47329.x86_64.rpm
│   │                               ├── perftest-4.4-0.11.gd240b65.47329.x86_64.rpm
│   │                               ├── qperf-0.4.9-9.47329.x86_64.rpm
│   │                               ├── repodata
│   │                               ├── sharp-2.0.0.MLNX20190922.a9ebf22-1.47329.x86_64.rpm
│   │                               ├── sockperf-3.6-0.git737d1e3e5576.47329.x86_64.rpm
│   │                               ├── srptools-41mlnx1-5.47329.x86_64.rpm
│   │                               ├── ucx-1.7.0-1.47329.x86_64.rpm
│   │                               ├── ucx-cma-1.7.0-1.47329.x86_64.rpm
│   │                               ├── ucx-devel-1.7.0-1.47329.x86_64.rpm
│   │                               ├── ucx-ib-1.7.0-1.47329.x86_64.rpm
│   │                               ├── ucx-ib-cm-1.7.0-1.47329.x86_64.rpm
│   │                               ├── ucx-knem-1.7.0-1.47329.x86_64.rpm
│   │                               └── ucx-rdmacm-1.7.0-1.47329.x86_64.rpm
│   └── lustre-client-2.12.3-1.el7.x86_64.rpm
├── prvsnr_uploads
│   ├── eos-install-prereqs-centos.sh
│   ├── eos-install-prereqs.sh
│   └── eos-install-prereqs-vm.sh
├── repodata
│   ├── 0d4b318f4891f3bbc5d4b44815e89e909b36d2a1bef3e55b5a1834fd647abc73-other.sqlite.bz2
│   ├── 28d5d775c6a0f993022052b1e24fc6cc7092690f33e198945dac6a1ed1da0c86-filelists.xml.gz
│   ├── 4dd28b99973b8c5089b5134aa9d166948949038d73a9f88ca4d404eaa0eab5a6-primary.sqlite.bz2
│   ├── 7c9fbba7753a97ba8485f88c011678e2645ae60bfa9f535e0275c310cda5a6c1-other.xml.gz
│   ├── babd8342b235d1964de2c3e431bf809eb5d0918f22ce158b6bd53233871e1f4d-primary.xml.gz
│   ├── f6f37d665ad6eb9a52b32bb3c667d1c7bbcf6e087076c32feed9b333a8afa412-filelists.sqlite.bz2
│   └── repomd.xml
└── s3server_uploads
    ├── log4cxx_eos-0.10.0-1.x86_64.rpm
    ├── log4cxx_eos-devel-0.10.0-1.x86_64.rpm
    ├── python2-jmespath-0.9.0-1.el7.noarch.rpm
    ├── python2-s3transfer-0.1.10-1.el7.noarch.rpm
    ├── python36-boto3-1.4.6-1.el7.noarch.rpm
    ├── python36-botocore-1.6.0-1.el7.noarch.rpm
    ├── python36-s3transfer-0.1.10-1.el7.noarch.rpm
    ├── python36-xmltodict-0.9.0-1.el7.noarch.rpm
    └── s3cmd-2.0.2-1.el7.noarch.rpm

33 directories, 355 files

Changes in cortx.spec file for generating Metarpm for build

For release build we are creating metarpm which will add all rpms mentioned in RELEASE.INFO file in cortx.spec file and after executing rpmbuild command on cortx.spec file we are getting one metarpm generated.

Refer cortx.spec file at -> https://github.com/Seagate/cortx-re/blob/metarpm-script/scripts/cortx_metarpm/cortx.spec

This cortx.spec file is generating rpm with release type name for example: beta,ga etc.

As per comments mentioned on https://jts.seagate.com/browse/EOS-8766 we need to change rpm name and also there is echo command in %build section of cortx.spec file which is expected to be removed.

But after removing %build and %install section, only source rpm is created and not rpm in /rpmbuild/RPMS folder.

Generic Workflow to build component packages

We should use generic workflow to build component rpm's . I have created one for Provisioner at provisioner-build.yml . It will also need changes in Dockerfile's .

jobs:
  # This workflow contains a single job called "build"
  build:
    # The type of runner that the job will run on
    runs-on: ubuntu-latest

    # Steps represent a sequence of tasks that will be executed as part of the job
    steps:
    # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
    - uses: actions/checkout@v2
    
    - name: Checkout
      uses: actions/checkout@v2
      with:
        # Repository name with owner. For example, actions/checkout
        repository: Seagate/cortx-prvsnr
        token: ${{ secrets.TOKEN }}
        path: provisioner

    # Runs a set of commands using the runners shell
    - name: Build Provisioner packages
      run: |
        echo "${{ secrets.GITHUB_TOKEN }}" | docker login docker.pkg.github.com -u ${{ github.actor }} --password-stdin
        docker pull docker.pkg.github.com/seagate/cortx-re/cortx_centos:7.7.1908
        docker run -i -v $PWD:/build docker.pkg.github.com/seagate/cortx-re/cortx_centos:7.7.1908 'sh /build/build_scripts/build_provisoiner.sh'
       
    - name: Upload files to a GitHub release
      uses: svenstaro/[email protected]
      with:
        repo_token: ${{ secrets.TOKEN }}
        release_name: provisioner
        file: provisioner/release/*.rpm
        tag: 1.0.0
        file_glob: true
        body: "Provisioner Release"

There are some better actions available to upload artifacts to GitHub releases. We can try out them. Do review and provide suggestions.

Interdependent component should pull rpm version from same branch.

Observed that during cortx-manager build for cortx-1.0 branch cortx-provisioner is installed from main branch.

16:43:16  + yum install -y cortx-prvsnr
16:43:16  Loaded plugins: fastestmirror, ovl, priorities
16:43:17  Determining fastest mirrors
16:43:25  Resolving Dependencies
16:43:25  --> Running transaction check
16:43:25  ---> Package cortx-prvsnr.x86_64 0:1.0.0-417_gitbe88d99_el7 will be installed
16:43:25  --> Processing Dependency: python36-PyYAML for package: cortx-prvsnr-1.0.0-417_gitbe88d99_el7.x86_64
16:43:25  --> Running transaction check
16:43:25  ---> Package python36-PyYAML.x86_64 0:3.13-1.el7 will be installed
16:43:25  --> Finished Dependency Resolution
16:43:25  
16:43:25  Dependencies Resolved
16:43:25  
16:43:25  ================================================================================
16:43:25   Package           Arch     Version                     Repository         Size
16:43:25  ================================================================================
16:43:25  Installing:
16:43:25   cortx-prvsnr      x86_64   1.0.0-417_gitbe88d99_el7    cortx-C7.7.1908   292 k
16:43:25  Installing for dependencies:
16:43:25   python36-PyYAML   x86_64   3.13-1.el7                  epel              149 k
16:43:25  
16:43:25  Transaction Summary
16:43:25  ================================================================================
16:43:25  Install  1 Package (+1 Dependent package)

Deployment Support for Custom Builds

custom build are not supported by HW deployment due to following reasons

  1. cortx_prep.sh script expects iso name should be cortx--single.iso but custom build iso is named as custom-build--single.iso
  2. Automated HW Deployment job expects cortx-prep.sh and cortx-os-*.iso should be available in build iso folder but it's not yet implemented in custom build

Cleanup Prod Jenkins Node configurations

There are many unused and disconnected Jenkins Agents in Production Jenkins. We need to remove them to free up infra resources. In order to do that we need the following info for each node 

  • Use who added node
  • VM/HW node configured in the node

We should create a Jenkins job to list down the above info for offline nodes. 

Refernce - 

  • To get user details - [https://eos-jenkins.colo.seagate.com/computer/0427_althummar/jobConfigHistory/] 
  • VM/HW Details - [https://eos-jenkins.colo.seagate.com/computer/0427_althummar/configure] 

 

Jenkins logs showing warnings for http://gitlab.mero.colo.seagate.com/eos/fs/nfs-ganesha.git

Prod Jenkins logs are showing,

2020-12-08 06:13:00.982+0000 [id=54]    WARNING c.c.h.p.f.c.PeriodicFolderTrigger#run: Queue refused to schedule org.jenkinsci.plugins.workflow.multibranch.WorkflowMultiBranchProject@67267635[Release_Engineering/github-work/GitHub Multibranch Jobs/nfs-multibranch]
2020-12-08 06:13:00.985+0000 [id=54]    WARNING c.c.h.p.f.c.PeriodicFolderTrigger#run: Queue refused to schedule org.jenkinsci.plugins.workflow.multibranch.WorkflowMultiBranchProject@754b1c5c[Archive/Cortx-PreMerge-Test/PV-PreMerge-ST]
[12/08/20 06:13:02] SSH Launch of qa-nfs-node on 10.237.65.146 failed in 4,726 ms
[12/08/20 06:13:02] SSH Launch of qa-perf-node on 10.237.65.111 failed in 4,803 ms
2020-12-08 06:14:00.983+0000 [id=48]    WARNING c.c.h.p.f.c.PeriodicFolderTrigger#run: Queue refused to schedule org.jenkinsci.plugins.workflow.multibranch.WorkflowMultiBranchProject@754b1c5c[Archive/Cortx-PreMerge-Test/PV-PreMerge-ST]
2020-12-08 06:14:22.483+0000 [id=35]    WARNING c.c.jenkins.GitHubRepositoryName#create: Could not match URL http://gitlab.mero.colo.seagate.com/eos/fs/nfs-ganesha.git
2020-12-08 06:14:22.484+0000 [id=35]    WARNING c.c.jenkins.GitHubRepositoryName#create: Could not match URL http://gitlab.mero.colo.seagate.com/eos/fs/nfs-ganesha.git
2020-12-08 06:14:22.484+0000 [id=35]    WARNING c.c.jenkins.GitHubRepositoryName#create: Could not match URL http://gitlab.mero.colo.seagate.com/eos/fs/nfs-ganesha.git
2020-12-08 06:14:22.484+0000 [id=35]    WARNING c.c.jenkins.GitHubRepositoryName#create: Could not match URL http://gitlab.mero.colo.seagate.com/eos/fs/nfs-ganesha.git
2020-12-08 06:14:22.484+0000 [id=35]    WARNING c.c.jenkins.GitHubRepositoryName#create: Could not match URL http://gitlab.mero.colo.seagate.com/eos/fs/nfs-ganesha.git
2020-12-08 06:14:22.485+0000 [id=35]    WARNING c.c.jenkins.GitHubRepositoryName#create: Could not match URL http://gitlab.mero.colo.seagate.com/eos/fs/nfs-ganesha.git
2020-12-08 06:14:22.485+0000 [id=35]    WARNING c.c.jenkins.GitHubRepositoryName#create: Could not match URL http://gitlab.mero.colo.seagate.com/eos/fs/nfs-ganesha.git
2020-12-08 06:14:22.485+0000 [id=35]    WARNING c.c.jenkins.GitHubRepositoryName#create: Could not match URL http://gitlab.mero.colo.seagate.com/eos/fs/nfs-ganesha.git
2020-12-08 06:14:22.485+0000 [id=35]    WARNING c.c.jenkins.GitHubRepositoryName#create: Could not match URL http://gitlab.mero.colo.seagate.com/eos/fs/nfs-ganesha.git
2020-12-08 06:14:22.485+0000 [id=35]    WARNING c.c.jenkins.GitHubRepositoryName#create: Could not match URL http://gitlab.mero.colo.seagate.com/eos/fs/nfs-ganesha.git
2020-12-08 06:14:22.485+0000 [id=35]    WARNING c.c.jenkins.GitHubRepositoryName#create: Could not match URL http://gitlab.mero.colo.seagate.com/eos/fs/nfs-ganesha.git
2020-12-08 06:14:22.486+0000 [id=35]    WARNING c.c.jenkins.GitHubRepositoryName#create: Could not match URL http://gitlab.mero.colo.seagate.com/eos/fs/nfs-ganesha.git
2020-12-08 06:14:22.486+0000 [id=35]    WARNING c.c.jenkins.GitHubRepositoryName#create: Could not match URL http://gitlab.mero.colo.seagate.com/eos/fs/nfs-ganesha.git

Release Automation Jenkins pipelines issues.

We should below issues in Release Automation Job pipeline

Release Notes for OVA 1.0.4 release.

We should implement below for regular OVA releases.

  • OVA Build frequency - Every Sprint End ( 2 Weeks)

  • OVA Versioning - As mentioned in Seagate/cortx#441 (comment) We should use semantic versioning for OVA . Currently we have released cortx-va-1.0.*.zip file at https://github.com/Seagate/cortx/releases/tag/VA . We should create new GitHub releases names cortx-va-1.0. and upload cortx-va-1.0..zip into it.

  • Release Notes - We can add Jira query or GitHub log query output to list down changes landed into main branch in sprint. For internal releases we used to send release notes with Jira issues fixed within sprint along with Provisioning guide. @Venkataraman has added Release Notes template at https://github.com/Seagate/cortx/blob/main/doc/RNTemplate.rst . We can improve and add it as description to GitHub release.

  • Refer sample LDR Release notes from Release Build #2714 for reference. Jira query used is Sprint = EES_Sprint-33 AND resolutiondate >= '2020-12-17 00:00' AND resolutiondate <= '2020-12-29 00:00' . Refer detailed output at - Sprint end Jira query

  • You can use get-jira-stats.sh and changelog.sh to fetch Git and Jiras stats for sprint automatically.

Migrate CI from Jenkins to GitHub Actions.

To migrate CI workflow from Jenkins to GitHub actions we should discuss and finalize our approach on below tasks. Please add if anything is missed. I have added possible solutions for few tasks. Please review and add your suggestions.

  1. Work with component team and use container envorinment for pre-merge jobs.
  2. Use self hosted runners. https://github.blog/2019-11-05-self-hosted-runners-for-github-actions-is-now-in-beta/
  • Integration Build Generation and location -
    Cortx need to be deployed as full stack for all the features. Currently we build all components individually and generate integrated release build. Release build contains rpm's from all the components. We should define strategy to build component level rpm's and generate release build from them. We should also finalize location (GitHub Release nomenclature) to place them.

    Possible solution
    We can create GitHub actions in every component repository and place artifact's (generated rpm's) at component repository. Component rpm's will be placed in component level GitHub releases. Release engineering repository will have GitHub action to collect all components rpm's, perform other steps (rpm-signing, dependency validation etc.) and place release build at Release Engineering repository.

  • Branch wise Release build generation and location -
    We are generating branch wise release builds and placing them in different directories of ci-storage. We need to plan for Release build structure to support branch wise releases.

    Possible solution
    Finalize Release name syntax and adopt it in GitHub releases. So that branch wise release's can be maintained.

  • Third party packages -
    We are using couple of third party packages which are not built by Seagate team. We are storing them in ci-storage uploads directory. We will need to store them in GitHub release.

    Possible solution
    Create release named uploads and use it in GitHub actions to download and install third party packages.

  • Generic workflow to build component Docker images and rpm's
    We should develop and use standard workflow's to build component Docker images and rpm's. This will help in debugging issues faster and support multiple OS and Kernel versions.

    Created #39 and #27 for workflow related discussion. Please add your suggestions/reviews there.

  • Automated notifications, validation and checks -
    Currently Jenkins has configured many automated notifications e.g. Email notification, Add release build details as Jira comment,. We have also configured couple of validation checks to avoid deployment failures e.g. RPM validation and dependency check. Motr CI customization etc. We will need to plan them in GitHub Actions.

CC: @amolkongre @johnbent @mukul-seagate11 @gauravchaudhari02 @pujamudaliar @venkuppu-chn @balajiramachandran-seagate

CSM build is not using latest Provisioner package.

Currently repositories in Docker images are pointed to http://ci-storage.mero.colo.seagate.com/releases/eos/github/master/rhel-7.7.1908/ after branch name change need them to update to http://ci-storage.mero.colo.seagate.com/releases/eos/github/release/rhel-7.7.1908/

Also now CentOS repository links are installing centos-release for CentOS 7.8 version. Hence need to exclude it duringyum upgrade all

Facing GitHub API quota issue in multi-branch pipelines.

We are using polling method to trigger Jenkins jobs on code commit. Seems we are exceeding GitHub API limit and that's causing delay in multi-branch pipelines execution.
Please refer - HA Multi-branch job.

Logs are as below,

Started by timer
[Mon Sep 14 05:25:00 UTC 2020] Starting branch indexing...
05:25:00 Connecting to https://api.github.com using cortx-admin/****** (RE Github Access)
05:25:00 GitHub API Usage: Current quota has 2936 remaining (483 over budget). Next quota of 5000 in 50 min. Sleeping for 8 min 8 sec.
05:28:00 GitHub API Usage: Still sleeping, now only 5 min 8 sec remaining.
05:31:00 GitHub API Usage: Still sleeping, now only 2 min 8 sec remaining.
05:33:09 GitHub API Usage: Current quota has 2320 remaining (590 over budget). Next quota of 5000 in 42 min. Sleeping for 9 min 51 sec.
05:36:09 GitHub API Usage: Still sleeping, now only 6 min 51 sec remaining.
05:39:09 GitHub API Usage: Still sleeping, now only 3 min 51 sec remaining.
05:42:09 GitHub API Usage: Still sleeping, now only 51 sec remaining.
05:43:00 GitHub API Usage: Current quota has 100 remaining (2194 over budget). Next quota of 5000 in 32 min. Sleeping until reset.

Release Branch Pre-Merge Testing

To maintain clean build in release branch we need to ensure build quality before new changes merged into release branch and this can be achieved by introducing CT (Continuous Testing) in the release branch pre merge pipeline (Multibranch pipeline)

Existing Multibranch Workflow :

  • RPM Build tested for all the pull request.

Disadvantage

  • Only RPM Build is tested using this pipeline but component basic functionalities are not tested.

Suggested Workflow

  • Perform VM deployment on each pull request targeted to release branch. later sanity test and other functionality related tests can be added to this.

  • Configure GitHub Pull request status check (required) for the above. This allows merges can be performed only when VM deployment is success.

[ Document ] : Migrate/Update RE document to Github Wiki

Jenkins stability

Today we faced Disk space issue on Prod Jenkins. Jenkins log reported error as,

2020-10-15 08:41:07.188+0000 [id=50] WARNING hudson.model.Queue#save: Failed to write out the queue file /var/jenkins_home/queue.xml
java.io.IOException: Disk quota exceeded

Jenkins Home is mounted over nfs at /mnt/data1/releases/jenkins/jenkins-prod

We found that there is enough disk space available on cortx-storage server.

[root@ssc-vm-0125 jenkins_workspace]# df -Th | grep cortx-storage
cortx-storage.colo.seagate.com:/mnt/data1 nfs4 4.9T 3.1T 1.9T 62% /mnt/data1
[root@ssc-vm-0125 jenkins_workspace]#

While debugging this issue along with @gowthamchinna found that jenkins_home directory is consuming 100Gb space.

[root@ssc-vm-0279 jenkins]# du -sh *
32G     jenkins-dev
100G    jenkins-prod

We are placing all our component artifacts at nfs mount and jenkins_home should be containing only configuration files. It should not consume this much space. On investigation found that space is consumed to Jenkins job log files. I am cleaning up old log files.

To avoid this in future for every Jenkins job we should enable log rotation,

image

Error in 3rd_party release

(venv_cortx) [root@localhost ~]# provisioner setup_provisioner srvnode-1:$(hostname -f) --logfile --logfile-filename /var/log/seagate/provisioner/setup.log --source rpm --config-path ~/config.ini --dist-type bundle --target-build ${​​​​​​CORTX_RELEASE_REPO}​​​​​​
2021-02-09 06:20:32,858 - INFO - Starting to build setup 'srvnode-1_root@localhost_22'
2021-02-09 06:20:32,863 - INFO - Profile location '/root/.provisioner/srvnode-1_root@localhost_22'
2021-02-09 06:20:32,863 - INFO - Preparing setup pillar
2021-02-09 06:20:32,864 - INFO - Generating setup keys [skipped]
2021-02-09 06:20:32,864 - INFO - Generating a roster file
2021-02-09 06:20:32,865 - INFO - Copying config.ini to file root
2021-02-09 06:20:32,871 - INFO - Preparing CORTX repos pillar
2021-02-09 06:20:32,887 - INFO - Ensuring 'srvnode-1' is ready to accept commands
2021-02-09 06:20:59,600 - INFO - Resolving node grains
2021-02-09 06:21:01,621 - INFO - Preparing salt-masters / minions configuration
2021-02-09 06:21:01,628 - INFO - Preparing factory profile
2021-02-09 06:21:01,668 - INFO - Installing Cortx yum repositories
2021-02-09 06:21:08,409 - INFO - Setting up custom python repository
2021-02-09 06:21:12,883 - INFO - Checking paswordless ssh
2021-02-09 06:21:17,125 - INFO - Configuring the firewall
2021-02-09 06:21:21,339 - INFO - Installing SaltStack
2021-02-09 06:21:31,000 - INFO - Installing provisioner from a 'rpm' source
2021-02-09 06:21:41,766 - INFO - Configuring salt minions
2021-02-09 06:21:46,698 - INFO - Configuring salt-masters
2021-02-09 06:22:11,553 - INFO - Copying factory data
2021-02-09 06:22:26,895 - INFO - Installing provisioner API
2021-02-09 06:22:32,678 - INFO - Starting salt minions
2021-02-09 06:22:37,222 - INFO - Ensuring salt-masters are ready
2021-02-09 06:22:51,941 - INFO - Ensuring salt minions are ready
2021-02-09 06:23:05,813 - INFO - Updating release distribution type
2021-02-09 06:23:17,736 - ERROR - salt command failed, reason {​​​​​​'srvnode-1': 'Traceback (most recent call last):\n File "/usr/bin/provisioner", line 10, in \n from importlib.metadata import distribution\nModuleNotFoundError: No module named 'importlib.metadata'\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "/usr/bin/provisioner", line 13, in \n from importlib_metadata import distribution\nModuleNotFoundError: No module named 'importlib_metadata'\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "/usr/lib/python3.6/site-packages/pkg_resources/init.py", line 570, in _build_master\n ws.require(requires)\n File "/usr/lib/python3.6/site-packages/pkg_resources/init.py", line 888, in require\n needed = self.resolve(parse_requirements(requirements))\n File "/usr/lib/python3.6/site-packages/pkg_resources/init.py", line 779, in resolve\n raise VersionConflict(dist, req).with_context(dependent_req)\npkg_resources.ContextualVersionConflict: (salt 3001.1 (/usr/lib/python3.6/site-packages), Requirement.parse('salt>=3002.2'), {​​​​​​'cortx-prvsnr'}​​​​​​)\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "/usr/bin/provisioner", line 15, in \n from pkg_resources import load_entry_point\n File "/usr/lib/python3.6/site-packages/pkg_resources/init.py", line 3095, in \n @_call_aside\n File "/usr/lib/python3.6/site-packages/pkg_resources/init.py", line 3079, in _call_aside\n f(*args, **kwargs)\n File "/usr/lib/python3.6/site-packages/pkg_resources/init.py", line 3108, in _initialize_master_working_set\n working_set = WorkingSet._build_master()\n File "/usr/lib/python3.6/site-packages/pkg_resources/init.py", line 572, in _build_master\n return cls._build_from_requirements(requires)\n File "/usr/lib/python3.6/site-packages/pkg_resources/init.py", line 585, in _build_from_requirements\n dists = ws.resolve(reqs, Environment())\n File "/usr/lib/python3.6/site-packages/pkg_resources/init.py", line 774, in resolve\n raise DistributionNotFound(req, requirers)\npkg_resources.DistributionNotFound: The 'salt>=3002.2' distribution was not found and is required by cortx-prvsnr'}​​​​​​, args {​​​​​​'targets': 'srvnode-1', 'fun': 'cmd.run', 'fun_args': ['provisioner pillar_set --fpath release.sls release/type '"bundle"''], 'fun_kwargs': {​​​​​​}​​​​​​, 'kw': {​​​​​​'roster_file': '/root/.provisioner/srvnode-1_root@localhost_22/srv/config/roster', 'ssh_options': ['UserKnownHostsFile=/dev/null', 'StrictHostKeyChecking=no']}​​​​​​, 'secure': False}​​​​​​
2021-02-09 06:23:17,737 - ERROR - provisioner failed
{​​​​​​'srvnode-1': 'Traceback (most recent call last):\n File "/usr/bin/provisioner", line 10, in \n from importlib.metadata import distribution\nModuleNotFoundError: No module named 'importlib.metadata'\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "/usr/bin/provisioner", line 13, in \n from importlib_metadata import distribution\nModuleNotFoundError: No module named 'importlib_metadata'\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "/usr/lib/python3.6/site-packages/pkg_resources/init.py", line 570, in _build_master\n ws.require(requires)\n File "/usr/lib/python3.6/site-packages/pkg_resources/init.py", line 888, in require\n needed = self.resolve(parse_requirements(requirements))\n File "/usr/lib/python3.6/site-packages/pkg_resources/init.py", line 779, in resolve\n raise VersionConflict(dist, req).with_context(dependent_req)\npkg_resources.ContextualVersionConflict: (salt 3001.1 (/usr/lib/python3.6/site-packages), Requirement.parse('salt>=3002.2'), {​​​​​​'cortx-prvsnr'}​​​​​​)\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "/usr/bin/provisioner", line 15, in \n from pkg_resources import load_entry_point\n File "/usr/lib/python3.6/site-packages/pkg_resources/init.py", line 3095, in \n @_call_aside\n File "/usr/lib/python3.6/site-packages/pkg_resources/init.py", line 3079, in _call_aside\n f(*args, **kwargs)\n File "/usr/lib/python3.6/site-packages/pkg_resources/init.py", line 3108, in _initialize_master_working_set\n working_set = WorkingSet._build_master()\n File "/usr/lib/python3.6/site-packages/pkg_resources/init.py", line 572, in _build_master\n return cls._build_from_requirements(requires)\n File "/usr/lib/python3.6/site-packages/pkg_resources/init.py", line 585, in _build_from_requirements\n dists = ws.resolve(reqs, Environment())\n File "/usr/lib/python3.6/site-packages/pkg_resources/init.py", line 774, in resolve\n raise DistributionNotFound(req, requirers)\npkg_resources.DistributionNotFound: The 'salt>=3002.2' distribution was not found and is required by cortx-prvsnr'}​​​​​​

Git checkout takes too long in VM

Git checkout takes too long ( ~15 min ) in cloudform VM.

Priority : High

Observation

  • This issue can be reproducible in the time-frame of 6 AM - 10 AM IST (Not sure when it exactly starts).
  • This issue only occurs in the Cloudform VM
Checkout Log in Citrix VDI

  • Repo : cortx-csm

  • Timetaken : 0.23 m

Network Speed

curl -s https://raw.githubusercontent.com/sivel/speedtest-cli/master/speedtest.py | python -
Retrieving speedtest.net configuration...
Testing from Seagate Technology LLC (192.55.14.36)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Singtel (Singapore) [6.13 km]: 5.158 ms
Testing download speed................................................................................
Download: 642.32 Mbit/s
Testing upload speed......................................................................................................
Upload: 767.29 Mbit/s

Git Checkout Log

$ time GIT_CURL_VERBOSE=1 GIT_TRACE=1 git clone https://<PAT>@github.com/Seagate/cortx-cs
m.git
07:55:44.667551 exec-cmd.c:238          trace: resolved executable dir: C:/Program Files/Git/mingw64/bin
07:55:44.670553 git.c:439               trace: built-in: git clone https://<PAT>@github.com/Seagate/cortx-csm.git
Cloning into 'cortx-csm'...
07:55:45.094549 run-command.c:663       trace: run_command: git remote-https origin https://<PAT>@github.com/Seagate/cortx-csm.git
07:55:45.134548 exec-cmd.c:238          trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
07:55:45.141547 git.c:704               trace: exec: git-remote-https origin https://<PAT>@github.com/Seagate/cortx-csm.git
07:55:45.141547 run-command.c:663       trace: run_command: git-remote-https origin https://<PAT>@github.com/Seagate/cortx-csm.git
07:55:45.204547 exec-cmd.c:238          trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
* Couldn't find host github.com in the _netrc file; using defaults
*   Trying 52.74.223.119:443...
* TCP_NODELAY set
* Connected to github.com (52.74.223.119) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: C:/Program Files/Git/mingw64/ssl/certs/ca-bundle.crt
  CApath: none
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=GitHub, Inc.; CN=github.com
*  start date: May  5 00:00:00 2020 GMT
*  expire date: May 10 12:00:00 2022 GMT
*  subjectAltName: host "github.com" matched cert's "github.com"
*  issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert SHA2 High Assurance Server CA
*  SSL certificate verify ok.
> GET /Seagate/cortx-csm.git/info/refs?service=git-upload-pack HTTP/1.1
Host: github.com
User-Agent: git/2.22.0.windows.1
Accept: */*
Accept-Encoding: deflate, gzip
Pragma: no-cache

* old SSL session ID is stale, removing
* Mark bundle as not supporting multiuse
< HTTP/1.1 401 Authorization Required
< Server: GitHub Babel 2.0
< Content-Type: text/plain
< Content-Length: 21
< WWW-Authenticate: Basic realm="GitHub"
< X-Frame-Options: DENY
< X-GitHub-Request-Id: 5824:63DC:2DDF87A:3EB5973:5F39EAA9
<
* Ignoring the response-body
* Connection #0 to host github.com left intact
* Issue another request to this URL: 'https://<PAT>@github.com/Seagate/cortx-csm.git/info/refs?service=git-upload-pack'
* Couldn't find host github.com in the _netrc file; using defaults
* Found bundle for host github.com: 0x2dee370 [serially]
* Re-using existing connection! (#0) with host github.com
* Connected to github.com (52.74.223.119) port 443 (#0)
* Server auth using Basic with user '<PAT>'
> GET /Seagate/cortx-csm.git/info/refs?service=git-upload-pack HTTP/1.1
Host: github.com
Authorization: Basic ZTM5ZDI0MGIyYmI1NWE3MzkxOTMxOTg1ZDAyMDFmNzQ5ODk2NzdmNTo=
User-Agent: git/2.22.0.windows.1
Accept: */*
Accept-Encoding: deflate, gzip
Pragma: no-cache

* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Server: GitHub Babel 2.0
< Content-Type: application/x-git-upload-pack-advertisement
< Transfer-Encoding: chunked
< Expires: Fri, 01 Jan 1980 00:00:00 GMT
< Pragma: no-cache
< Cache-Control: no-cache, max-age=0, must-revalidate
< Vary: Accept-Encoding
< X-Frame-Options: DENY
< X-GitHub-Request-Id: 5824:63DC:2DDF8E0:3EB59EF:5F39EAAA
<
* Connection #0 to host github.com left intact
07:55:47.600703 run-command.c:663       trace: run_command: git fetch-pack --stateless-rpc --stdin --lock-pack --thin --check-self-contained-and-connected --cloning https://<PAT>@github.com/Seagate/cortx-csm.git/
07:55:47.659702 exec-cmd.c:238          trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
07:55:47.668701 git.c:439               trace: built-in: git fetch-pack --stateless-rpc --stdin --lock-pack --thin --check-self-contained-and-connected --cloning https://<PAT>@github.com/Seagate/cortx-csm.git/
* Couldn't find host github.com in the _netrc file; using defaults
* Found bundle for host github.com: 0x2dee370 [serially]
* Re-using existing connection! (#0) with host github.com
* Connected to github.com (52.74.223.119) port 443 (#0)
* Server auth using Basic with user '<PAT>'
> POST /Seagate/cortx-csm.git/git-upload-pack HTTP/1.1
Host: github.com
Authorization: Basic ZTM5ZDI0MGIyYmI1NWE3MzkxOTMxOTg1ZDAyMDFmNzQ5ODk2NzdmNTo=
User-Agent: git/2.22.0.windows.1
Accept-Encoding: deflate, gzip
Content-Type: application/x-git-upload-pack-request
Accept: application/x-git-upload-pack-result
Content-Encoding: gzip
Content-Length: 6224

* upload completely sent off: 6224 out of 6224 bytes
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Server: GitHub Babel 2.0
< Content-Type: application/x-git-upload-pack-result
< Transfer-Encoding: chunked
< Expires: Fri, 01 Jan 1980 00:00:00 GMT
< Pragma: no-cache
< Cache-Control: no-cache, max-age=0, must-revalidate
< Vary: Accept-Encoding
< X-Frame-Options: DENY
< X-GitHub-Request-Id: 5824:63DC:2DDF9FA:3EB5AF9:5F39EAAB
<
remote: Enumerating objects: 291, done.
remote: Counting objects: 100% (291/291), done.
remote: Compressing objects: 100% (213/213), done.
07:55:49.360686 run-command.c:663       trace: run_command: git index-pack --stdin -v --fix-thin '--keep=fetch-pack 22760 on sgspictvw2228' --check-self-contained-and-connected --pack_header=2,56013
07:55:49.435685 exec-cmd.c:238          trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
07:55:49.472199 git.c:439               trace: built-in: git index-pack --stdin -v --fix-thin '--keep=fetch-pack 22760 on sgspictvw2228' --check-self-contained-and-connected --pack_header=2,56013
* Connection #0 to host github.com left intactiB | 8.96 MiB/s
remote: Total 56013 (delta 132), reused 143 (delta 72), pack-reused 55722 eceiving objects: 100% (56013/56013), 40.56 MiB | 8Receiving objects: 100% (56013/56013), 41.69 MiB | 8.09 MiB/s, done.

Resolving deltas: 100% (25949/25949), done.
07:55:56.126192 run-command.c:663       trace: run_command: git rev-list --objects --stdin --not --all --quiet '--progress=Checking connectivity'
07:55:56.173192 exec-cmd.c:238          trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
07:55:56.183192 git.c:439               trace: built-in: git rev-list --objects --stdin --not --all --quiet '--progress=Checking connectivity'
Checking out files: 100% (686/686), done.

real    0m23.450s
user    0m0.000s
sys     0m0.031s

Checkout Log in Cloudform VM

  • Repo : cortx-csm

  • Timetaken : 14.17 m

Network Speed

[root@ssc-vm-0034 736373]# curl -s https://raw.githubusercontent.com/sivel/speedtest-cli/master/speedtest.py | python -
Retrieving speedtest.net configuration...
Testing from Seagate Technology LLC (134.204.222.36)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Nitel (Denver, CO) [48.60 km]: 35.627 ms
Testing download speed................................................................................
Download: 198.21 Mbit/s
Testing upload speed................................................................................................
Upload: 310.32 Mbit/s

Git Checkout Log

[root@4be3b7e549a9 /]# time GIT_CURL_VERBOSE=1 GIT_TRACE=1 git clone https://<PAT>@github.com/Seagate/cortx-csm.git
02:32:13.129992 git.c:371               trace: built-in: git 'clone' 'https://<PAT>@github.com/Seagate/cortx-csm.git'
Cloning into 'cortx-csm'...
02:32:13.133696 run-command.c:369       trace: run_command: 'git-remote-https' 'origin' 'https://<PAT>@github.com/Seagate/cortx-csm.git'
* Couldn't find host github.com in the .netrc file; using defaults
* About to connect() to github.com port 443 (#0)
*   Trying 140.82.114.4...
* Connected to github.com (140.82.114.4) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
*       subject: CN=github.com,O="GitHub, Inc.",L=San Francisco,ST=California,C=US
*       start date: May 05 00:00:00 2020 GMT
*       expire date: May 10 12:00:00 2022 GMT
*       common name: github.com
*       issuer: CN=DigiCert SHA2 High Assurance Server CA,OU=www.digicert.com,O=DigiCert Inc,C=US
> GET /Seagate/cortx-csm.git/info/refs?service=git-upload-pack HTTP/1.1
User-Agent: git/2.12.2
Host: github.com
Accept: */*
Accept-Encoding: gzip
Accept-Language: en-US, *;q=0.9
Pragma: no-cache

< HTTP/1.1 401 Authorization Required
< Server: GitHub Babel 2.0
< Content-Type: text/plain
< Content-Length: 21
< WWW-Authenticate: Basic realm="GitHub"
< X-GitHub-Request-Id: 2A98:2BC3:3987949:54AED2F:5F39EC2D
< X-Frame-Options: DENY
<
* Ignoring the response-body
* Connection #0 to host github.com left intact
* Issue another request to this URL: 'https://<PAT>@github.com/Seagate/cortx-csm.git/info/refs?service=git-upload-pack'
* Couldn't find host github.com in the .netrc file; using defaults
* Found bundle for host github.com: 0xad4a60
* Re-using existing connection! (#0) with host github.com
* Connected to github.com (140.82.114.4) port 443 (#0)
* Server auth using Basic with user '<PAT>'
> GET /Seagate/cortx-csm.git/info/refs?service=git-upload-pack HTTP/1.1
Authorization: Basic ZTM5ZDI0MGIyYmI1NWE3MzkxOTMxOTg1ZDAyMDFmNzQ5ODk2NzdmNTo=
User-Agent: git/2.12.2
Host: github.com
Accept: */*
Accept-Encoding: gzip
Accept-Language: en-US, *;q=0.9
Pragma: no-cache

< HTTP/1.1 200 OK
< Server: GitHub Babel 2.0
< Content-Type: application/x-git-upload-pack-advertisement
< Transfer-Encoding: chunked
< Expires: Fri, 01 Jan 1980 00:00:00 GMT
< Pragma: no-cache
< Cache-Control: no-cache, max-age=0, must-revalidate
< Vary: Accept-Encoding
< X-GitHub-Request-Id: 2A98:2BC3:39879CA:54AED5C:5F39EC2D
< X-Frame-Options: DENY
<
* Connection #0 to host github.com left intact
02:32:15.583453 run-command.c:369       trace: run_command: 'fetch-pack' '--stateless-rpc' '--stdin' '--lock-pack' '--thin' '--check-self-contained-and-connected' '--cloning' 'https://<PAT>@github.com/Seagate/cortx-csm.git/'
02:32:15.584471 exec_cmd.c:118          trace: exec: 'git' 'fetch-pack' '--stateless-rpc' '--stdin' '--lock-pack' '--thin' '--check-self-contained-and-connected' '--cloning' 'https://<PAT>@github.com/Seagate/cortx-csm.git/'
02:32:15.586117 git.c:371               trace: built-in: git 'fetch-pack' '--stateless-rpc' '--stdin' '--lock-pack' '--thin' '--check-self-contained-and-connected' '--cloning' 'https://<PAT>@github.com/Seagate/cortx-csm.git/'
* Couldn't find host github.com in the .netrc file; using defaults
* Found bundle for host github.com: 0xad4a60
* Re-using existing connection! (#0) with host github.com
* Connected to github.com (140.82.114.4) port 443 (#0)
* Server auth using Basic with user '<PAT>'
> POST /Seagate/cortx-csm.git/git-upload-pack HTTP/1.1
Authorization: Basic ZTM5ZDI0MGIyYmI1NWE3MzkxOTMxOTg1ZDAyMDFmNzQ5ODk2NzdmNTo=
User-Agent: git/2.12.2
Host: github.com
Accept-Encoding: gzip
Content-Type: application/x-git-upload-pack-request
Accept: application/x-git-upload-pack-result
Content-Encoding: gzip
Content-Length: 6214

* upload completely sent off: 6214 out of 6214 bytes
< HTTP/1.1 200 OK
< Server: GitHub Babel 2.0
< Content-Type: application/x-git-upload-pack-result
< Transfer-Encoding: chunked
< Expires: Fri, 01 Jan 1980 00:00:00 GMT
< Pragma: no-cache
< Cache-Control: no-cache, max-age=0, must-revalidate
< Vary: Accept-Encoding
< X-GitHub-Request-Id: 2A98:2BC3:3987A2E:54AEE1E:5F39EC2E
< X-Frame-Options: DENY
<
remote: Enumerating objects: 291, done.
remote: Counting objects: 100% (291/291), done.
remote: Compressing objects: 100% (213/213), done.
02:32:16.435083 run-command.c:369       trace: run_command: 'index-pack' '--stdin' '-v' '--fix-thin' '--keep=fetch-pack 53 on 4be3b7e549a9' '--check-self-contained-and-connected' '--pack_header=2,56013'
02:32:16.436121 exec_cmd.c:118          trace: exec: 'git' 'index-pack' '--stdin' '-v' '--fix-thin' '--keep=fetch-pack 53 on 4be3b7e549a9' '--check-self-contained-and-connected' '--pack_header=2,56013'
02:32:16.437700 git.c:371               trace: built-in: git 'index-pack' '--stdin' '-v' '--fix-thin' '--keep=fetch-pack 53 on 4be3b7e549a9' '--check-self-contained-and-connected' '--pack_header=2,56013'
* Connection #0 to host github.com left intactiB | 74.00 KiB/s
remote: Total 56013 (delta 132), reused 143 (delta 72), pack-reused 55722
Receiving objects: 100% (56013/56013), 41.69 MiB | 68.00 KiB/s, done.
Resolving deltas: 100% (25949/25949), done.
02:46:30.154135 run-command.c:369       trace: run_command: 'rev-list' '--objects' '--stdin' '--not' '--all' '--quiet' '--progress=Checking connectivity'
02:46:30.154769 exec_cmd.c:118          trace: exec: 'git' 'rev-list' '--objects' '--stdin' '--not' '--all' '--quiet' '--progress=Checking connectivity'
02:46:30.156481 git.c:371               trace: built-in: git 'rev-list' '--objects' '--stdin' '--not' '--all' '--quiet' '--progress=Checking connectivity'

real    14m17.110s
user    0m3.559s
sys     0m1.760s

CORTX-28330: Share OVA build process with community

Can we share the scripts and documentations to allow community members to build their own OVA images from the source code? If so, this would help the community a LOT because developers could have an easy way to test their changes.

RE Base Docker Image Naming Conventions

We need to update our base docker image with correct versioning to support future activities. These docker images are acts as a build environment for cortx components.

Consider

  • Docker image names cortx, cortx_base, cortx_core, cortx_ will be reserved for future cortx docker workflow.
  • We need to follow global docker image tagging standards before publishing the image.
  • This docker image is not a application image so component build version number don't play any role in this naming process.

Suggested Docker image naming pattern

  • If we plan to use single docker image for the build process

    • cortx_build_env : <os_name>-<os_version>
  • If we plan to use multiple docker image

    • cortx_base_build_env : <os_name>-<os_version>
    • cortx_core_build_env : <os_name>-<os_version>

Note : docker image name cortx, cortx_base, cortx_core will be reserved for future cortx docker workflow

Example

cortx_build_env:centos-7.7.1908
cortx_build_env:centos-7.8.2003
cortx_build_env:centos-8.0.1905 
cortx_build_env:rhel-7.7
cortx_build_env:rhel-7.8
...

How others doing

Build environment docker images mostly not available in public domain anyhow added some application image naming patterns here

minio/minio:RELEASE.2020-09-26T03-44-56Z-s390x

postgres:13-alpine

maven:3.6.3-ibmjava-alpine

sonarqube:8.4.2-community

telegraf:1.14.5-alpine

jenkins:2.60.2-alpine

kapacitor:1.5.5-alpine

@shailesh-vaidya please provide your suggestion on this

cortx-1.0.0-XXX-single.iso size increased from 1.7G to 3.2G

Priyank reported a issue that starting from build 519, cortx-1.0.0-XXX-single.iso size increased from 1.7G to 3.2G.

We need to perform below 2 things to address the issue

  1. Fix the size issue in cortx-1.0 release pipeline
  2. QA already started using Build #531, #532 so regenerate and replace the single ISO with size fix before sharing the ISO for factory release.

Improve manual PR validation Jenkins pipelines

Due to security issues we need to disable multi-branch job's for components. We have created have Jenkins pipelines with manual trigger for validation at http://eos-jenkins.colo.seagate.com/job/Cortx-PR-Build/
However this will be additional step for Seagate developers also for PR. Currently Jenkins pipelines take repo url and branch names as input to trigger job.

As discussed with @gowthamchinna . We can improve it further to make it easy.

  1. Explore https://github.com/jenkinsci/ghprb-plugin#github-pull-request-builder-plugin . It has functionalities like trigger by PR comment from specified person, Add approval before code cloning etc.
  2. We can add comment in PR after manually triggered job is complete.
  3. Just take PR url as input, clone code based on it and run test cases.

RELEASE.INFO file shows OS: "Red Hat Enterprise Linux Server release 7.8 (Maipo)"

Latest build RELEASE.INFO shows OS: "Red Hat Enterprise Linux Server release 7.8 (Maipo)" is should be Red Hat Enterprise Linux Server release 7.7 (Maipo)

AFAIK we are using Dockerfile to build Base Image and rhel7-init:7.7 has 7.7 version.

[root@ssc-vm-0245 ~]# docker run -it registry.access.redhat.com/rhel7-init:7.7 cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.7 (Maipo)
[root@ssc-vm-0245 ~]#

Please verify and correct.

Opensource integrated CI

We need to create mechanism to build integrated build for all CORTX components. Same is requested in Seagate/cortx#204 . This ticket is to track development for this.

Plan -

  • Create Docker image with required dependency packages and configuration for building CORTX component.
  • Image will have Makefile which contains targets/steps to build components packages.
  • End user will clone https://github.com/Seagate/cortx repo and execute docker command like docker run -v /var/artifacts:/var/artifacts -v /root/cortx:/cortx-workspace cortx_build:7.7.1908 make clean cortx-prvsnr
  • After successful execution rpm's will be available at /var/artifacts/

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.