microsoft / lisa Goto Github PK

View Code? Open in Web Editor NEW

174.0 30.0 154.0 23.81 MB

LISA is developed and maintained by Microsoft, to empower Linux validation.

Home Page: https://aka.ms/lisa

License: MIT License

Shell 0.33% Python 99.08% Java 0.01% Dockerfile 0.02% PowerShell 0.12% C 0.04% Bicep 0.40%

linux automation-framework azure testing hyperv cloudtesting linux-compatibility e2e-testing

lisa's Introduction

Linux Integration Services Automation (LISA)

Linux Integration Services Automation (LISA) is a Linux quality validation system, which consists of two parts：

A test framework to drive test execution.
A set of test suites to verify Linux kernel/distribution quality.

LISA was originally designed and implemented for Microsoft Azure and Windows HyperV platforms; now it can be used to validate Linux quality on any platforms if the proper orchestrator module is implemented.

Why LISA

Scalable: Benefit from the appropriate abstractions, LISA can be used to test the quality of numerous Linux distributions without duplication of code implementation.
Customizable: The test suites created on top of LISA can be customized to support different quality validation needs.
Support multiple platforms: LISA is created with modular design, to support various of Linux platforms including Microsoft Azure, Windows HyperV, Linux bare metal, and other cloud based platforms.
End-to-end: LISA supports platform specific orchestrators to create and delete test environment automatically; it also provides flexibility to preserve environments for troubleshooting if test(s) fails.

Documents

Contribute

You are very welcome to contribute to this repository. Please follow the contribution document for details.

Road map

LISA is in active developing, and a lot of exciting features are being implemented. We're listening to your feedback.

License

The entire codebase is under MIT license.

lisa's People

Contributors

Stargazers

Watchers

Forkers

lilideng iamshital sireeshapotnuru adriansuhov ader1990 chvalean stefanlupsa glewis9999 xian123 lixuemin2016 tong-xu v-chmett divyajyothikasam johnsongeorge-w leifei87 v-sijliu lizzha seansp-zz prashantmhetar huijuanzhao huijinghei yuxisun1217 zlin888 zhaohuijuan andyleejordan emalkan abhimarathe gaohan-source lubaihua33 squirrelsc shiningcreek harz566 vinaykotak arbal juhlee-microsoft santoshx senugala adrianamusic global-localhost global19 global19-atlassian-net tryweirder glima mcgov lesscodingmorehappiness dsrivastavv sunnyguoqq francis-liu pupacha ksenijas aasim-syed qpc-database hpourreza wenyiq slowy07 abdelrahman0w moustafa-moustafa classicvalues pohkengintel kamalca standardgalactic marmikreal srikkanth rashirg acidburn0zzz cwize1 tyhicks ketaminekid423 vyadavmsft anirudhrb srveniga kudzip matthewtorr-msft shruti25ratnam python-repository-hub saurabh-sengar physolia ibanknatoprad souradeep100 rainierside playfloor 00mjk shradhagupta5 arjun-sharda-contribution-forks snosratiershad test-mass-forker-org-1 tonybaloney 0xba1a adityagesh jose-teuttli rakshaa2000 munyite001 avylove frmunozp r-dailey ellie-di darkwhite29 narrieta norakoiralamsft rohitrawatazure

lisa's Issues

The Is-VmAlive retry count of 20 is too small on custom kernel installs on VMs with multiple nics on Azure

On Azure, the VMs that have multiple nics take a longer time to reboot and enable the networking stack, making the Is-VMAlive function fail.

The function has a retry count of 20 as default and a 3 second timeout between retries (around 60 seconds).

We should increase the default retry 50 to cater for the lazy networking. This increase should not affect kernel panics check, as that scenario is handled independently.

Channel information in big VM has limit, 64

Test case: ETHTOOL-GET-SET-CHANNEL

LISAv2-OneVM-juhlee-mv2-KS42-20190517091637-role-0:/home/lisa # ethtool -l eth0
Channel parameters for eth0:
Pre-set maximums:
RX: 0
TX: 0
Other: 0
Combined: 64
Current hardware settings:
RX: 0
TX: 0
Other: 0
Combined: 8

For bigger VM size, this is not correct. No information of 'hw settings' but 'pre-set max' value is 64 as limited. #define VRSS_CHANNEL_MAX 64
My test shows 'Expected: 208 channels and actual 8.' but the max is 64 and pre-set max displays '64'.

Expected result: if the channels is more than 64, then it is 64 as max.

Azure: Do not wait for RG cleanup

Remove the code for waiting for resource clean up on Azure, as it takes too much time (around 5 minutes of test time).

Handle obsolete packages in RHEL 8.x

obsolete packages - rhel8 has a long list of obsolete packages, automation is failing due to not finding them (We found this in 8.0 GA testing, and fix in the package update call)

dpdk version is empty on v19.02

The dpdk version was retrieved from the file pkg/dpdk.spec, which was removed in the last stable version 19.02 by this commit:
DPDK/dpdk@36dd40a

A fix is needed, as the results uploaded to the db do not have a version and those results cannot be plotted in power bi.

VERIFY-VHD-PREREQUISITES: Fix compatibility with redhat8

Test fails on redhat8:

[Azure VERIFY-VHD-PREREQUISITES eastus2] 05/24/2019 10:07:04 : [INFO ] REDHAT_TEST_SUDOERS_VERIFICATION_SUCCESS
[Azure VERIFY-VHD-PREREQUISITES eastus2] 05/24/2019 10:07:04 : [INFO ] REDHAT_TEST_NETWORK_MANAGER_NOT_INSTALLED
[Azure VERIFY-VHD-PREREQUISITES eastus2] 05/24/2019 10:07:04 : [INFO ] REDHAT_TEST_NETWORK_FILE_SUCCESS
[Azure VERIFY-VHD-PREREQUISITES eastus2] 05/24/2019 10:07:04 : [INFO ] REDHAT_TEST_IFCFG_ETH0_FILE_SUCCESS
[Azure VERIFY-VHD-PREREQUISITES eastus2] 05/24/2019 10:07:04 : [INFO ] REDHAT_TEST_UDEV_RULES_SUCCESS
**[Azure VERIFY-VHD-PREREQUISITES eastus2] 05/24/2019 10:07:04 : [ERROR] Expected String : REDHAT_TEST_GRUB_VERIFICATION_SUCCESS not present. Please check logs.**
[Azure VERIFY-VHD-PREREQUISITES eastus2] 05/24/2019 10:07:04 : [INFO ] REDHAT_TEST_RHUIREPOSITORIES_AVAILABLE
[Azure VERIFY-VHD-PREREQUISITES eastus2] 05/24/2019 10:07:04 : [INFO ] Test Status : Completed
[Azure VERIFY-VHD-PREREQUISITES eastus2] 05/24/2019 10:07:04 : [INFO ] Test Result : FAIL

05/24/2019 10:06:57 AM : INFO : Checking console=ttyS0 rootdelay=300..
05/24/2019 10:06:57 AM : INFO : Getting Contents of /boot/grub2/grub.cfg
...
05/24/2019 10:06:57 AM : ERROR : console=ttyS0 not present
05/24/2019 10:06:57 AM : ERROR : rootdelay=300 not present

HyperV, LTP: use VHDx instead of VHD for test disk (for both Gen1 and Gen2)

Current test definition is adding a VHD disk to be used during LTP, this does not work however for Gen2VM for example.
We can easily use the other setupScript AddVhdxHardDisk/RemoveVhdxHardDisk which supports VHDx and is independent of VM generation.

@LiliDeng @lizzha @mbivolan @ader1990 @iamshital @juhlee-microsoft please let me know if you make use of VHD in this TC by any chance.

Reference:
https://github.com/LIS/LISAv2/blob/master/XML/TestCases/CommunityTests.xml#L70

UseExistingRG param shall support even if no LoadBalancer exists

Sometimes LoadBalancer does not exist in RG, so it blocks automation run-time with '-UseExistingRG'. Suggest to create a new LoadBalancer if none is there.

LISAv2 archives the LIS archive on custom LIS install => creating log archive of more than 7GB

[P0] Test passes even if sub-test is failed

   ID TestArea             TestCaseName                                                                TestResult TestDuration(in minutes) 
-------------------------------------------------------------------------------------------------------------------------------------------
    1 GPU                  NVIDIA-CUDA-DRIVER-VALIDATION-Standard_NC6                                        PASS                11.34 
	Using nVidia driver : CUDA 
	lsvmbus: Expected "PCI Express pass-through" count: 1, count inside the VM: 1 : PASS 
	lspci: Expected "3D controller: NVIDIA Corporation" count: 1, found inside the VM: 0 : FAIL 
	lshw: Expected Display adapters: 1, total adapters found in VM: 0 : FAIL 
	nvidia-smi: Expected GPU count: 1, found inside the VM: 0 : FAIL 
    2 GPU                  NVIDIA-CUDA-DRIVER-VALIDATION-Standard_NC12                                       PASS                10.16 
	Using nVidia driver : CUDA 
	lsvmbus: Expected "PCI Express pass-through" count: 2, count inside the VM: 2 : PASS 
	lspci: Expected "3D controller: NVIDIA Corporation" count: 2, found inside the VM: 0 : FAIL 
	lshw: Expected Display adapters: 2, total adapters found in VM: 0 : FAIL 
	nvidia-smi: Expected GPU count: 2, found inside the VM: 0 : FAIL 
    3 GPU                  NVIDIA-CUDA-DRIVER-VALIDATION-Standard_NC24                                       PASS                 9.52 
	Using nVidia driver : CUDA 
	lsvmbus: Expected "PCI Express pass-through" count: 4, count inside the VM: 4 : PASS 
	lspci: Expected "3D controller: NVIDIA Corporation" count: 4, found inside the VM: 0 : FAIL 
	lshw: Expected Display adapters: 4, total adapters found in VM: 0 : FAIL 
	nvidia-smi: Expected GPU count: 4, found inside the VM: 0 : FAIL

Add additional verification point with md5sum before/after file-copy

as well, can you also do md5sum before/after file-copy?

In FCOPY-NON-ASCII.ps1 we do md5sum for the file.
It might go as a separate work item to maybe compute md5sum for all fcopy tests, but not in this PR.

Originally posted by @chvalean in #175

package installation update for file system tests (RHEL 8.x)

pass after manual changes. missing dependencies in rhel8: indent and llvm-ocaml-devel and btrfs-progs-devel and python (replaced with python36)

LISAv2 summary is missing some aborted tests

In below logs, Test ID 17 and 18 are missing.

[LISAv2 Test Results Summary]
Test Run On           : 04/30/2019 14:43:23
ARM Image Under Test  : Canonical : UbuntuServer : 16.04-LTS : latest
Test Category         : Functional
Test Area             : CORE
Total Test Cases      : 20 (18 Passed, 0 Failed, 2 Aborted, 0 Skipped)
Total Time (dd:hh:mm) : 0:1:4
   ID TestArea             TestCaseName                                                                TestResult TestDuration(in minutes)
-------------------------------------------------------------------------------------------------------------------------------------------
    1 CORE                 TIME-CLOCKEVENT-SMP                                                               PASS                 0.53
    2 CORE                 RELOAD-MODULES-SMP                                                                PASS                11.02
    3 CORE                 LSVMBUS                                                                           PASS                 0.52
    4 CORE                 VERIFY-DEPLOYMENT-PROVISION                                                       PASS                 2.32
        FirstBoot : PASS
        FirstBoot : Call Trace Verification : PASS
        Reboot : PASS
        Reboot : Call Trace Verification : PASS
    5 CORE                 VERIFY-LINUX-CONFIGURATION                                                        PASS                 0.63
    6 CORE                 LIS-MODULES-CHECK                                                                 PASS                 0.64
    7 CORE                 VERIFY-LIS-MODULES-VERSION                                                        PASS                 0.51
    9 CORE                 VERIFY-VHD-PREREQUISITES                                                          PASS                 0.96
   10 CORE                 TIMESYNC-NTP                                                                      PASS                 0.89
   11 CORE                 TIME-CLOCKSOURCE                                                                  PASS                 0.56
   12 CORE                 TIME-CLOCKEVENT-UP                                                                PASS                 0.53
   13 CORE                 INITRD-MODULES-CHECK                                                              PASS                 1.33
   14 CORE                 CPU-VERIFY-ONLINE                                                                 PASS                 0.53
   15 CORE                 TIMESYNC-BASIC                                                                    PASS                 0.89
   16 CORE                 VERIFY-LINUX-DISK-SETUP                                                           PASS                 0.59
        Reboot : Call Trace Verification : PASS
   19 CORE                 VERIFY-DEPLOYMENT-PROVISION-EPHEMERAL-MANAGED-DISK                             Aborted                    0
   20 CORE                 L3-CACHE-CHECK                                                                    PASS                 0.54

[P0] Upload timeout is causing big delay in aborting test

05/05/2019 21:34:40 : [INFO ] dos2unix: converting file C:\LISAv2\VM82\TestResults\2019-05-05-14-08-02-7037\VMBUS_PANIC_NOTIFIER\runtest.sh to Unix format...

05/05/2019 21:34:40 : [INFO ] Uploading C:\LISAv2\VM82\TestResults\2019-05-05-14-08-02-7037\VMBUS_PANIC_NOTIFIER\runtest.sh to lisa : 10.128.72.199, port 22 using Password authentication

05/05/2019 21:44:40 : [ERROR] Upload Timout!

05/05/2019 21:44:49 : [WARN ] Error in upload, attempt 1, retrying

05/05/2019 21:44:59 : [INFO ] Uploading C:\LISAv2\VM82\TestResults\2019-05-05-14-08-02-7037\VMBUS_PANIC_NOTIFIER\runtest.sh to lisa : 10.128.72.199, port 22 using Password authentication

05/05/2019 21:54:59 : [ERROR] Upload Timout!

05/05/2019 21:55:06 : [WARN ] Error in upload, attempt 2, retrying

05/05/2019 21:55:17 : [INFO ] Uploading C:\LISAv2\VM82\TestResults\2019-05-05-14-08-02-7037\VMBUS_PANIC_NOTIFIER\runtest.sh to lisa : 10.128.72.199, port 22 using Password authentication

05/05/2019 22:05:18 : [ERROR] Upload Timout!

05/05/2019 22:05:26 : [WARN ] Error in upload, attempt 3, retrying

05/05/2019 22:05:36 : [INFO ] Uploading C:\LISAv2\VM82\TestResults\2019-05-05-14-08-02-7037\VMBUS_PANIC_NOTIFIER\runtest.sh to lisa : 10.128.72.199, port 22 using Password authentication

05/05/2019 22:15:37 : [ERROR] Upload Timout!

05/05/2019 22:15:46 : [WARN ] Error in upload, attempt 4, retrying

05/05/2019 22:15:56 : [INFO ] Uploading C:\LISAv2\VM82\TestResults\2019-05-05-14-08-02-7037\VMBUS_PANIC_NOTIFIER\runtest.sh to lisa : 10.128.72.199, port 22 using Password authentication

05/05/2019 22:25:56 : [ERROR] Upload Timout!

05/05/2019 22:26:03 : [WARN ] Error in upload, attempt 5, retrying

05/05/2019 22:26:13 : [INFO ] Uploading C:\LISAv2\VM82\TestResults\2019-05-05-14-08-02-7037\VMBUS_PANIC_NOTIFIER\runtest.sh to lisa : 10.128.72.199, port 22 using Password authentication

05/05/2019 22:36:14 : [ERROR] Upload Timout!

05/05/2019 22:36:22 : [WARN ] Error in upload, attempt 6, retrying

05/05/2019 22:36:32 : [INFO ] Uploading C:\LISAv2\VM82\TestResults\2019-05-05-14-08-02-7037\VMBUS_PANIC_NOTIFIER\runtest.sh to lisa : 10.128.72.199, port 22 using Password authentication

05/05/2019 22:46:33 : [ERROR] Upload Timout!

05/05/2019 22:46:42 : [WARN ] Error in upload, attempt 7, retrying

05/05/2019 22:46:52 : [INFO ] Uploading C:\LISAv2\VM82\TestResults\2019-05-05-14-08-02-7037\VMBUS_PANIC_NOTIFIER\runtest.sh to lisa : 10.128.72.199, port 22 using Password authentication

05/05/2019 22:56:53 : [ERROR] Upload Timout!

05/05/2019 22:57:00 : [WARN ] Error in upload, attempt 8, retrying

05/05/2019 22:57:10 : [INFO ] Uploading C:\LISAv2\VM82\TestResults\2019-05-05-14-08-02-7037\VMBUS_PANIC_NOTIFIER\runtest.sh to lisa : 10.128.72.199, port 22 using Password authentication

05/05/2019 23:07:11 : [ERROR] Upload Timout!

05/05/2019 23:07:19 : [WARN ] Error in upload, attempt 9, retrying

05/05/2019 23:07:29 : [INFO ] Uploading C:\LISAv2\VM82\TestResults\2019-05-05-14-08-02-7037\VMBUS_PANIC_NOTIFIER\runtest.sh to lisa : 10.128.72.199, port 22 using Password authentication

05/05/2019 23:17:30 : [ERROR] Upload Timout!

05/05/2019 23:17:38 : [INFO ] Calling function - Upload-RemoteFile. Error in upload after 10 attempt, hence giving up

05/05/2019 23:17:38 : [INFO ] dos2unix: converting file .\Testscripts\Linux\CollectLogFile.sh to Unix format...

05/05/2019 23:17:38 : [INFO ] Uploading .\Testscripts\Linux\CollectLogFile.sh to lisa : 10.128.72.199, port 22 using Password authentication

05/05/2019 23:27:38 : [ERROR] Upload Timout!

05/05/2019 23:27:46 : [WARN ] Error in upload, attempt 1, retrying

05/05/2019 23:27:56 : [INFO ] Uploading .\Testscripts\Linux\CollectLogFile.sh to lisa : 10.128.72.199, port 22 using Password authentication

05/05/2019 23:37:57 : [ERROR] Upload Timout!

05/05/2019 23:38:05 : [WARN ] Error in upload, attempt 2, retrying

05/05/2019 23:38:15 : [INFO ] Uploading .\Testscripts\Linux\CollectLogFile.sh to lisa : 10.128.72.199, port 22 using Password authentication

05/05/2019 23:48:16 : [ERROR] Upload Timout!

05/05/2019 23:48:25 : [WARN ] Error in upload, attempt 3, retrying

05/05/2019 23:48:35 : [INFO ] Uploading .\Testscripts\Linux\CollectLogFile.sh to lisa : 10.128.72.199, port 22 using Password authentication

05/05/2019 23:58:36 : [ERROR] Upload Timout!

05/05/2019 23:58:43 : [WARN ] Error in upload, attempt 4, retrying

05/05/2019 23:58:53 : [INFO ] Uploading .\Testscripts\Linux\CollectLogFile.sh to lisa : 10.128.72.199, port 22 using Password authentication

05/06/2019 00:08:53 : [ERROR] Upload Timout!

05/06/2019 00:09:02 : [ERROR] EXCEPTION : Calling function - Upload-RemoteFile. Error in upload after 5 attempt, hence giving up

05/06/2019 00:09:02 : [ERROR] Source : Line 125 in script .\Libraries\TestHelpers.psm1.

05/06/2019 00:09:02 : [ERROR] Ignorable error in collecting final data from VMs.

05/06/2019 00:09:02 : [INFO ] Start to do clean up for case VMBUS_PANIC_NOTIFIER

05/06/2019 00:09:02 : [INFO ] Collecting logs from IP : 10.128.72.199 PORT : 22

05/06/2019 00:09:02 : [INFO ] dos2unix: converting file .\Testscripts\Linux\CORE-LogCollector.sh to Unix format...

05/06/2019 00:09:02 : [INFO ] Uploading .\Testscripts\Linux\CORE-LogCollector.sh to lisa : 10.128.72.199, port 22 using Password authentication

05/06/2019 00:19:03 : [ERROR] Upload Timout!

05/06/2019 00:19:12 : [WARN ] Error in upload, attempt 1, retrying

05/06/2019 00:19:22 : [INFO ] Uploading .\Testscripts\Linux\CORE-LogCollector.sh to lisa : 10.128.72.199, port 22 using Password authentication

05/06/2019 00:29:22 : [ERROR] Upload Timout!

05/06/2019 00:29:29 : [WARN ] Error in upload, attempt 2, retrying

05/06/2019 00:29:39 : [INFO ] Uploading .\Testscripts\Linux\CORE-LogCollector.sh to lisa : 10.128.72.199, port 22 using Password authentication

05/06/2019 00:39:40 : [ERROR] Upload Timout!

05/06/2019 00:39:49 : [WARN ] Error in upload, attempt 3, retrying

05/06/2019 00:39:59 : [INFO ] Uploading .\Testscripts\Linux\CORE-LogCollector.sh to lisa : 10.128.72.199, port 22 using Password authentication

05/06/2019 00:50:00 : [ERROR] Upload Timout!

05/06/2019 00:50:10 : [WARN ] Error in upload, attempt 4, retrying

05/06/2019 00:50:20 : [INFO ] Uploading .\Testscripts\Linux\CORE-LogCollector.sh to lisa : 10.128.72.199, port 22 using Password authentication

05/06/2019 01:00:21 : [ERROR] Upload Timout!

05/06/2019 01:00:29 : [WARN ] Error in upload, attempt 5, retrying

05/06/2019 01:00:39 : [INFO ] Uploading .\Testscripts\Linux\CORE-LogCollector.sh to lisa : 10.128.72.199, port 22 using Password authentication

05/06/2019 01:10:40 : [ERROR] Upload Timout!

05/06/2019 01:10:48 : [WARN ] Error in upload, attempt 6, retrying

05/06/2019 01:10:58 : [INFO ] Uploading .\Testscripts\Linux\CORE-LogCollector.sh to lisa : 10.128.72.199, port 22 using Password authentication

05/06/2019 01:20:59 : [ERROR] Upload Timout!

05/06/2019 01:21:08 : [WARN ] Error in upload, attempt 7, retrying

05/06/2019 01:21:18 : [INFO ] Uploading .\Testscripts\Linux\CORE-LogCollector.sh to lisa : 10.128.72.199, port 22 using Password authentication

05/06/2019 01:31:19 : [ERROR] Upload Timout!

05/06/2019 01:31:26 : [WARN ] Error in upload, attempt 8, retrying

05/06/2019 01:31:37 : [INFO ] Uploading .\Testscripts\Linux\CORE-LogCollector.sh to lisa : 10.128.72.199, port 22 using Password authentication

05/06/2019 01:41:38 : [ERROR] Upload Timout!

05/06/2019 01:41:46 : [WARN ] Error in upload, attempt 9, retrying

05/06/2019 01:41:56 : [INFO ] Uploading .\Testscripts\Linux\CORE-LogCollector.sh to lisa : 10.128.72.199, port 22 using Password authentication

05/06/2019 01:51:56 : [ERROR] Upload Timout!

05/06/2019 01:52:05 : [ERROR] EXCEPTION : Calling function - Upload-RemoteFile. Error in upload after 10 attempt, hence giving up

[P0] Get-IPv4ViaKVP : 8 hours delay in aborting single test


05/05/2019 11:51:45 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 11:56:57 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 12:02:08 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 12:07:19 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 12:12:30 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 12:17:41 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 12:22:52 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 12:28:03 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 12:33:13 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 12:38:25 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 12:43:35 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 12:48:46 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 12:53:57 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 12:59:08 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 13:04:19 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 13:09:30 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 13:14:41 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 13:19:52 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 13:25:03 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 13:30:14 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 13:35:25 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 13:40:36 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 13:45:47 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 13:50:58 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 13:56:09 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 14:01:20 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 14:06:31 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 14:11:42 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 14:16:53 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 14:22:04 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 14:27:15 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 14:32:26 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 14:37:37 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 14:42:48 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 14:47:59 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 14:53:10 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 14:58:21 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 15:03:32 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 15:08:43 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 15:13:54 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 15:19:05 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 15:24:16 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 15:29:27 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 15:34:38 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 15:39:49 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 15:45:00 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 15:50:11 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 15:55:22 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 16:00:33 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 16:05:44 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 16:10:55 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 16:16:06 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 16:21:17 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 16:26:28 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 16:31:39 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 16:36:50 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 16:42:01 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 16:47:12 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 16:52:23 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 16:57:34 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 17:02:45 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 17:07:56 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 17:13:07 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 17:18:18 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 17:23:29 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 17:28:40 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 17:33:51 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 17:39:02 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 17:44:13 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 17:49:24 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 17:54:35 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 17:59:46 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 18:04:57 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 18:10:08 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 18:15:19 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 18:20:30 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 18:25:41 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 18:30:52 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 18:36:03 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 18:41:14 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 18:46:25 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 18:51:36 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 18:56:47 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 19:01:58 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 19:07:09 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 19:12:20 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 19:17:31 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 19:22:42 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 19:27:53 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 19:33:04 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 19:38:15 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 19:43:26 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 19:48:37 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 19:53:48 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 19:58:59 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 20:04:10 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 20:09:21 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 20:14:32 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 20:19:43 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 20:24:54 : [WARN ] Get-IPv4ViaKVP: No IPv4 address found for VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0

05/05/2019 20:25:04 : [ERROR] Wait-ForVMToStartKVP: VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0 did not start KVP within timeout period (1000)

05/05/2019 20:25:04 : [ERROR] Get-IPv4AndWaitForSSHStart: Unable to get ipv4 from VM LISAv2-OneVM-centos_7.2_x64-d-284-EW29-636926271935-role-0 via KVP within timeout period (1000)

SLES 15: Fix log collection

When running lisav2 with SUSE 15 image:

Collecting Operating System Logs.....

Welcome to SUSE Linux Enterprise Server 15  (x86_64) - Kernel \r (\l).

eth0: \4{eth0} \6{eth0}
Operating system Log process finished...
Collecting Waagent Details....
./CORE-LogCollector.sh: line 11: /usr/sbin/waagent: No such file or directory
cat: /var/log/waagent.log: No such file or directory
Collecting Microsoft Linux Integration Service Data..

better handle dhclient commands

SLES15 local Hyper-V image does not include the dhclient tool.
Automation assumes incorrectly that dhclient is present in an image, when it can surely happen that any distro might not have it by default.

This makes teste that rely on it like BVT-NET-IFUP-IFDOWN or BVT-CORE-RELOAD-MODULES to fail, but this is a false-positive failure as the tool is missing, not that it returns an error that is checked.
Must update scripts that make use of it to be fully compatible with SLES15.

Reference:
https://github.com/LIS/LISAv2/search?q=dhclient&unscoped_q=dhclient

@juhlee-microsoft for ack, we need a task for 1904.

Post VM deployment failures are not captured in telemetry database

Targetted issues -
Azure:

OSProvisioningTimedOut

Why this is required?
When running thousands of tests, it is not possible to go through the console logs to check all the results.
So we rely on power-bi reports / junit captured results.
But, the trivial error - 'OSProvisioningTimedOut' is not making it's way to Telemetry DB. So this gives a false sense of security.

LISAv2 to support running a test with multiple VM sizes (OverrideVMSize accepts multiple values)

Some tests are naming like <TC_name-VMSize>. A single test clones to multiple because of VM sizes.

Suggestion: Keep the single test case in the repo but add a new param, 'AllowedVMSize' in TC xml file. This variable is an array of supported VM sizes, so that the framework can filter out if this run-time is correct or not, depending on the result of AllowedVMSize filtering. This param is optional.

AllowedVMSize=DS15_v2,HB60rs,L80_v2, for example.

First, the framework based change is required. And then individual test cases are updated.

-ResourceCleanup Delete not work when -DeployVMPerEachTest specified

Unable to install ntpd. <- Test NA since chrony

Need to change infra - pckg installation ans verification points in each test script.

There should be some fallback and test redesign:

by default go with chronyd
fallback to ntpd if chronyd is not available

Optional OFED driver installation

A change request comes from HPC team for CentOS HPC image.

Cleanup duplicate DPDK install code in the dpdkSetup.sh and dpdkUtils.sh

The code from dpdkSetup.sh and dpdkUtils.sh should be unified and duplicates removed.

DPDK new TC step

We need to add a test step that verifies that testpmd can be started successfully with --no-pci flag.

Improve VERIFY-DPDK-BUILD-AND-TESTPMD-TEST to also have this step run at the begining. Testpmd with --no-pci flag was reported to trigger a kernel crash/hang.

Example of command:

testpmd --no-pci -m 1024 -c 0x3 -- -i --total-num-mbufs=16384 --coremask=0x2 --rxq=1 --txq=1

Run-LISAv2.ps should take -TestPriority param

This new parameter should be added to infra and pick up the tests with priority base.

framework update required for SLES15

This was found from DPDK testing, and content of LISAv2-OneVM3NIC-DPDKDaily-JV42-20190410161730-role-0-distroVersion.txt shows 'unknown'
And uptime log, LISAv2-OneVM3NIC-DPDKDaily-JV42-20190410161730-role-0-uptime.txt, shows UPTIME_COMMAND_ERROR.
This is reproducible in SLES 15.

Update KDUMP test cases

grub setup changed in rhel8, crashkernel must be now added to /boot/grub2/grubenv

Show all test number when using multiple override vm sizes

Issue reported by @lizzha here: #138

SR-IOV: Generic fixes

All mlx4 references must expanded to also support mlx5 driver (Hyper-V case only for now with CX4 cards).
Reference: https://github.com/LIS/LISAv2/search?q=mlx4_core&unscoped_q=mlx4_core
This line might be NA on SLES15, evaluate adding the proper command for sles15 and still support older sles12 releases.
https://github.com/LIS/LISAv2/search?q=rcSuSEfirewall2&unscoped_q=rcSuSEfirewall2

Infiniband tests: Fix test failure exit

RDMA tests are not supported on RHEL8, however the automation will not ack this and exit immediately.
The test will continue to run until getting into timeout.
Logging is correct, but the test execution should end and exit immediately.

If RHEL8 is to be supported, packages must be updated/handled separately, as RHEL8 has many obsolete packages, or different names for some (like python -> python3).

Condition: https://github.com/LIS/LISAv2/blob/master/Testscripts/Linux/SetupRDMA.sh#L63

MPI type intel does not support on 'redhat_8' or not implement. Test still ran for 123 minutes
MPI type mvapich does not support on 'redhat_8' or not implement. Test did not immediately exit, it ran for 124 minutes

HyperV: Add support for custom kernel install for generation 2 VMs

LISAv2 fails to modify the proper grub.cfg on efi systems, as the path is no longer /boot/grub/grub.cfg.

Depending on the distro version, that path can be:

RHEL - /boot/efi/rhel/grub/grub.cfg
SLES - /boot/efi/sles/grub/grub.cfg

Ubuntu path must be checked too.

Handle kernel versions in DPDK run-time

https://docs.microsoft.com/en-us/azure/virtual-network/setup-dpdk defines the min-supported kernel versions in each distro.
Need to determine how it handles if dpdk compilation failed in each new kernel version, if we need to validate dpdk compilation in each kernel pipeline, and how it handles dependent packages likes kernel source or etc.

Source: #223

Current LISAv2 has PowerShell dependency

LISAv2 should be launching in PowerShell console today, and this limitation does not help for certain condition like Windows Core only, or internal project. Need new design of LISAv2 packaging or launching.

LISAv2 abnormal exit due to unhandled exception

05/21/2019 12:11:00 : [INFO ] Calling function - Upload-RemoteFile. Error in upload after 10 attempt, hence giving up

05/21/2019 12:11:00 : [INFO ] dos2unix: converting file .\Testscripts\Linux\CollectLogFile.sh to Unix format...

05/21/2019 12:11:00 : [INFO ] Uploading .\Testscripts\Linux\CollectLogFile.sh to lisa : 52.191.136.173, port 22 using Password authentication

05/21/2019 12:11:02 : [WARN ] Error in upload, attempt 1, retrying

05/21/2019 12:11:12 : [INFO ] Uploading .\Testscripts\Linux\CollectLogFile.sh to lisa : 52.191.136.173, port 22 using Password authentication

05/21/2019 12:11:15 : [WARN ] Error in upload, attempt 2, retrying

05/21/2019 12:11:25 : [INFO ] Uploading .\Testscripts\Linux\CollectLogFile.sh to lisa : 52.191.136.173, port 22 using Password authentication

05/21/2019 12:11:27 : [WARN ] Error in upload, attempt 3, retrying

05/21/2019 12:11:37 : [INFO ] Uploading .\Testscripts\Linux\CollectLogFile.sh to lisa : 52.191.136.173, port 22 using Password authentication

05/21/2019 12:11:39 : [WARN ] Error in upload, attempt 4, retrying

05/21/2019 12:11:49 : [INFO ] Uploading .\Testscripts\Linux\CollectLogFile.sh to lisa : 52.191.136.173, port 22 using Password authentication

05/21/2019 12:11:52 : [ERROR] EXCEPTION: Calling function - Upload-RemoteFile. Error in upload after 5 attempt, hence giving up

05/21/2019 12:11:52 : [ERROR] Source: Line 125 in script .\Libraries\TestHelpers.psm1.

05/21/2019 12:11:52 : [INFO ] Copying all files back to original working directory: C:\Jenkins-Slaves\azure-slave-2-executor-7\workspace\tool-run-lisav2-tests-in-parallel@27.

05/21/2019 12:12:22 : [INFO ] Setting workspace back to original location: C:\Jenkins-Slaves\azure-slave-2-executor-7\workspace\tool-run-lisav2-tests-in-parallel@27

05/21/2019 12:12:22 : [INFO ] Forcefully exiting with exit code 0 as ExitWithZero flag was set to True

05/21/2019 12:12:22 : [INFO ] LISAv2 exit code: 0

SRIOV-VERIFY-MAX-VF-CONNECTION-MAX-VCPU: stabilize test-case

Sometimes, the test will fail at least in Azure with the below.
On retry or manual check of the interfaces, all are present, so it's probably a bad timing in script execution.

05/28/2019 11:55:38 : [INFO ] TEST SCRIPT SUMMARY ~~~~~~~~~~~~~~~
Expected VF count: 8. Actual VF count: 8
Successfully sent file from VM1 to VM2 through eth0
Successfully sent file from VM1 to VM2 through eth1
Successfully sent file from VM1 to VM2 through eth2
Successfully sent file from VM1 to VM2 through eth3
Successfully sent file from VM1 to VM2 through eth4
Successfully sent file from VM1 to VM2 through eth5
Successfully sent file from VM1 to VM2 through eth6
Unable to ping 10.0.0.5 through eth7

[P0] Revert to Managed Disk for Gen2 VM provision

[Azure VERIFY-DEPLOYMENT-PROVISION eastus] 5:17:37 PM - Resource Microsoft.Compute/virtualMachines 'LISAv2-OneVM-microsoft-BY76-20190516171703-role-0' failed with message '{
[Azure VERIFY-DEPLOYMENT-PROVISION eastus] "error": {
[Azure VERIFY-DEPLOYMENT-PROVISION eastus] "code": "BadRequest",
[Azure VERIFY-DEPLOYMENT-PROVISION eastus] "### message": "Generation 2 Hypervisor Image and VM Size can only be used with managed-disk VMs."
[Azure VERIFY-DEPLOYMENT-PROVISION eastus] }
[Azure VERIFY-DEPLOYMENT-PROVISION eastus] }'VERBOSE: 5:17:37 PM - Resource Microsoft.Network/networkInterfaces

RHEL8: python check failure in lisav2

These logs repeat endlessly in RHEL8 RTM:

[Azure Functional WALA x] 05/07/2019 18:28:14 : [ERROR] Opened main channel
[Azure Functional WALA x] 05/07/2019 18:28:14 : [ERROR] Allocated pty (ospeed 38400bps, ispeed 38400bps)
[Azure Functional WALA x] 05/07/2019 18:28:14 : [ERROR] Started a shell/commandSent EOF messageServer sent command exit status 0Disconnected: All channels closed
[Azure Functional WALA x] 05/07/2019 18:28:14 : [ERROR] Linux machine returned exit code : 1
[Azure Functional WALA x] 05/07/2019 18:28:14 : [ERROR] Failed to execute : which python || which python2 || which python3. Retrying...

DPDK: skip TC VERIFY-OVS-DPDK on non debian platforms

VERIFY-OVS-DPDK supports only DEBIAN based platforms for the moment, so it makes sense to skip the test on the other platforms.

XML global config file contains sensitive information after the run is complete

After a LISAv2 run with secrets file path given as a parameter, XML\GlobalConfigurations.xml contains the secrets and it may leak sensitive information.

Possible fix: create a copy of the existing XML\GlobalConfigurations.xml and revert the XML\GlobalConfigurations.xml contents to the initial copy at the end of the lisav2 run.

OverrideVMSize: redeploys on the same vm size as the previous test case

While running GPU tests, we found an issue related to the implementation of #73

After the 1st test case runs on all it's vm sizes, the 2nd test case only deploys on the last size of the previous test case.

-TestNames 'NVIDIA-GRID-DRIVER-VALIDATION,GPU-PCI-RESCIND'
-OverrideVMSize 'Standard_NV6,Standard_NV12,Standard_NV24'

[LISAv2 Test Results Summary]

Test Run On           : 04/18/2019 11:14:54
ARM Image Under Test  : Canonical : UbuntuServer : 18.10 : latest
Total Test Cases      : 2 (4 Passed, 1 Failed, 1 Aborted, 0 Skipped)
Total Time (dd:hh:mm) : 0:1:48

   ID TestArea             TestCaseName                                                                TestResult TestDuration(in minutes) 
-------------------------------------------------------------------------------------------------------------------------------------------
    1 GPU                  NVIDIA-GRID-DRIVER-VALIDATION-Standard_NV6                                        FAIL                 4.01 
	Using nVidia driver : GRID 
	lsvmbus: Expected "PCI Express pass-through" count: 1, count inside the VM: 1 : PASS 
	lspci: Expected "3D controller: NVIDIA Corporation" count: 1, found inside the VM: 1 : PASS 
	lshw: Expected Display adapters: 1, total adapters found in VM: 1 : PASS 
	nvidia-smi: Expected GPU count: 1, found inside the VM: 0 : FAIL 
    2 GPU                  NVIDIA-GRID-DRIVER-VALIDATION-Standard_NV12                                       PASS                 6.79 
	Using nVidia driver : GRID 
	lsvmbus: Expected "PCI Express pass-through" count: 2, count inside the VM: 2 : PASS 
	lspci: Expected "3D controller: NVIDIA Corporation" count: 2, found inside the VM: 2 : PASS 
	lshw: Expected Display adapters: 2, total adapters found in VM: 2 : PASS 
	nvidia-smi: Expected GPU count: 2, found inside the VM: 2 : PASS 
    3 GPU                  NVIDIA-GRID-DRIVER-VALIDATION-Standard_NV24                                    ABORTED                49.59 
	Using nVidia driver : GRID 
    4 GPU                  GPU-PCI-RESCIND-Standard_NV6                                                      PASS                  6.5 
	Using nVidia driver : GRID 
	Successfully rescinded the PCI device. :  
	lsvmbus: Expected "PCI Express pass-through" count: 4, count inside the VM: 4 : PASS 
	lspci: Expected "3D controller: NVIDIA Corporation" count: 4, found inside the VM: 4 : PASS 
	lshw: Expected Display adapters: 4, total adapters found in VM: 4 : PASS 
	nvidia-smi: Expected GPU count: 4, found inside the VM: 4 : PASS 
    5 GPU                  GPU-PCI-RESCIND-Standard_NV12                                                     PASS                 5.13 
	Using nVidia driver : GRID 
	Successfully rescinded the PCI device. :  
	lsvmbus: Expected "PCI Express pass-through" count: 4, count inside the VM: 4 : PASS 
	lspci: Expected "3D controller: NVIDIA Corporation" count: 4, found inside the VM: 4 : PASS 
	lshw: Expected Display adapters: 4, total adapters found in VM: 4 : PASS 
	nvidia-smi: Expected GPU count: 4, found inside the VM: 4 : PASS 
    6 GPU                  GPU-PCI-RESCIND-Standard_NV24                                                     PASS                 6.89 
	Using nVidia driver : GRID 
	Successfully rescinded the PCI device. :  
	lsvmbus: Expected "PCI Express pass-through" count: 4, count inside the VM: 4 : PASS 
	lspci: Expected "3D controller: NVIDIA Corporation" count: 4, found inside the VM: 4 : PASS 
	lshw: Expected Display adapters: 4, total adapters found in VM: 4 : PASS 
	nvidia-smi: Expected GPU count: 4, found inside the VM: 4 : PASS

2nd Test case, 1st vm size
GPU-PCI-RESCIND-Standard_NV6 started running.
but lower in the test the test name is tagged as
TestName = NVIDIA-GRID-DRIVER-VALIDATION-Standard_NV24

04/18/2019 12:29:24 : [INFO ] GPU-PCI-RESCIND-Standard_NV6 started running.
04/18/2019 12:29:24 : [INFO ] Current test setup: OneVM
04/18/2019 12:29:24 : [INFO ] Checking the subscription usage...
04/18/2019 12:29:25 : [INFO ] Estimating VM #1 usage.
[...]
04/18/2019 12:29:25 : [INFO ] Estimated subscription usage is under allowed limits.
04/18/2019 12:29:25 : [INFO ] Creating Resource Group : LISAv2-OneVM-ubuntuAz-BD96-20190418122925.
04/18/2019 12:29:25 : [INFO ] Verifying that Resource group name is not in use.
04/18/2019 12:29:25 : [INFO ] Try to delete resource group LISAv2-OneVM-ubuntuAz-BD96-20190418122925...
04/18/2019 12:29:25 : [INFO ] Checking if LISAv2-OneVM-ubuntuAz-BD96-20190418122925 exists...
04/18/2019 12:29:25 : [INFO ] LISAv2-OneVM-ubuntuAz-BD96-20190418122925 does not exists.
04/18/2019 12:29:25 : [INFO ] Using location : westus2
VERBOSE: Performing the operation "Replacing resource group ..." on target 
"LISAv2-OneVM-ubuntuAz-BD96-20190418122925".
VERBOSE: 12:29:26 PM - Created resource group 
'LISAv2-OneVM-ubuntuAz-BD96-20190418122925' in location 'westus2'
04/18/2019 12:29:26 : [INFO ] Resource Group LISAv2-OneVM-ubuntuAz-BD96-20190418122925 created.
04/18/2019 12:29:26 : [INFO ] Setting LISAv2-OneVM-ubuntuAz-BD96-20190418122925 tag : BuildUser = azure-slave-2$
[...]
04/18/2019 12:29:27 : [INFO ] Setting LISAv2-OneVM-ubuntuAz-BD96-20190418122925 tag : TestName = NVIDIA-GRID-DRIVER-VALIDATION-Standard_NV24
04/18/2019 12:29:28 : [INFO ] Setting LISAv2-OneVM-ubuntuAz-BD96-20190418122925 tag : BuildMachine = WORKGROUP\azure-slave-2
04/18/2019 12:29:28 : [INFO ] Setting LISAv2-OneVM-ubuntuAz-BD96-20190418122925 tag : CreationTime = 2019/04/18 12:29:28

Manually verified the deployed vm with lspcu and lspci we see it has 4 GPU adapters and 24 processor cores, matching size Standard_NV24 GPU optimized virtual machine sizes

The same thing happened when running the tests on NC* Series of virtual machines, (here the tests failed because they could not install the driver) but the Deployed machines are the same

-TestNames 'NVIDIA-CUDA-DRIVER-VALIDATION,GPU-PCI-RESCIND'
-OverrideVMSize 'Standard_NC6,Standard_NC12,Standard_NC24'

    1 GPU                  NVIDIA-CUDA-DRIVER-VALIDATION-Standard_NC6                                        FAIL                 4.04 
	Using nVidia driver : CUDA 
	lsvmbus: Expected "PCI Express pass-through" count: 1, count inside the VM: 1 : PASS 
	lspci: Expected "3D controller: NVIDIA Corporation" count: 1, found inside the VM: 1 : PASS 
	lshw: Expected Display adapters: 1, total adapters found in VM: 1 : PASS 
	nvidia-smi: Expected GPU count: 1, found inside the VM: 0 : FAIL 
    2 GPU                  NVIDIA-CUDA-DRIVER-VALIDATION-Standard_NC12                                       FAIL                 4.91 
	Using nVidia driver : CUDA 
	lsvmbus: Expected "PCI Express pass-through" count: 2, count inside the VM: 2 : PASS 
	lspci: Expected "3D controller: NVIDIA Corporation" count: 2, found inside the VM: 2 : PASS 
	lshw: Expected Display adapters: 2, total adapters found in VM: 2 : PASS 
	nvidia-smi: Expected GPU count: 2, found inside the VM: 0 : FAIL 
    3 GPU                  NVIDIA-CUDA-DRIVER-VALIDATION-Standard_NC24                                       FAIL                 5.43 
	Using nVidia driver : CUDA 
	lsvmbus: Expected "PCI Express pass-through" count: 4, count inside the VM: 4 : PASS 
	lspci: Expected "3D controller: NVIDIA Corporation" count: 4, found inside the VM: 4 : PASS 
	lshw: Expected Display adapters: 4, total adapters found in VM: 4 : PASS 
	nvidia-smi: Expected GPU count: 4, found inside the VM: 0 : FAIL 
    4 GPU                  GPU-PCI-RESCIND-Standard_NC6                                                      FAIL                 5.37 
	Using nVidia driver : CUDA 
	Successfully rescinded the PCI device. :  
	lsvmbus: Expected "PCI Express pass-through" count: 4, count inside the VM: 4 : PASS 
	lspci: Expected "3D controller: NVIDIA Corporation" count: 4, found inside the VM: 4 : PASS 
	lshw: Expected Display adapters: 4, total adapters found in VM: 4 : PASS 
	nvidia-smi: Expected GPU count: 4, found inside the VM: 0 : FAIL 
    5 GPU                  GPU-PCI-RESCIND-Standard_NC12                                                     FAIL                 5.44 
	Using nVidia driver : CUDA 
	Successfully rescinded the PCI device. :  
	lsvmbus: Expected "PCI Express pass-through" count: 4, count inside the VM: 4 : PASS 
	lspci: Expected "3D controller: NVIDIA Corporation" count: 4, found inside the VM: 4 : PASS 
	lshw: Expected Display adapters: 4, total adapters found in VM: 4 : PASS 
	nvidia-smi: Expected GPU count: 4, found inside the VM: 0 : FAIL 
    6 GPU                  GPU-PCI-RESCIND-Standard_NC24                                                  Aborted                    0

Need to update regex of VM family section for M_v2

New VM size, Mv2 released but the LISAv2 code has missing regex for Mv2.

https://github.com/LIS/LISAv2/blob/master/Libraries/Azure.psm1#L128

SLES: add suport for -CustomKernel "localfile:"

Lisav2 does not support installing custom kernel on SLES 15/SLES 12.

Linux kernel headers deb fails to install on Ubuntu 19.04

It seems there is a libssl dependency issue:

dpkg: dependency problems prevent configuration of linux-headers-4.19.24-c52ea7b07f93:
linux-headers-4.19.24-c52ea7b07f93 depends on libssl1.0.0 (>= 1.0.0); however:
Package libssl1.0.0 is not installed.

On Ubuntu 19.04. libssl 1.1 is installed.

Azure Gen2 VM support

In the API this is referenced by the HyperVGenerationType property, as defined here:
https://github.com/Azure/azure-rest-api-specs/blob/master/specification/compute/resource-manager/Microsoft.Compute/stable/2019-03-01/compute.json#L4690

Update ntttcp version once new version release

#264

VM-HOT-RESIZE: refine list of tested VM sizes

Current test takes over 3 hours and goes through 48 VM sizes.

@adriansuhov suggestion to improve this is to take only the smallest and largest VM size from each SKU family.

For example, current code goes through each of these: "“Standard_A0 Standard_A1 Standard_A2 Standard_A3 Standard_A5 Standard_A4 Standard_A6 Standard_A7”"

With the changes, it will deploy only A0 and A7 from the A series.

@xian123 as the original test case owner, can you please work on this?

LISAv2: Introduce SupportedVMSize flag to test cases

new tag in xml:

INFINIBAND-INTEL-MPI-2VM
Standard_NC6,Standard_NV24

This can be overrided by the -CustomParameter flag to lisav2.
Flag can also support wildcard matching - Standard_NV*

First phase to cover: NVME, GPU, DPDK and RDMA.

GetDistro() in utils.sh detects CentOS 7.6 as rhel_6

PRODUCTION-CHECKPOINT-3CHAIN-VHD: revise test case

On Hyper-V this test must be made compatible with lisav2 functionality.
It should not remove icabase main snapshot, and must be seen how to handle the existing checkpoints with previous test failures.

[validation_functional_hyperv] 05/27/2019 14:20:55 : [INFO ] Cleaning up snapshots...
[validation_functional_hyperv] 05/27/2019 14:20:55 : [INFO ] LISAv2-OneVM-msftk-TA43-636945230527-role-0 has 12 snapshots. Removing all except PRODUCTION-CHECKPOINT-FAILBACK-FAIL
[validation_functional_hyperv] 05/27/2019 14:20:56 : [INFO ] Removed snapshot ICAbase
[validation_functional_hyperv] 05/27/2019 14:20:56 : [INFO ] Removed snapshot DYNAMIC-MEMORY-HOT-REMOVE-THEN-PRESSURE-FAIL
[validation_functional_hyperv] 05/27/2019 14:20:57 : [INFO ] Removed snapshot DYNAMIC-MEMORY-HOT-ADD-EAT-MEM-SMP-FAIL
[validation_functional_hyperv] 05/27/2019 14:20:57 : [INFO ] Removed snapshot DYNAMIC-MEMORY-HOT-ADD-EAT-MEM-UP-FAIL
[validation_functional_hyperv] 05/27/2019 14:20:58 : [INFO ] Removed snapshot FCOPY-NEGATIVE-FAIL
[validation_functional_hyperv] 05/27/2019 14:20:58 : [INFO ] Removed snapshot KVP-TEST-RESCIND-FAIL
[validation_functional_hyperv] 05/27/2019 14:20:59 : [INFO ] Removed snapshot NET-IP-INJECTION-FAIL
[validation_functional_hyperv] 05/27/2019 14:20:59 : [INFO ] Removed snapshot NET-HOT-ADD-MULTINIC-SKIPPED
[validation_functional_hyperv] 05/27/2019 14:20:59 : [INFO ] Removed snapshot NET-HOT-REMOVE-MULTINIC-SKIPPED
[validation_functional_hyperv] 05/27/2019 14:21:00 : [INFO ] Removed snapshot NET-BOOT-NONIC-HOT-ADD-NIC-SKIPPED
[validation_functional_hyperv] 05/27/2019 14:21:00 : [INFO ] Removed snapshot NET-HOT-ADD-REMOVE-MAXNIC-SKIPPED

Need slim-style LISAv2 automation design

Would like to remove PS wrapper because target host does not have PS cmdlet available, pinned resource group has no internet connection, or special requirements there.

It is nice to have new utility to generate constants.sh, all required test scripts in to a single tar-ball file without execution. So testing package can be transferred to the target system and executed manually (Semi-automation) Manually collect the output (text format)
Clone the LISAv2 git repo in Linux VM, and run an utility to execute the test scripts against the same network Linux VM.