Is this a testing framework for SDL Core?

Proposal: Test Scripts - Refactoring of Common modules

ATF test scripts use a lot of functions in commonly used modules, like:

• data accessors: file, database
• functions for tables: print, copy
• sequences:
  • create mobile session
  • register/activate mobile application
  • perform policy table update
• etc.

All these functionality are not tests artifacts but tools that tests use.

The purpose of this proposal is to clean up these common modules that everyone is able to use while creating ATF test scripts.

Review: smartdevicelink/sdl_evolution#377

https://github.com/CustomSDL/sdl_atf

gives 404

Is this url meant to point to this repository instead?

Proposal: ATF Facade

This proposal describes unified interface for ATF (Automated Test Framework) - ATF Facade. ATF should provide unified and flexible interface for creation test scripts to perform checks of SDL functionality.

Main ATF Facade purposes:

• Simplify access to ATF features for test scripts
• Separate test scripts from ATF
• Provide well-structured intuitive interface of ATF for test scripts
• Unify way of using ATF for creating automated tests
• Create single point of ATF features documentation
• Provide ability to create more human-readable tests

Review: smartdevicelink/sdl_evolution#363

License file should be updated from "Copyright (c) 2017 - 2018 SmartDeviceLink Consortium, Inc." to "Copyright (c) 2017 - 2020 SmartDeviceLink Consortium, Inc."

Add the following LICENSE to sdl_atf

Copyright (c) 2017 SmartDeviceLink Consortium, Inc.
All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:

• Redistributions of source code must retain the above copyright notice, this
  list of conditions and the following disclaimer.

• Redistributions in binary form must reproduce the above copyright notice,
  this list of conditions and the following disclaimer in the documentation
  and/or other materials provided with the distribution.

• Neither the name of SmartDeviceLink Consortium, Inc. nor the names of its
  contributors may be used to endorse or promote products derived from
  this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Need to implement functionality that will allow to test SDL in secure mode.

ATF needs some changes for smartdevicelink/sdl_core#2159. The evolution proposal adds a new mandatory fullAppID parameter to RegisterAppInterface, so ATF has to provide such a parameter for every parameter set used for that RPC, including the ones in the config file.

export QMAKE={PATH_TO_QMAKE} should be used instead of export QMAKE=${PATH_TO_QMAKE}

1. PATH_TO_QMAKE should be full path and should include the name of qmake. Example: export QMAKE={/opt/qt53/bin/qmake}
   

Bintray boost url is dead and needs to be updated to sourceforge.

Summary

ATF doed not parse structure like '"menuIcon" : null'

Steps to reproduce

1. Register and activate mobile app
2. Request AddSubMenu without menuIcon
   Script for reproduce smartdevicelink/sdl_atf_test_scripts#2142

Expected result

SDL sends "menuIcon" : null in UI.AddSubmenu to HMI, ATF parse "menuIcon" : null and fail test with message "SDL sends menuIcon in request to HMI in case parameter is absent in request"

Actual result

In case SDL sends message like

 {
   "params" : {
      "appID" : 1676936826,
      "menuID" : 1000,
      "menuIcon" : null,
      "menuParams" : {
         "menuName" : "SubMenupositive",
         "position" : 500
      }
   }
}

ATF will parse it like

[appID]: 814596906
[menuID]: 1000
[menuParams]: {
    [menuName]: 'SubMenupositive'
    [position]: 500
}

So atf tests will not failed with error because of redundant null parameter.

Can be reproduced in case SDL defect smartdevicelink/sdl_core#2744 is reproduced.

Proposal: ATF support of additional transports (BT and USB)

Currently some number of test scripts require manual checks using real transport. This part of manual work should be also automated with Automated test framework.

Automated test framework should support communication with SDL via :

• TCP (already supported)
• Bluetooth
• USB

Full Bluetooth and USB testing of ATF should use real device for communication with SDL. In future this approach also will allow to test SDL on custom OEM head units.

Review: smartdevicelink/sdl_evolution#372

Functionality needs to be added to test the Constructed Payloads feature.

API's use different naming with core. Should be renamed to SmartDeviceLink RAPI

Proposal: ATF Script Runner

ATF is a great tool for both developers and testers that helps to test SDL functionality. But it lacks a powerful test script runner. This proposal introduces such a component.

Review: smartdevicelink/sdl_evolution#374

Currently ATF can load only API.xml with the interface "SmartDeviceLink RAPI", so in case the interface name will be updated, ATF also must be updated.

OEM will probably have some extensions of SmartDeviceLink RAPI and own name of root interface.
ATF is able to extract this name form Provided xml file and should not hardcode this value

Proposal: Security ATF feature

This proposal describes support of security sessions in ATF (Automated Test Framework).
ATF should be able to test SDL security feature.

Main ATF features for checking SDL security are:

• Support of secure SDL sessions;
• Support of testing broken handshake;
• Support of sending raw data in secure channel;
• Support of expected secure requests\responses\notifications;
• Support Transport Layer Security (TLS) and Datagram Transport Layer Security (DTLS).

Review: smartdevicelink/sdl_evolution#310

Current: "please contact us" in CONTRIBUTING.md file sends email to [email protected]

Expected: "please contact us" in CONTRIBUTING.md file should send email to [email protected]

Proposal: ATF Video streaming support

Video streaming is a major feature of SDL. Currently it should be tested manually for each release. Testing of video streaming is rather complicated and expensive, and there is big probability of human mistakes during testing.

Currently ATF is able to stream data from mobile side, but not able to handle streaming from HMI side, and to check data is received and not corrupted.

ATF should provide ability to cover mentioned use cases related to video streaming.

Review: smartdevicelink/sdl_evolution#371

Proposal: Remote Automated Testing

This proposal is about modification of Automated Test Framework (ATF) for providing ability to test SDL on remote workstations using automated test scripts.

smartdevicelink/sdl_evolution#633

Steering Committee Decision:

The Steering Committee voted to fully accept this proposal on 2019-01-15.

I have everything installed and make completed successfully. The readme notes that I can use:

./start.sh [options] [script file name]

But does not provide any examples of options or scripts which I should use, what will be tested, or how to interpret results.

After merge of #181 Remote Connection functionality became broken.
ATF is unable to establish mobile connection:

[11:24:40,588] Update_Preloaded_PT                                               [SUCCESS] (542 ms)
SDL started
HMI initialized
HMI is ready
Error: TCP Connection not established during 7000 ms
Operation now in progress
[11:24:43,256] Start_SDL__HMI__connect_Mobile                                    [FAIL] (12069 ms)

In branch release/4.1.0, there is no source code in https://github.com/CustomSDL/sdl_atf, could you please update README?

$ git clone https://github.com/CustomSDL/sdl_atf

Proposal: Change ATF test reports folder structure

Main output of ATF is test reports. ATF provide such reports:

• Console logs - name of test cases with execution status FAILED/PASSED.
• SDL logs - SDL log received by ATF via telnet logger.
• Transport logs - all received and sent data from/to SDL. Transport log can have default view - messages on protocol level (except binaries) and full view - log will be expanded with packages: streamings, service messages, heartbeat, json files.
• Detailed report - all expected and sent massages related to test cases with expected and actual result.Also contains info about test cases: name, time, sequence, status, duration.

This proposal is about creating clear and useful structure of test scripts reports.

Review: smartdevicelink/sdl_evolution#378

Current configuration of install goal of build ATF does not provide default install path for ATF build and cmake tries to install it to system (in most cases to /usr/local).
So user have to set cmake variable CMAKE_INSTALL_PREFIX each time of build ATF.
(in a way like that: $ cmake . -DCMAKE_INSTALL_PREFIX=./install)

It will be better to set some default installation path for ATF build with ability to change it if user want.

Currently if there are 2 mobile applications try to send the same RPC inside a particular step, ATF shows a message:

RPC service Warning: Message with correlationId: 3 in session 2 was sent earlier by ATF
Function with cor_id : 3 was not sent by ATF

This is due to the fact that ATF checks uniqueness of CorrelationIds across all mobile sessions.

The task is to change this verification in a way that CorrelationId will be checked across one session.

Proposal: WebEngine support for SDL JavaScript

This proposal is adding a new transport to the SDL JavaScript library to support (progressive) web apps to run on a WebEngine or a browser.

Review: smartdevicelink/sdl_evolution#767

Steering Committee Decision:

The Steering Committee voted to accept this proposal with the following revisions:

• When formatting locales in the manifest.json file, use standard (RFC 5646), and convert locale strings when necessary.
• Do not specify the location of the manifest's <script> tag placement within the proposal.
• All revisions included in this PR: https://github.com/smartdevicelink/sdl_evolution/pull/888/files

The proposal .md file was updated to reflect these revisions on 12/19/19.

Sometimes ATF is unable to parse multi-frame messages for 2 mobile sessions.
This is happened when SDL sends large messages which needs to be split into pieces (multi-frames) on a one mobile connection and for 2 or more mobile sessions simultaneously.

This leads to aborting of ATF script with error:

QIODevice::read (QTcpSocket): device not open
PANIC: unprotected error in call to Lua API (./modules/protocol_handler/protocol_handler.lua:291: attempt to perform arithmetic on field 'totalSize' (a nil value))

ATF uses QT timers for internal needs:

• in order to execute some functions after some period of time. In connecttest/RUN_AFTER()
• to check current status of test step while it's executiing. In testbase/main()
• to handle HeartBeat functionality. In heartbeat_monitor

Currently it's used default accuracy for this timer. It leads to 5% deviation comparing to expected value.
Some tests require more accurate behavior (e.g. OnResetTimeout feature, Policy tests etc.)

The task is to increase timer's accuracy.

Currently there is a luamq submodule in ATF.
It was initially introduced to support scripts for LowVoltage feature.
Originally this feature intended to use MQ messages. However proposal was changed to use Signals. Scripts were updated accordingly.
So currently luamq module is not required to run any lua test script and can be removed.

License file should be updated to "Copyright (c) 2017 - 2018 SmartDeviceLink Consortium, Inc."

1. Endless cycles of test execution in case SDL forever sends expected by test RPC with delay < default.
2. SDL logs are not saved in testing reports.

In ATF there is a possibility to check that some event occurs.
E.g.:

common.hmi.getConnection():ExpectNotification("SDL.OnStatusUpdate"):Times(0)

In this case ATF will wait until timeout (defined by zeroOccurrenceTimeout) expires.
If event occurs within this timeout expectation will be marked as SUCCESS
However if such expectation is defined in callback of the function timeout is not taken into account.
E.g.

  session:StartService(7)
  :Do(function()
      common.hmi.getConnection():ExpectNotification("SDL.OnStatusUpdate"):Times(0)
    end)

In this case ATF won't wait and mark expectation as SUCCESS just right away.

Proposal: SDL watchdog

Create service responsible for running and monitoring multiple SDL processes and handling SDL configurations. Service will provide API to test scripts, and script will be able to run SDL with certain configuration. Also this service should provide possibility to run multiple SDL instances (for parallel testing), monitor and control their life cycle.

Review: smartdevicelink/sdl_evolution#373

There is an XML reporter module in ATF (modules/reporter.lua)
Creating of XML report could be controlled by config.excludeReport configuration parameter, which is set to false.
This means report will be created automatically by default.
On some environments creating of this report leads to slow message processing.

Occurrence: Always

Description

ATF doe not disable HB by using HB flags sendHeartbeatToSDL = false, self.heartbeatEnabled = false in case of 2 connections.

Steps to reproduce

1. Execute attached DisableHB.lua script

Steps in script:

1. ATF starts SDL and performs initialization of HMI.
2. Opens 2 connections and in fist connection HB is disabled.

Expected result

Wait function is executed without any fails and interruptions.

Actual result

ATF closes fist session by HB timeout.

Environment

SDL commit: develop 4eb09bf
OS with SDL: Ubuntu 16.04
VM used for OS with SDL (yes/no): (no by default)
ATF version: a6a2a9b
Transport: TCP

Attachments

Note: extract attached files in 'atf/test_scripts/Defects' folder
Defects.zip

Within WebEngine feature implementation there was introduced new BUILD_WEBSOCKET_SERVER_SUPPORT SDL build option.
If such option is not exists in SDL's build_config.txt ATF set it to ON by default.
This led to script fails if they run on old SDL build.
There is an idea to set this option to OFF by default in order to have better compatibility with old SDL builds

By default SDL has a few loggers enabled (see log4cxx.properties file).
Three of them save data to files:

• SmartDeviceLinkCoreLogFile => SmartDeviceLinkCore.log
• TransportManagerLogFile => TransportManager.log
• ProtocolFordHandlingLogFile => ProtocolFordHandling.log

Nevertheless ATF gets SDL logs trough Telnet logger which is not utilize files at all.
The idea is to disable some of the SDL loggers which writes data to file system.
This increase performance on environments with parallel test execution.

Test Script
008_M2M_Timeout.txt

Output
008_M2M_Timeout_20190328163947.txt

The RUN_AFTER function (used in line 77 of the test script) does not always fire when the timeout occurs, causing the test to fail. Reproduction rate of this bug is ~1/10

ATF Version: feature/app_services branch
ATF Test Scripts Version: feature/app_services branch
SDL Core Version: release/5.1.0 branch

Proposal: ATF Selenium Support

This proposal is to integrate the Generic HMI into ATF tests scripts by using Selenium to control a web browser programmatically.

Review: smartdevicelink/sdl_evolution#1164

Currently ATF support only next list of control messages:

FRAME_INFO = {
  HEARTBEAT = 0x00,
  LAST_FRAME = 0x00,
  START_SERVICE = 0x01,
  START_SERVICE_ACK = 0x02,
  START_SERVICE_NACK = 0x03,
  END_SERVICE = 0x04,
  END_SERVICE_ACK = 0x05,
  END_SERVICE_NACK = 0x06,
  SERVICE_DATA_ACK = 0xFE,
  HEARTBEAT_ACK = 0xFF
}

When ATF receives other values in Frame Info field of SDL frame (see frame header) ATF aborts script run.

According of protocol specification should be added handling of SDL frames with next values of Frame Info field:
0x07 Register Secondary Transport
0x08 Register Secondary Transport ACK
0x09 Register Secondary Transport NAK
0xFD Transport Event Update

Help! :)

Need more specific directions on how to use this repository with sdl core.

1. What cmake flags are required when building sdl core to use this testing framework? Build for qt hmi or can these tests be used with either hmi type?
2. Do you run ./start.sh already after sdl core is running?
3. What are the arguments supplied to the start.sh script? Can I get an example?
4. Is the "scripts" argument supplied with scripts from this repository
5. What are the different possible options arguments?
6. When running ./start.sh I get the error

   Lua error:
   ./modules/reporter.lua:1: module 'xml' not found:

Since smartdevicelink/sdl_core#3778 ATF is unable to find and parse the MOBILE_API.xml, causing parallel runs to fall back to the XML within the data folder and causing normal runs to fail outright.

Some changes are needed to find the MOBILE_API.xml within the rpc_spec submodule.

Perhaps also some changes could be added so that parallel runs print a warning when falling back to an older API XML.

Proposal: Modernize Ubuntu Support V20 LTS

This proposal would like to provide support for Ubuntu 20 LTS and bump the minimum required version of Ubuntu to Ubuntu 18 LTS. Currently Ubuntu 18 is the recommended platform and Ubuntu 16 is the minimum required version.

These changes will mainly affect the build process for SDL Core and ATF.

Review: smartdevicelink/sdl_evolution#1103

Steering Committee Decision:
The Steering Committee voted to fully accept this proposal.