georgejecook / rooibos Goto Github PK

View Code? Open in Web Editor NEW

73.0 8.0 30.0 7.13 MB

simple, flexible, fun brightscript test framework for roku scenegraph apps

License: MIT License

JavaScript 1.95% Brightscript 0.43% CSS 2.48% TypeScript 50.15% BrighterScript 44.99%

rooibos's Introduction

Simple, mocha and junit inspired, flexible, and fun Brightscript test framework for Roku apps

Development

Rooibos is an independent open-source project maintained exclusively by volunteers.

You might want to help! Get in touch via the slack group or raise issues.

Sample project

https://github.com/georgejecook/rooibos-roku-sample

Quick start

https://github.com/georgejecook/rooibos/blob/master/docs/index.md#getting-started

FAQ

Is Rooibos ready for production use?

Yes, it's been used in production by quite a few folks, I'll provide a list in due-course; but there are 10'000's of tests in production in some very respectable companies.

Is Rooibos actively maintained?

I love rooibos, and always keep it up to date. I use it all the time.

rooibos's People

Contributors

Stargazers

Watchers

rooibos's Issues

rooibosFunctionMap.brs exceeds Roku character count limit

My client has a significant number (~3,000) of unit tests . When I attempt to pre-process all suites the resulting rooibosFunctionMap.brs does not load:

------ Compiling dev ------
*** ERROR compiling /pkg:/source/tests/rooibos/rooibosFunctionMap.brs:
Error loading file. (compile error &hb9) in pkg:/source/tests/rooibos/rooibosFunctionMap.brs
07-18 14:26:25.567 [scrpt.cmpl] Compiling App, id 'dev'

It appears this error is due to the generated file exceeding some Roku file, character count limit: 2,097,151

I have created a file with 2,097,152 characters (one character over the limit) for you to replicate the issue.

I think we need some rule that will split the rooibosFunctionMap.brs up into smaller files when a particular test limit is exceeded.

generate lcov/gcov file for front-end coverage report

This will enable the use of extensions such as:
https://marketplace.visualstudio.com/items?itemName=ryanluker.vscode-coverage-gutters
https://marketplace.visualstudio.com/items?itemName=alexdima.vscode-lcov
https://marketplace.visualstudio.com/items?itemName=markis.code-coverage

there are node modules that can help with that:
https://www.npmjs.com/package/karma-sonarqube-unit-reporter

An lcov.info file looks like this:

TN:
SF:./components/ABC.brs
DA:5,0
DA:29,1
DA:33,1
end_of_record
TN:
SF:./components/File.brs
...

for it to work, the line num must be +1
and lines with zeros should be added as well.

stub warning message for non existent message on target appears too often

this code in RBS_BTS_STUB and RBS_BTS_MOCK is wrong

    target[methodName] =  m["MockCallback" + id]
    target.__mocks = m.mocks
    if (allowNonExisting)
      ? "WARNING - mocking call " ; methodName; " which did not exist on target object"
    end if```

`@Only` annotation should be case insensitive

@Only needs to be caps…maybe rooibos should allow @only as well as @onlY (normalize case)?

remove #if FALSE lines from line coverage

lines inside #if FALSE can never be reached, and should be considered commented-out code.

RBS_CMN_FindElementIndexInArray can crash

this line can cause issues if there's a type mismatch

else if compareValue = value

RBS_CMN_AsString should pretty print array values

equals mismatches show empty when the type is array, due to this problem. should limit the max output, too

Update documentation and samples for `SGNode` testing

SGNode based testing needs to be updated to account for the inclusion of rooibosFunctionMap.brs in the SGNode subclass used as a test proxy. Not doing so results in issues accessing the generated function maps and such at test execution time.

Please update the following:

As well as any other example that includes SGNode testing.

to string method has same infinite loop crash as equals, for aa's which have mocked functions

same fix line 1366 of rooibosDist needs to be
if key <> "__mocks" and key <> "__stubs"

if setting a mock fails, then subsequent mocks fail to set silently, creating confusing results

given this test


'@Test valid - live data
function DSPNMT_executeSession_live()
  session = {}
  request = {providerName: "applicaster", feedId: "id_123" }
  mockResult = FormatJson({ "title": "Mock Feed Title"})
  m.constants.stubProviders = {
    ' providerScheme : feedID
    "applicaster": "id_123"
  }
  url = "#url"
  m.ExpectOnce(m.module, "_providerStubbed", [request], false)
  m.ExpectNone(m.module, "_readStubJSON")
  m.ExpectOnce(m.session, "getUrlForRequest", [request], url)
  m.ExpectOnce(m.module, "execute", url)

  result = m.module.executeRequest(session, request)
  m.assertEqual(result, mockResult)
end function

m.session is invalid - this caused the mock for _readStubJSON to somehow not be set. Needs investigation.

mock param checking will crash device, when one of the expected args has mocked methods

Add command to generate TestSuite for file/node from template

An interesting feature which would make TDD more seamless would be to have a command such as rooibosc --generate <TestSuiteName> which would generate the corresponding .xml and .brs test suite files, with correct includes and dependencies. Maybe with parameters for the SGNode to extend or the file to include.

This could work with a template such as this:
components\tests\specs{{TestSuiteName}}Tests.xml:

<?xml version="1.0" encoding="utf-8" ?>
<component name="{{TestSuiteName}}Tests" extends="Node" xsi:noNamespaceSchemaLocation="https://devtools.web.roku.com/schema/RokuSceneGraph.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    <interface>
        <function name="Rooibos_RunNodeTests" />
    </interface>
    <!-- Import test framework. -->
    <script type="text/brightscript" uri="pkg:/source/tests/specs/{{TestSuiteName}}Tests.brs" />
    <script type="text/brightscript" uri="pkg:/source/tests/rooibos/rooibosFunctionMap.brs" />
    <script type="text/brightscript" uri="pkg:/source/tests/rooibos/rooibosDist.brs" />
</component>

source\tests\specs{{TestSuiteName}}Test.brs:

'@SGNode {{TestSuiteName}}Tests
'@TestSuite

@Only on TestSuite only runs the first It group

'@Only
'@TestSuite [RBSA] Rooibos before after tests

'+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
'@It tests one
'+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

'@Test it one
function RBSA__it_test_one() as void
    m.AssertTrue(true)
end function

'+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
'@It tests two
'+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

'@Test it two
function RBSA__it_test_two() as void
    m.AssertTrue(true)
end function

[main] RunUserInterface
Starting test using test scene with name TestsSceneTestsScene
#########################################################################
#TEST START : ###UNDEFINED_TEST_ID###
Loading test config from default location : pkg:/source/tests/testconfig.json

[START TEST REPORT]

[RBSA] Rooibos before after tests [ONLY] (pkg:/source/tests/BeforeAfterTest.brs(1))
 |-tests one
 | |--it one : ..................................................................Success        pkg:/source/tests/BeforeAfterTest.brs(9)
 |


[END TEST REPORT]

Total  = 1 ; Passed  =  1 ; Failed   =  0 ; Ignored   =  0
 Time spent:  0ms


RESULT: Success
BRIGHTSCRIPT: ERROR: roUrlTransfer.PostFromString: Sync-wait interrupted: pkg:/source/lib/rooibos/rooibos.cat.brs(2042)

checking for test naming collisions needs improvements with Params tests - can lead to vague error messages for params

Naming collisions for tests need better handling. If a naming collision occurs for a test with params, and there is a mismatch in param types, then it can lead to unclear crash scenarios because the params test is still added, so it is applied to the original (collided) test case.

Need to take care to ensure that any params test case, which is part of a name-collided test does not get added.

report the line number of the expectation that caused the log failure

in test methods with many expects, it's not immediately clear which failed - not too hard to work out; but would be great to not waste people's time on this, and just let them click the link to focus

Missing Unitest coverage statistics

Hi,

It would be great to know the actual coverage of the unit-tests.
We want to increase our confidence in the application and alert when unit-tests below a specific percentage.

Thanks.

BR,
Aviad A

Check the version of rooibosC was compatible with the installed rooibosDist.brs

We don't currently check to see if the generated files rooibos uses, were generated by a version fo rooibos-preprocessor that is compatible with the rooibosDist.brs

rooibosDist.brs should contain a minProcessorVersion, which should be verified. If there is a mismatch, then the a clear error message should be displayed.

should not run tests with duplicate '@Test names

in cases where tests have duplicate names, it's possible for all kinds of silliness to ensue - it could bet getting wrong results, or in a worse case - having tests with incompatible params/function arguments.

This can cause a lot of confusion - we should therefore not execute a test if it's a duplicate, including the original, instead adding a very clear warning.

Remove eval calls from the app

Eval has been stated as being unsafe by several roku engineers who have hinted that it might not be future proof in various posts.

eval is currently used in 2 places:

To get a pointer to functions
To evaluate arguments for parameterized tests

There is a simple way to avoid this, which is to bundle a simple javascript with the test library, which will parse the tests and create mappings in a file, which can be used by the framework

It could also be a future version of the SceneGraph SDK has a different provision for resolving methods, in which case that could be used for case 1, and case 2 could become a simple javascript parser - with a special syntax for creating nodes objects (the only real use for eval I can think of in the parsing of params), which would be less obtrusive and more performant.

"showOnlyFailures": true not working

Upon starting, testconfig is found and read by Rooibos, I get this log:
Loading test config from default location : pkg:/source/tests/testconfig.json

However, "showOnlyFailures": true doesn't affect the behavior, since I get all the successes as well.
I believe the configuration is being overloaded by Rooibos, since breaking a path such as testPath doesn't affect the behavior. This is my current config:

{
	"testPath": "pkg:/source/tests/specs",
	"logLevel": 1,
	"testsDirectory": "pkg:/source/tests/specs",
	"failFast": false,
	"showOnlyFailures": true,
	"maxLinesWithoutSuiteDirective": 30,
	"supportLegacyTests": false
}

testConfig.json Location

The docs and examples have the testConfig.json living in source/test/rooibos however the code has a default location of /source/test

Example Location

Default Location

include lineoffsetting sensors

sensors which are in the same line as return statements are not called. which causes the line to not be accounted.

code should use 2 spaces instead of tabs.

I just took the eclipse default when I started this project.

oops :)

Typo `AssertArratNotEmpty`

rooibos/src/Rooibos_BaseTestSuite.brs

Line 800 in a3c9981

 Function RBS_BTS_AssertArratNotEmpty(item as dynamic, msg = "" as string) as dynamic 

Update API docs for Stub

The API documentation should be updated for Stub.

https://georgejecook.github.io/rooibos/module-BaseTestSuite.html#Stub

Stub(target, methodName, expectedArgsopt, returnValueopt, allowNonExistingMethodsopt) → {Object}

rooibos.cat.brs

function RBS_BTS_Stub(target, methodName, returnValue = invalid, allowNonExistingMethods = false) as object

The problem -> expectedArgs is not supported for Stub

Accept whitespace (carriage return) between annotations and test functions

'@Test Status Code Fail Match
'@Params["http://httpbin.org/status/400", 200]

function Atst__status_code_fail(url, statusCode) as void

    response = Requests().get(url, {retryCount:0})
    m.AssertEqual(response.status_code, statusCode)

end function

Will not work

@Ignore annotations work, but are not accounted in the test report

The test report displays Ignored = 0 as test cases are being ignored.
There are no details of which test cases are being ignored either.
This feature has never worked for me, not with 2.0, 2.3, or 3.0.1-beta.

If non-array type is passed to expectedArgs, a proper warning should be shown

The framework correctly interprets a non-array value as invalid; but this might not be clear why to the user.

Suggestion: expectation fails with message "You must pass an array of expected Args as the expectedArgs parameter"

@AfterEach not running

May be user error, but I do not see @AfterEach running

'@TestSuite [RBSA] Rooibos before after tests

'+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
'@It tests
'+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

'@BeforeEach
function RBSA__BeforeEach() as void
	? "!!! Before"
end function


'@AfterEach
function RBSA__AfterEach() as void
    ? "!!! After"
end function

'@Only
'@Test before after
function RBSA__before_after() as void

    assertResult = m.Fail("reason")

    isFail = m.currentResult.isFail
    m.currentResult.Reset()

    m.AssertFalse(assertResult)
    m.AssertTrue(isFail)
end function

[main] RunUserInterface
Starting test using test scene with name TestsSceneTestsScene
#########################################################################
#TEST START : ###UNDEFINED_TEST_ID###
Loading test config from default location : pkg:/source/tests/testconfig.json
!!! Before

[START TEST REPORT]

[RBSA] Rooibos before after tests (pkg:/source/tests/BeforeAfterTest.brs(1))
 |-tests
 | |--before after : ............................................................Success        pkg:/source/tests/BeforeAfterTest.brs(20)
 |


[END TEST REPORT]

Total  = 1 ; Passed  =  1 ; Failed   =  0 ; Ignored   =  0
 Time spent:  0ms

Run command; E2E

Sorry this is not an issue and can be delete after it's answered, but I'm looking into your repo and wondering if this can do end 2 end testing? Also I cloned your repo and want to perform the sample AssertionTests.brs, but do not see a run command. Thanks, new to brightscript and Roku testing.

3.0.4 - Function Call Operator ( ) attempted on non-function

Function RBSFM_getIgnoredTestInfo is not defined.

Function Call Operator ( ) attempted on non-function. (runtime error &he0) in pkg:/source/tests/rooibos/rooibosDist.brs(1779)
1779:   ignoredInfo = RBSFM_getIgnoredTestInfo()
Backtrace:
#3  Function rbs_logger_printstatistic(statobj As Object) As Void
   file/line: pkg:/source/tests/rooibos/rooibosDist.brs(1779)
#2  Function rbs_tr_run() As Void
   file/line: pkg:/source/tests/rooibos/rooibosDist.brs(2026)
#1  Function rooibos__init() As Dynamic
   file/line: pkg:/source/tests/rooibos/rooibosDist.brs(39)
#0  Function main(args As Dynamic) As Dynamic
   file/line: pkg:/source/Main.brs(6)

BRIGHTSCRIPT: ERROR: roUrlTransfer.PostFromString: Sync-wait interrupted

I get this error every time I run tests, as the last line, after coverage report:

BRIGHTSCRIPT: ERROR: roUrlTransfer.PostFromString: Sync-wait interrupted: pkg:/source/tests/rooibos/rooibosDist.brs(2205)
 ```

design contribution

Hallo dear
I am a graphics designer.i am seeing that in your projects the logo is so simple.so i want to make a better logo for this projects.if you have any requirements then let me know.its totally free for open source projects.
Thank you.

Sample code needs tidying up

The sample code in samples/nonEvalSample doesn't work straight off the bat. Some files are incorrectly referenced with the wrong paths etc.

The directory structure is also slightly confusing and it's not clear right away which files are test specs and which are part of the framework.

@Only on test group AND test suite causes first group to run

@only on the test suite AND the third It group runs the first It group AND the third It group. Probably related to but I wanted to give you more debugging ;) #14

'@Only
'@TestSuite [RBSA] Rooibos before after tests

'+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
'@It tests one
'+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

'@Test it one
function RBSA__it_test_one() as void
    m.AssertTrue(true)
end function

'+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
'@It tests two
'+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

'@Test it two
function RBSA__it_test_two() as void
    m.AssertTrue(true)
end function

'+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
'@Only
'@It tests three
'+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

'@Test it three
function RBSA__it_test_three() as void
    m.AssertTrue(true)
end function

[main] RunUserInterface
Starting test using test scene with name TestsSceneTestsScene
#########################################################################
#TEST START : ###UNDEFINED_TEST_ID###
Loading test config from default location : pkg:/source/tests/testconfig.json

[START TEST REPORT]

[RBSA] Rooibos before after tests [ONLY] (pkg:/source/tests/BeforeAfterTest.brs(1))
 |-tests one
 | |--it one : ..................................................................Success        pkg:/source/tests/BeforeAfterTest.brs(9)
 |
 |-tests three
 | |--it three : ................................................................Success        pkg:/source/tests/BeforeAfterTest.brs(28)
 |


[END TEST REPORT]

Total  = 2 ; Passed  =  2 ; Failed   =  0 ; Ignored   =  0
 Time spent:  0ms


RESULT: Success
BRIGHTSCRIPT: ERROR: roUrlTransfer.PostFromString: Sync-wait interrupted: pkg:/source/lib/rooibos/rooibos.cat.brs(2042)

`EqArray` vs `EqArrays` Documentation mismatch

rooibos/source/rooibos.cat.brs

Line 100 in 6524f2f

this.EqArrays = RBS_BTS_EqArray

The documentation says EqArray
https://georgejecook.github.io/rooibos/module-BaseTestSuite.html#EqArray

but the function is actually EqArrays

Re-Introduce Legacy Support for BRSTest

My client has a large body of unit tests (2.5K+) running on BRSTest. They are keen to migrate to Rooibos but need to support the existing unit tests.

add log level flags to rooibos

logging output can be quite noisy - add folllowing log levels

results: only results and warnings
info: results + extra info
verbose: results + extra + all log messages

examples of verbose messages are things like:
Multi return values all used up - repeating last value

Introduce Mutation Testing

Mutation testing is interesting feature to introduce, and completely feasible with existent tooling. This would introduce coverage on the quality of existing tests themselves.

The program need to pre-process all existing code and capture possible mutations, such as:

Logical operators: change range (> to >=), flip (> to <), negate (= to <>), etc.
Mathematical operators: (change ++ to --), etc.
Removing void-returning function calls (functions such as loggers, which don't change state, should be added to an ignore list).

After a list of mutations has be gathered per test suite, they are applied in a loop, running the tests suite after each change. If a test fails, the mutation is ''killed," if no tests fails, the mutation "survived."

Special care needs to be applied in BrightScript for the equality operator, not to be confused with the assignment operator, depending on the context.

Popular tools for other languages are PIT and Stryker.

Unable to have slashes in Param strings

If I am trying to implement testing url strings I may have something like this:

'@Test modifyIP should modify ip from 101 to alternative if 101 exists
'@Params["http://101.rooibos.com", "http://104.rooibos.com", "104"]
function cdnSwitcher__modifyIP_shouldModifyFirstThreeDigitsCorrectlyIf101Exists(startUrl as String, endUrl as String, streamSourcesPrefix as String) as Void
  m.cdnInfo.streamSourcesPrefix = streamSourcesPrefix

  SUT = modifyIP(startUrl, false, m.cdnInfo)

  m.assertEqual(SUT, endUrl)
end function

Currently this fails and we need to remove the :// from the url strings to test. There may also be other characters but this is the only one I've come across so far.

CodeCoverage.brs crashed m.resolvedMap is invalid

rooibosC 3.0.7-beta
rooibosDist version v3.0.1-beta

BrightScript Micro Debugger.
Enter any BrightScript statement, debug commands, or HELP.

Suspending threads...
Thread selected:  1*   ...:/components/CodeCoverage.brs(16)    lineMap = m.resolvedMap[entry.f]

Current Function:
013:  function onEntryChange()
014:  entry = m.top.entry
015:    if entry <> invalid
016:*     lineMap = m.resolvedMap[entry.f]
017:  
018:      if lineMap = invalid
019:        lineMap = {}
020:      end if
Array operation attempted on variable not DIM'd. (runtime error &he7) in pkg:/components/CodeCoverage.brs(16)
016:     lineMap = m.resolvedMap[entry.f]
Backtrace:
#0  Function onentrychange() As Dynamic
   file/line: pkg:/components/CodeCoverage.brs(16)
Local Variables:
global           Interface:ifGlobal
m                roAssociativeArray refcnt=2 count:2
entry            roAssociativeArray refcnt=1 count:3
linemap          <uninitialized>
Threads:
ID    Location                                Source Code
 0    pkg:/source/main.brs(12)                m.screen.show()
 1*   ...:/components/CodeCoverage.brs(16)    lineMap = m.resolvedMap[entry.f]
  *selected

Brightscript Debugger> p m.resolvedMap
invalid

Brightscript Debugger> p m
<Component: roAssociativeArray> =
{
    global: <Component: roSGNode:Node>
    top: <Component: roSGNode:CodeCoverage>
}

Brightscript Debugger> p m.top
<Component: roSGNode:CodeCoverage> =
{
    change: <Component: roAssociativeArray>
    focusable: false
    focusedChild: <Component: roInvalid>
    id: ""
    entry: <Component: roAssociativeArray>
    expectedMap: invalid
    filePathMap: invalid
    resolvedMap: invalid
    save: false
}

.rooibosrc.json:

{
  "outputPath": "source/tests/rooibos/",
  "projectPath": "out/.tmp",
  "testsFilePattern": [
    "**/tests/**/*.brs",
    "!**/rooibosDist.brs",
    "!**/rooibosFunctionMap.brs",
    "!**/TestsScene.brs",
    "!**/TestUtils.brs"
  ],
  "sourceFilePattern": [
    "**/*.brs",
    "**/*.xml",
    "!**/main.brs",
    "!**/tests/**/*.brs",
    "!**/tests",
    "!**/rooibosDist.brs",
    "!**/rooibosFunctionMap.brs",
    "!**/TestsScene.brs",
    "!**/ThreadUtils.brs"
  ],
  "isRecordingCodeCoverage": true
}

add ability to install rooibosDist.brs, from rooibosC

would be great to be able to initialize a project from rooibosC, and keep the dist file up to date

Add posibility to send results to a server

We need to add possibility to send test results to a server.

The make command is not working outside of BSD based os

rooibos/Makefile

Line 27 in 7010306

 cd src && ls | xargs -J % sed -E -e ${COMMENT_LINES_RE} -e ${BLANK_LINES_RE} % >> ../$(DISTDIR)/rooibos.cat.brs 

The problem: The -J arg from xargs command is not available on GNU based system (ex: debian).

'@Only on an itgroup, can cause no tests to run if '@Only is on a different suite

This irksome little bug needs resolving.

if one suite has '@only at the suite level, and another (different suite) has '@only at the group level (but not suite) - then no tests will run.

`AssertArrayContainsOnly` doesn't seem to work...

Brightscript Debugger> ?methodName
RBS_CMN_IsArray

Brightscript Debugger> ?type(methodName)
String

Brightscript Debugger> ?methodName(item)
Function Call Operator ( ) attempted on non-function. (runtime error &he0) in $LIVECOMPILE(48)

Brightscript Debugger> ?RBS_CMN_IsArray(item)
true