Every time the source pipeline is rebuilt, timestamps of the video buffers restart counting from zero.

If the source pipeline restarts while gst_samples is running, the actual timeout value is getting unpredictable, as gst_samples times out if timestamp - self.start_timestamp > timeout_secs, and both timestamp and self.start_timestamp are reset to 0 while timeout_secs stays the same. (https://github.com/drothlis/stb-tester/blob/master/stbt/__init__.py#L1749)

E.g. if we are running wait_for_match('something.png', timeout_secs=20) and the source pipeline restarts after 10 seconds, it raises MatchTimeout after ~30 seconds.

This issue replaces (part of) #214.

stbt.get_config only works for python scripts run via stbt run;
otherwise it raises the following exception:

  File "./run-templatematch-tests.py", line 62, in <module>
    default_noise_threshold = float(stbt.get_config('confirm_threshold'))
  File "/usr/libexec/stbt/stbt.py", line 330, in get_config
    return load_config(tool)[key]
  File "/usr/libexec/stbt/stbt.py", line 997, in load_config
    assert(system_config in files_read)
AssertionError

This isn't a problem for test scripts, which are always run via stbt run. But it is a problem for stand-alone utilities which want to read
configuration values from stbt.conf.

To work around this, stand-alone utilities should use the command-line
tool stbt config instead.

This is quite easy to reproduce on my system with:

while tests/run-tests.sh test_wait_for_match_confirm_methods_produce_different_results
do true ; done

I have also seen it fail on travis-ci.

The failure is always in the first stbt-run of the testcase, i.e.
incorrectly matching with confirm_method="normed-absdiff".

The matching itself should be entirely deterministic, so I'm not sure
what is going on here.

@szemate reports:

Any test that turns off the set-top-box fails when the box is turned
back on with:

Traceback (most recent call last):
File "/usr/libexec/stbt/stbt-run", line 50, in <module>
  execfile(args.script)
File "reboot.py", line 6, in <module>
  power.cold_reboot()
File "/home/szemate/uitests/library/power.py", line 41, in cold_reboot
  ensure_boot_sequence()
File "/home/szemate/uitests/library/power.py", line 55, in ensure_boot_sequence
  timeout_secs=120)
File "/usr/libexec/stbt/stbt.py", line 520, in wait_for_match
  image, timeout_secs, match_parameters=match_parameters):
File "/usr/libexec/stbt/stbt.py", line 349, in detect_match
  for sample in _display.gst_samples(timeout_secs):
File "/usr/libexec/stbt/stbt.py", line 1253, in gst_samples
  sample = self.get_sample(max(10, timeout_secs))
File "/usr/libexec/stbt/stbt.py", line 1237, in get_sample
  raise UITestError(str(gst_sample))
UITestError: Device '/dev/video0' does not support video capture:
Device '/dev/video0' does not support video capture
gstv4l2object.c(2401): gst_v4l2_object_set_format ():
/GstPipeline:pipeline2/GstV4l2Src:v4l2src1:
Call to G_FMT failed: (Bad address)

stbt batch run requires that the test script you run is in a git repository.
We should also support svn, and other version-control systems.

The vagrant script specifies an Ubuntu 12.04 base image.
12.04 doesn't have GStreamer 1 packages.
Thanks to Thomas von Eyben for reporting this issue.

@LewisHaley reports:

I've found some bugs: stbt batch run attempts to auto-classify Blackmagic card
lock up on NoVideo (in check_capture_hardware), however:

1. it greps for "subdevice" which with gstreamer1 is now "device-number"

2. runs lsof -F Lnc /dev/blackmagic${subdevice:-0}, however the
   blackmagic cards are now listed as "/dev/blackmagic/dv*"

3. it doesn't appear that "VideoInputFrameArrived" is part of the gstreamer
   debug anymore, so "Blackmagic card froze" is given at the failure reason
   incorrectly. This also force stops test execution when it isn't necessary to.
   I did stbt tv -f to see if I got video throughput from the BM card, I did, then
   I set the tests off again and they worked without having to restart the PC.

fadfe6e added a flush during pipeline teardown, to avoid what we call "long teardown" with non-live sources like filesrc -- the pipeline doesn't actually get the EOS within 10 seconds so stbt times out and exits forcefully.

Unfortunately this caused a live v4l source (real video-capture hardware) to always suffer from long teardown.

e051ba8 changed the order of sending eos vs. flush_stop, for an unrelated reason (to fix an intermittent long teardown found with a non-live source in our self-tests). This commit also improved the long teardown with our live source, from 100% of test runs to ~2% of test runs. I did a manual bisection (this table goes from oldest commit at the top to newest commit at the bottom):

TODO:

• Decide whether we want to keep the improved behaviour for non-live sources, or revert both fadfe6e and e051ba8.
• Revert fadfe6e and e051ba8.

When running a local script with extra/runner/run I always receive this failure:
test.py ... ./run: line 101: ts: command not found
./run: line 100 ts: command not found
FAILED

Line 100:
ts '[%Y-%m-%d %H:%M:%.S %z] ' < rawout | tee stdout.log >&3 &

Line 101:
ts '[%Y-%m-%d %H:%M:%.S %z] ' < rawerr | tee stderr.log >&4 &

Commit dd44303 changed the wait_for_match signature from:

wait_for_match(
    image,
    timeout_secs=10,
    consecutive_matches=1,
    match_method=1,
    match_threshold=0.8,
    confirm_method=2,
    erode_passes=1,
    confirm_threshold=None,
    noise_threshold=None)

to:

wait_for_match(
    image,
    timeout_secs=10,
    consecutive_matches=1,
    noise_threshold=None,
    **kwargs)

because the default values for the templatematch-related parameters are
now loaded from stbt.conf.

On further reflection, this makes the API slightly more obscure (yes
it's documented in the man page, but developers often like to refer
to the source code directly), and it also means that tools like pylint
won't catch mis-spellings of those parameter names.

The explicit parameter names need to return (in wait_for_match and the
other related functions too), but should default to None which means
"use the default value from stbt.conf".

https://travis-ci.org/drothlis/stb-tester/builds/40982063 :

ERROR: _stbt.control.test_that_virtual_remote_is_symmetric_with_virtual_remote_listen
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
File "/home/travis/build/drothlis/stb-tester/_stbt/control.py", line 519, in test_that_virtual_remote_is_symmetric_with_virtual_remote_listen
vr = uri_to_remote('vr:localhost:2033')
File "/home/travis/build/drothlis/stb-tester/_stbt/control.py", line 40, in uri_to_remote
return factory(**m.groupdict())
File "/home/travis/build/drothlis/stb-tester/_stbt/control.py", line 123, in __init__
self._connect()
File "/home/travis/build/drothlis/stb-tester/_stbt/control.py", line 132, in _connect
return _connect_tcp_socket(self.hostname, self.port)
File "/home/travis/build/drothlis/stb-tester/_stbt/control.py", line 437, in _connect_tcp_socket
s.connect((address, port))
File "/usr/lib/python2.7/socket.py", line 224, in meth
return getattr(self._sock,name)(*args)
error: [Errno 111] Failed to connect to remote control at localhost:2033: [Errno 111] Connection refused

...but not when run using ./tests/run-tests.sh test_record on master commit 2504477.

Showing '/tmp/stb-tester.BKF/log':
stbt-record: Arguments:
control: test
verbose: 1
source_pipeline: videotestsrc is-live=true ! video/x-raw-yuv,width=320,height=240
control_recorder: file:///dev/fd/63
output_file: test.py
sink_pipeline: ximagesink sync=false
Traceback (most recent call last):
  File "stb-tester/tests/../stbt-record", line 67, in <module>
    sys.exit(main(sys.argv))
  File "stb-tester/tests/../stbt-record", line 40, in main
    record(stbt.uri_to_remote_recorder(args.control_recorder), script)
  File "stb-tester/tests/../stbt-record", line 53, in record
    buf = stbt.get_frame()
  File "stb-tester/stbt.py", line 531, in get_frame
    return _display.capture_screenshot()
  File "stb-tester/stbt.py", line 709, in capture_screenshot
    return gst_to_opencv(self.appsink.get_property("last-buffer"))
  File "stb-tester/stbt.py", line 855, in gst_to_opencv
    (gst_buffer.get_caps().get_structure(0)["height"],
AttributeError: 'NoneType' object has no attribute 'get_caps'

With GStreamer 1.0 there is a 20s delay during VidiU pipeline shutdown. This is probably a GStreamer bug. Investigation required.

There's some kind of dependency failure in the Makefile because make install
will fail if stbt-camera.d/gst/stbt-gst-plugins.so does not exist. Doing
make stbt-camera.d/gst/stbt-gst-plugins.so beforehand makes make install
work (but make install-stbt-camera fails!). From what I understand, the camera
stuff is not (should not) be built unless enable_stbt_camera=yes on the
command line.

@szemate reports:

The most serious issue is that the video processing is very slow and
not enough frames are dropped. [For example a] delay from pressing EPG
to the EPG actually appearing on screen. I had to increase
timeout_secs because the latency between the set-top-box output and
stb-tester was higher than 10 seconds.

Since porting to GStreamer 1.0 there has been a ~4x slowdown in terms of getting frames through stb-tester. This seems to be because it takes a long time to copy the contents of the numpy arrays into output GStreamer buffers required to display on screen and record the video.

This could be fixed by finding a faster way to copy the data or by avoiding copying the data in the first place.

If you call stbt.frames() while an iterator from a previous call to
stbt.frames() is still alive, attempting to iterate over the second
iterator will hang indefinitely:

it1 = stbt.frames()
for frame, timestamp in it1:
    ...
    break
it2 = stbt.frames()
it2.next()  # This deadlocks

The cause is the line with self.lock in stbt.Display.frames.
This lock is intended to stop stbt.Display.on_new_buffer from acting
while the user code is processing frames; but it has the unintended
consequence of preventing the user from using 2 different frames
iterators.

For now, a workaround is to explicitly delete the first iterator before
creating the second one. (In the example above, insert del it1 before
it2 = stbt.frames().)

Separate from the non-beta packages, so that existing users don't get the beta package (which requires config changes) if they do yum upgrade.

If your stbt.conf has save_video=[anything] and you want to not
save a video for a specific stbt run call, the way to turn save_video
is to put save_video= (i.e. completely blank). save_video=False
results in "False.webm". It would be less
confusing if these were valid options:
save_video=True : save "video.webm" ϟ
save_video=False : don't save a video
save_video=[anything else] : save [anything else].webm (as it currently is)

ϟ Or maybe "[name of test].webm"?

It is useful to be able to record a sequence of actions on the set top box
whilst monitoring the output on stbt tv. Also it might be useful (if possible)
to capture screenshots whilst stbt tv is running, instead of having to close
stbt tv and then running stbt screenshot.

This is so errors such as "Blackmagic card froze" or "Blackmagic card in use" are correctly reported as Errors/Yellow in stbt report.

Tested by running:

stbt run -v <(echo 'wait_for_match("some-image.png", timeout_secs=600)')

and while it's running, unplug and re-plug the video input to the HDPVR.

This last worked with 0.13. Neither 0.14 nor the current master branch
(fee91f6) recover from the initial video loss. You just get this over
and over:

stbt-run: warning: Attempting to recover from video loss: Stopping source pipeline and waiting 5s...
stbt-run: warning: Restarting source pipeline...
stbt-run: warning: Restarted source pipeline
stbt-run: warning: Attempting to recover from video loss: Stopping source pipeline and waiting 5s...
stbt-run: warning: Restarting source pipeline...
stbt-run: warning: Restarted source pipeline
stbt-run: warning: Attempting to recover from video loss: Stopping source pipeline and waiting 5s...
stbt-run: warning: Restarting source pipeline...
stbt-run: warning: Restarted source pipeline
stbt-run: warning: Attempting to recover from video loss: Stopping source pipeline and waiting 5s...
stbt-run: warning: Restarting source pipeline...
stbt-run: warning: Restarted source pipeline

With GStreamer 1.0 and the BlackMagic Intensity Pro during pipeline teardown (after the test script has run to completion) stbt run hangs indefinitely.

Dependencies:

• stbt.match (pull request #182)
• stbt.match_all (pull request #212)

Transparency support (stbt.match ignores transparent pixels in the reference image):

• #542.

Translucency support (match things on a translucent overlay, ignoring the background content that shows through):

• Try thresholding frame & template images to eliminate background #432
• Try implementing our own first pass: Scoring algorithm gives different weight to each pixel based on alpha
  • Figure out weighting
  • Figure out the best colour space to do this in
• Try SSIM as second pass
• ???

Description of the test

• The set-top-box gets power-cycled.
• The test periodically attempts to enter the main menu. The test passes if it's possible to enter the main menu within 5 minutes from turning the power on.
• During boot-up the set top box displays two splash screens before getting back to live TV.
• The two splash screens have different resolutions, therefore the VidiU encoder restarts twice.

Issues

(1.) Running the test with --save-video and the following source pipeline:

source_pipeline = rtmpsrc location=rtmp://testaut-2-pc-1/live/testaut-2-stb-1\ live=1 ! decodebin ! videoscale ! video/x-raw,width=1280,height=720

results in the following GStremer error:

GStreamer error: negotiation problem.
gstvideoencoder.c(1368): gst_video_encoder_chain (): /GstPipeline:pipeline1/GstVP8Enc:vp8enc0:
encoder not initialized

Logs: https://github.com/szemate/stb-tester/blob/save-video-issue/logs/with-save-video-with-videoscale-without-videorate.log

(2.) Without --save-video the test is running happily.
Logs: https://github.com/szemate/stb-tester/blob/save-video-issue/logs/without-save-video-with-videoscale.log

(3.) Interestingly, with --save-video but without resizing the video:

source_pipeline = rtmpsrc location=rtmp://testaut-2-pc-1/live/testaut-2-stb-1\ live=1

there is no GStreamer error but the test fails with NoVideo.
Logs: https://github.com/szemate/stb-tester/blob/save-video-issue/logs/with-save-video-without-videoscale-without-videorate.log

(4.) I found a useful forum post[1] that recommended adding videorate to the source pipeline because the H.264 encoder doesn't like that the timestamps restart counting from zero when the source pipeline is rebuilt. Modifying the video saving pipeline as:

video_pipeline = (
    "t. ! queue leaky=downstream ! videorate ! videoconvert ! "
    "vp8enc cpu-used=6 min_quantizer=32 max_quantizer=32 ! "
    "webmmux ! filesink location=%s" % save_video)

eliminates the above errors, however, the saved video is useless as it only contains a few frames from before the box gets power-cycled.
Saved video: https://raw.githubusercontent.com/szemate/stb-tester/save-video-issue/logs/corrupted-video.webm
Logs: https://github.com/szemate/stb-tester/blob/save-video-issue/logs/with-save-video-with-videoscale-with-videorate.log

[1] http://gstreamer-devel.966125.n4.nabble.com/dynamic-pipeline-getting-not-negotiated-error-when-adding-filesink-tp4661824p4661825.html

Configuration

I was running the tests with:

• stb-tester 0.20
• Fedora 20
• Linux kernel 3.15.6-200
• GStreamer 1.2.4
• libvpx 1.3.0

My stbt.conf:

[global]
control = irnetbox:testaut-2-switch-pc.dev.youview.co.uk:1:/home/szemate/uitests/config/humax.irnetbox.conf
power_outlet = aten:testaut-2-pdu.dev.youview.co.uk:1
restart_source = False
sink_pipeline = xvimagesink sync=false
source_pipeline = rtmpsrc location=rtmp://testaut-2-pc-1/live/testaut-2-stb-1\ live=1 ! decodebin ! videoscale ! video/x-raw,width=1280,height=720

The javascript of index.html breaks with an error in tablequeryjs if any of the cells of the test results' table is longer than ~150 characters. As a result it is not possible to open test run details. This issue is easy to reproduce by inserting a long text into the "notes" column and reloading the page.

Whenever I use mutli-threading in my stbt scripts (for detecting random errors that cannot be expected by the script, like a popup error after disconnecting the network cable manually), from time to time there seems to be race conditions about the video frame, if I got the error message right. The error is not reproducible 100% and happens only intermittent. Most probably this could be fixed by Lock, but I would assume the stbt internals themselves need to respect the Lock as well.

Is there a plan to support multi-threading and fix this?

Exception:

UnboundLocalError: local variable 'res' referenced before assignment
Traceback (most recent call last):
 File "/usr/lib/stbt/stbt-run", line 49, in <module>
    execfile(args.script)
 File "/usr/lib/stbt/stbt.py", line 613, in wait_for_match
    raise MatchTimeout(res.frame, _template_name(image), timeout_secs)  # pylint: disable=W0631,C0301

Code example:

import threading
self._stop = threading.Event()
self._thread = threading.Thread(name='threadname', 
        target=do_something_in_parallel, 
        args=(self._stop,1))
self._thread.daemon = True
self._thread.start()
 # wait for something special to happen
self._stop.set() # Fire the stop event to stop the parallel action

def do_something_in_parallel(stop):
  while not stop.isSet():
    # do your parallel stuff

It would be very useful if the higher-level functions which called match allowed the user to pass through a region parameter to match.

As per Lewis's comment:

FYI the voaacenc element is not available on Fedora (including via RPMFusion) because
of licence issues regarding the AAC audio patent [1]. This is what is causing the selftests
to hang.

[1] https://bugzilla.rpmfusion.org/show_bug.cgi?id=1742

aacenc is used in generating the calibration videos and in the selftests.

What we'll do for the 0.21 release is disable stbt camera & associated selftests on Fedora.

In tests/run-tests.sh we set GST_PLUGIN_PATH to point at the source directory.
We also delete the GStreamer registry (cache); see 1e2f7d1.

Since 74cc837 (where we use gst-launch-0.10 instead of gst-launch),
deleting the GStreamer registry seems (I think) to only take effect
for the very next invocation of gst-launch-0.10:

rm ~/.gstreamer-0.10/registry.*
gst-launch-0.10 [...] stbt-templatematch [...]  # Uses the plugin from the source directory
gst-launch-0.10 [...] stbt-templatematch [...]  # Uses the plugin from /usr/lib/gstreamer-0.10/

This doesn't happen (I think) when using gst-launch instead of gst-launch-0.10.

This is on Fedora 17, where
/usr/bin/gst-launch-0.10 is provided by the "gstreamer-0.10.36-1" package, and
/usr/bin/gst-launch is provided by the "gstreamer-tools-0.10.36-1" package.

Backtrace reported by @LewisHaley:

Thread 1 (Thread 0x7fa57cc5a700 (LWP 16740)):
#0  0x00007fa57fdee3a4 in RTMP_ReadPacket () from /lib64/librtmp.so.1
#1  0x00007fa57fdf2667 in RTMP_GetNextMediaPacket () from /lib64/librtmp.so.1
#2  0x00007fa57fdf2778 in Read_1_Packet () from /lib64/librtmp.so.1
#3  0x00007fa57fdf31ec in RTMP_Read () from /lib64/librtmp.so.1
#4  0x00007fa5841da36a in gst_rtmp_src_create () from /usr/lib64/gstreamer-1.0/libgstrtmp.so
#5  0x00000030aa42ac7e in gst_base_src_get_range () from /lib64/libgstbase-1.0.so.0
#6  0x00000030aa42c74b in gst_base_src_loop () from /lib64/libgstbase-1.0.so.0
#7  0x00000030aa090559 in gst_task_func () from /lib64/libgstreamer-1.0.so.0
#8  0x00000035cc86f406 in g_thread_pool_thread_proxy () from /lib64/libglib-2.0.so.0
#9  0x00000035cc86ea45 in g_thread_proxy () from /lib64/libglib-2.0.so.0
#10 0x00000035ca807f33 in start_thread () from /lib64/libpthread.so.0
#11 0x00000035ca0f4ded in clone () from /lib64/libc.so.6

This happens about 3% of test runs. It happens during the pipeline
teardown, and the test run is otherwise successful. (It's still annoying
though because the test run gets reported as an error, and the recorded
video isn't seekable.)

I've seen the "Blackmagic card froze" classification be incorrectly given with the captured decklinksrc.log looking like:

[2015-01-14 18:08:00.402810 +0000]  run: Checking Blackmagic video-capture device
[2015-01-14 18:08:02.109335 +0000]  0:00:01.593533503  5119      0x21b0070 INFO             decklinksrc gstdecklinksrc.cpp:364:gst_decklink_src_send_event:<decklinksrc0> EOS pending
[2015-01-14 18:08:02.109422 +0000]  0:00:01.593598653  5119      0x218e450 INFO             decklinksrc gstdecklinksrc.cpp:770:gst_decklink_src_task:<decklinksrc0> EOS pending
[2015-01-14 18:08:02.109448 +0000]  0:00:01.593617981  5119      0x218e450 INFO             decklinksrc gstdecklinksrc.cpp:932:gst_decklink_src_task:<decklinksrc0:videosrc> pushing EOS event
[2015-01-14 18:08:02.109470 +0000]  0:00:01.593638790  5119      0x218e450 INFO             decklinksrc gstdecklinksrc.cpp:934:gst_decklink_src_task:<decklinksrc0:audiosrc> pushing EOS event

When the subsequent test starts, the Blackmagic card works correctly. I think that the error is that the sleep 1 is sometimes not long enough for the card to activate and thus "VideoInputFrameArrived" is not printed by the Gstreamer debug. I going to run some tests to see if I can reproduce this.

The new implementation of wait_for_match (ca4201d) occasionally gives
incorrect results because the OpenCV matchTemplate gives the wrong output,
but only when using decklinksrc (the GStreamer element for Blackmagic
video-capture cards).

See https://github.com/drothlis/decklinksrc-stress-tests

I'm finding that 100% of the time my saved videos with stbt-0.20 and a VidiU
are unseekable (and e.g. Firefox shows the video as being massively long even
though it isn't).

My source pipeline is simply
rtmpsrc location=rtmp://wll1p00066/live/lewis\ live=1
as suggested in docs/backward-compatibility.md.

I think this is usually a problem with the file not indexing/being indexed correctly
on EOF (or something like that).

It would be very useful to be able to link to a stbt report with a filter search as part of the URL,
as opposed to having to give a link and a filter string to input.

@szemate reports:

stbt-run ignores ^C (keyboard interrupt).
It prints the following exception and does not exit:

Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/lib64/python2.7/threading.py", line 811, in
__bootstrap_inner
  self.run()
File "/usr/lib64/python2.7/threading.py", line 764, in run
  self.__target(*self.__args, **self.__kwargs)
File "/usr/lib64/python2.7/site-packages/gi/overrides/GLib.py", line 518, in run
  raise KeyboardInterrupt
KeyboardInterrupt

then at the end of the test it raises the following exception:

Traceback (most recent call last):
File "/usr/lib64/python2.7/site-packages/gi/overrides/GLib.py", line
629, in <lambda>
  return (lambda data: callback(*data), user_data)
File "/usr/libexec/stbt/stbt.py", line 1375, in restart_source
  self.source_pipeline.set_state(Gst.State.NULL)
AttributeError: 'NoneType' object has no attribute 'set_state'

This is with the HDPVR and the following stbt.conf:

[global]
source_pipeline = v4l2src device=/dev/video0 ! tsdemux ! h264parse
sink_pipeline = xvimagesink sync=false
restart_source = true

press_until_match is a useful alternative to a sequence of waits,
and would benefit from having all parameters that it does.

Dear stb-tester team,
I did a fresh install of Ubuntu 14.04 and stb-tester 0.21 and stb-tester-camera.
When trying to run the camera calibration stbt --with-experimental camera calibrate I get stuck when trying to analyse the colours.ts.
I cleared the .cache/stbt videos in order to re-create the reference videos. The colours.ts video is recognized, it looks like the script reads in some values and plots it to the colours graph, but it does not stop at 50 (as it did in former versions), but keeps on running endless.
[#####################################] 102 / 50

I uploaded a video for reference:
https://11968737038841875939.googlegroups.com/attach/e5e13e943065fd20/out-1.ogv
Looks like a bug to me

in 0.21.

Best,
Robert

stbt._display.last_sample stores the most recent GSt buffer until stbt.gst_samples removes it or it's replaced by a newer buffer. To put it other way, if the video stream stops then gst_samples
yields the very last sample of the video stream even though it's called minutes later.

This behaviour has unwanted effects in the following situations:

stbt.press("POWER_OFF")
time.sleep(5)
# Make sure that the device under test is off
try:
    stbt.get_frame()
except stbt.NoVideo:
    pass
else:
    raise stbt.UITestFailure("The device is still on")

In this case stbt.get_frame never raises NoVideo, as it returns the
sample that had been stored before the box was powered off.

stbt.wait_for_match("menu.png")
stbt.press("POWER_OFF")
time.sleep(5)
stbt.press("POWER_ON")
# Make sure that we no longer see menu.png because
# it should disappear if the device under test is restarted
try:
    stbt.wait_for_match("menu.png")
except stbt.MatchTimeout:
    pass
else:
    raise stbt.UITestFailure("menu.png hasn't disappeared")

In this case we have a race condition between stbt and the device under
test. The device needs some time to start delivering frames after it's
powered on; if it doesn't happen before gst_samples is called then
wait_for_match("menu.png") returns false positive as the first sample
that gst_samples yields is the one that had been stored before the
device was powered off.

runner/run does not consider a test no existing ("IOError: [Errno 2] No such file or directory") an interesting failure, runner/run -k does not exit. Really, we don't want runner/run -kk (keep running regardless of any failures) to continue running either, so runner should check that all test paths are accurate before starting.

Because src_rpm is defined as stb-tester-$(ESCAPED_VERSION)-1.fc20.src.rpm.

The -1 before fc20.src.rpm indicates the release version is hardcoded. I'm not sure if this is intentional, but it's a bit confusing for the uninitiated.

From @LewisHaley's comment on #199:

From what I can see, there are a couple of problems with tests/test-camera.sh.
The first being that test-camera is still run even if the package dependencies
for it aren't installed (all the C++ devel packages). That being said, some of the
tests hang even once all the dependencies are installed and the plugins are
built. I've specifically seen the test test_that_validation_video_served_over_http_is_correct
hang repeatedly.

This would be useful because sometimes the result of a stbt.press()
(on screen) does not linger long enough for all necessary template
matching to occur.

All that needs to be done is to make the _match_template() function
in stbt.py public, so users can put as many of them inside a stbt.frames()
loop as they require, (and give it a docstring, documentation, etc). The user
can then handle the "Have I got all the matches/nomatches that I want for this
frame?"

@szemate reports:

I occasionally got the following exception, right after starting a test:

Traceback (most recent call last):
File "/usr/libexec/stbt/stbt-run", line 50, in <module>
  execfile(args.script)
[...]
File "/usr/libexec/stbt/stbt.py", line 349, in detect_match
  for sample in _display.gst_samples(timeout_secs):
File "/usr/libexec/stbt/stbt.py", line 1253, in gst_samples
  sample = self.get_sample(max(10, timeout_secs))
File "/usr/libexec/stbt/stbt.py", line 1237, in get_sample
  raise UITestError(str(gst_sample))
UITestError: Internal data flow error.: Internal data flow error.
gstbasesrc.c(2812): gst_base_src_loop ():
/GstPipeline:pipeline1/GstAppSrc:appsrc:
streaming task paused, reason not-negotiated (-4)

This is with the HDPVR and the following stbt.conf:

[global]
source_pipeline = v4l2src device=/dev/video0 ! tsdemux ! h264parse
sink_pipeline = xvimagesink sync=false
restart_source = true

The video recorded by stbt run --save-video and stbt batch run isn't seekable. In VLC it displays an exceedingly long duration like 351909:20:18.

The irNetBox device only allows one TCP connection at a time, so one client
emitting IR signals could cause other clients to fail. So we retry the TCP
connection to the irNetBox 5 times with exponential backoff (see
IRNetBox.__init__ in irnetbox.py).

Apparently even re-trying 5 times isn't enough; with more than 10
clients running stbt test scripts, we start seeing "failed to connect"
even after the 5 retries.

I think that the best solution will be a proxy server, speaking the same
protocol as the irNetBox, sitting between the clients and the irNetBox.
The clients would connect to the proxy (which would support multiple
simultaneous connections), and the proxy would forward the messages from
the clients on to the irNetBox over a single TCP connection. (And
vice-versa for the responses from the irNetBox to the clients.)

Clients wouldn't even need to know that they aren't talking to a real
irNetBox.

If we put the implementation right in irnetbox.py then we can run the
proxy server like this:

python -m irnetbox [[<interface to listen on>:]<port to listen on>] <irNetBox address>

Overlay text on output video of commands sent with stbt.press().
It must persist for ~1s so must be drawn over each frame. This makes
the output video easier to analyse than having to compare what happens
on screen to the logs.

[2014-07-07 22:17:59.325469 +0100]    File "/home/lewis.haley/libexec/stbt/stbt.py", line 425, in detect_match
[2014-07-07 22:17:59.325500 +0100]      for sample in _display.gst_samples(timeout_secs):
[2014-07-07 22:17:59.325531 +0100]    File "/home/lewis.haley/libexec/stbt/stbt.py", line 1512, in gst_samples
[2014-07-07 22:17:59.325562 +0100]      sample = self.get_sample(max(10, timeout_secs))
[2014-07-07 22:17:59.325594 +0100]    File "/home/lewis.haley/libexec/stbt/stbt.py", line 1493, in get_sample
[2014-07-07 22:17:59.325625 +0100]      self.source_pipeline, Gst.DebugGraphDetails.ALL, "NoVideo")
[2014-07-07 22:17:59.325655 +0100]  TypeError: Argument 0 does not allow None as a value

The source of the error:

def get_sample(self, timeout_secs=10):
    try:
        # Timeout in case no frames are received. This happens when the
        # Hauppauge HDPVR video-capture device loses video.
        gst_sample = self.last_sample.get(timeout=timeout_secs)
        self.novideo = False
    except Queue.Empty:
        self.novideo = True
        Gst.debug_bin_to_dot_file_with_ts(
            self.source_pipeline, Gst.DebugGraphDetails.ALL, "NoVideo")
        raise NoVideo("No video")
    if isinstance(gst_sample, Exception):
        raise UITestError(str(gst_sample))

    return gst_sample

Other than that (presumably) Display.source_pipeline is None when
Gst.bin_to_dot_file_with_ts is called I don't know what the error is.
There is a
stbt-run: warning: Attempting to recover from video loss: Stopping source pipeline and waiting 5s...
just before the exception is raised, so perhaps a race condition.

Of particular annoyance is that the exception message (and thus the
failure-reason in stbt batch run is incredibly vague and not obviously
an infrastructure problem.

The code was added in 7b3eaf3.