khaivngo / quickeval Goto Github PK

In the scientist mode, I incorporated 10 image groups in one experiment. Then, in the observer mode, as an observer I did give ratings on all of them. But the online and exported reports gave results only for group 2 to 10. The results for group 1 were missing. Perhaps you can have a try on the experiment “Perceived Projection Contrast”. It is available on the website: http://www.ansatt.hig.no/mariusp/quick/index.php

There is no option to display test images in a totally randomized order at the initial status of the experiments. The shuffled order should be carefully recorded and reflected in reports. Now it simply displays the images with respect to the original uploading order. If the scientist manually change the uploading order of test images, then the calculated Z-scores presented in the reports are useless.

Adding more advanced screen calibration methods. (I.e. search for other tools to do this than what is currently implemented).

Original - write message to scientist that an original is always needed

It would be great if one could "tick off" Isihara test while creating an experiment, so that it will "force" the observer to do it before commencing the experiment.

-Tendency to display a column with undefined data.
-Pictures with long titles forces the table out of parent width.

I cannot see a function to remove instructions from the history. This might be a nice feature to add, since it might become many instructions after a while (especially with the testing that has been done).

There also exists methods that allows to do incomplete experiments (i.e. filling in the missing parts of the data). These methods could be considered as a feature to reduce the number of comparisions needed (for example in paired comparison).

Trying quickeval on my android tablet i am not able to select an image, i can only pan.

Changing the background color does not work. I tried setting it to 255255255, 101010, and 000, but it is still gray (i.e. no change)

Maybe we should consider adding the feature to modify existing experiments (for example changing the instructions only). Perhaps not changing the experimental method, but some settings.

The raw data from rank order experiments are not displayed correctly. For the first image nothing is shown (see attached image). While for the second image, the data from the first and second is shown.

It seems like there is a problem with calculation of z-scores. I set up an experiment, and based on the number of observations it should be able to calculate z-scores. I have attached a screenshot. The data is available on http://www.ansatt.hig.no/mariusp/quick/

The test images displayed in the rank order mode were corrupted by the gray masks whenever the images were dragged to leave their initial positions. It can be very difficult for observers to determine whether the images were simply masked or the test image were distorted for experimental purpose. Alternatively, it is possible to apply colors to the labels or make them bold for example rather than corrupting the test images.

For example, this is the initial status of rank order without dragging anything.

After dragging a few images.

The fields "name of the image set" and "description" is not removed/cleared when the "X" is clicked.

In many cases physically printed images are compared to an original on a monitor. It would be nice to have this as a feature in QuickEval, where the responses from the observers are recorded.

Separate z-scores and raw data with a title for each (and possibly with a horizontal line).

Suggestion: show on the result page the number of observers per image and the average time that the observers have used. (i.e. something similar to have is shown on the scientist panel dashboard).

I tried to change user access for a user from "observers" to "scientist", but it does not change. It seems like it is not updated in the database.

In the "upload images" page it should be mentioned what type of image formats that are supported.

In the "development" branch the raw data for paired comparison data is not shown. The table is there, but it is empty.

Another useful extension of QuickEval would be to support evaluation of videos. This would require other evaluation methods.

It seems like the order in which the image pairs are shown in paired comparison is the same for all observers. Is this correct?

It would be great to have this in random, and that the pairs are shown in random as the default setting. Having the same order can still be an option.

Show results - Having more distortions/images than 5, the graph only shows three images. (i.e. the three first ones in the image set)

I am using Windows 8.1 Pro and Firefox 35.0.1

It seems to be that the results that are being calculated (Z-scores) and the raw data shown are including "incomplete" data (i.e. observers who have not completed the experiment) in addition to complete data (i.e observers who have completed the entire experiment).

It would be great if one could decide to show data with only complete data, and an option to show them with incomplete data.

Softcopy quality ruler could also be a method to include. In the paper "Adapting ISO 20462 Softcopy Quality Ruler Method for on-line Image Quality Studies" by Peter D. Burns, Jonathan B. Phillips and Don Williams, it was shown that it is possible to use this method online.

The bottom pan field with thumbnails is overlapping with the images when when the computer has a lower resolution/window.

Maybe the easiest solution here is just to have the page to scroll.

Add "viewing distance" as meta data for experiments.

The z-score calculation is not correct. The sum of the mean z-score should always sum to 0. In the test attached, the sum is below 0.

The idea is to be able to display the current access level of a user without submitting or changing is access level, and also display is new access level upon successful submission.

Utilize existing that already checks for if email existing and modify so it also returns access level of current user.

Under set up experiment, when you want to add a custom field to ask the observer to fill. If you remove this field then the page goes "blank". See attached image.

Adding screen calibration as an option when setting up the experiment.

In the case of having more than 5 levels of image distortions, some distorted images at the bottom in the scrolling list will be invisible. In this case, for example, it would be very strange for the observer to insert distorted image 7 in front of distorted image 1. The situation can be even worse if there are more image distortion levels. One possible solution could be moving all distorted images or the scrolling bar automatically when the observer drag the image distortions.

In the thumbnail list in rank order should probably have an option when setting up the experiment where the scientist can choose to have the thumbnails shown in a random order (now it is shown in the same order as the images are uploaded).

I think that the default setting should be random.

Observer mode : front page -> when a user has selected an experiment, it is not obvious that he/she needs to press "start experiment" below . It might be good to highlight this better. Either with a box around the experiment or something similar.

Jump down to page or something similar

If the image sets contain varying number of images, then the z-score plot showing a "strange" label. In the example attached, the first set has 3 images, and the second has only 2. When showing the results for the second image set, there is a label named "2" that shows up. This should not be shown, and the plot should be changed to having only two images/labels.

When clicking the icon for showing the image in a new tab/window, it is opened twice. Happens at least in Chrome. The first tab/window has a gray background, the second a white.

Triplet comparison could be an interesting experimental method to include. Description of the method can be found in ISO 20462-2:2005 Photography -- Psychophysical experimental methods for estimating image quality -- Part 2: Triplet comparison method.