ibm-cloud / openwhisk-darkvisionapp Goto Github PK

Problem

Solution

On hover the icons should change to color: #E35853.

Problem

There are minor issues with vertical spacing, line-height, and font-size on the Video Details page.

Solution

See cloudant/nodejs-cloudant#154 (comment)

In the old UI design we had counters at the top showing the total number of videos/images vs. the number of videos/images to be processed. The new UI design is missing these information showing Dark Vision at work.

Take this image http://www.socialdemokraterna.se/upload/Stefan_Lofven/Bilder/Stefan%20Lofven.jpg

It is a 4MB file, 5616 x 3744. ImageMagick conversion fails - seems like it requires more memory than available to the action.

Activation: analysis (ccc)
[
    "2017-04-03T08:35:50.39339173Z  stdout: [ bbb ] Processing image.jpg from document",
    "2017-04-03T08:35:51.875478614Z stdout: [ bbb ] KO ( 1.479 s) { Error: Command failed:",
    "2017-04-03T08:35:51.875511335Z stdout: at ChildProcess.onExit (/nodejsAction/node_modules/gm/lib/command.js:301:17)",
    "2017-04-03T08:35:51.875520462Z stdout: at emitTwo (events.js:106:13)",
    "2017-04-03T08:35:51.875526287Z stdout: at ChildProcess.emit (events.js:191:7)",
    "2017-04-03T08:35:51.875532128Z stdout: at maybeClose (internal/child_process.js:877:16)",
    "2017-04-03T08:35:51.87553974Z  stdout: at Process.ChildProcess._handle.onexit (internal/child_process.js:226:5) code: null, signal: 'SIGKILL' }"
]

• https://github.com/watson-developer-cloud/node-sdk/#natural-language-understanding
• https://www.ibm.com/watson/developercloud/natural-language-understanding/api/v1/#introduction
• cf create-service natural-language-understanding free <service-name>

• update OpenWhisk action
• change toolchain
• update README.md
• update DEPLOY_MANUALLY.md
• remove AlchemyLanguage

for some reason creating trigger fails with cryptic error:

macs-mbp:processing aslom$ wsk trigger create vision-cloudant-trigger --feed vision-cloudant/changes    -p dbname cloudant-a1 -p includeDoc true
error: failed to create trigger feed vision-cloudant-trigger
{
    "activationId": "75e2430d583741768406f1f558653cbc",
    "annotations": [],
    "end": 1468607989929,
    "logs": [],
    "name": "changes",
    "namespace": "[email protected]",
    "publish": false,
    "response": {
        "result": {
            "error": null
        },
        "status": "application error",
        "success": false
    },
    "start": 1468607986701,
    "subject": "[email protected]",
    "version": "0.0.52"
}

no matter what parameters are passed or not ...

macs-mbp:processing aslom$ wsk trigger create vision-cloudant-trigger --feed vision-cloudant/changes
error: failed to create trigger feed vision-cloudant-trigger
{
    "activationId": "a5719acbaef64a83802e92b7e6d0ef33",
    "annotations": [],
    "end": 1468605016622,
    "logs": [],
    "name": "changes",
    "namespace": "[email protected]",
    "publish": false,
    "response": {
        "result": {
            "error": null
        },
        "status": "application error",
        "success": false
    },
    "start": 1468605014490,
    "subject": "[email protected]",
    "version": "0.0.52"
}

and here is full script output:

macs-mbp:processing aslom$ ./deploy-darkvision.sh --install
Current namespace is [email protected]_dev.
Creating vision package
ok: created package vision
Adding service credentials as parameter
ok: updated package vision
Binding cloudant
ok: created binding vision-cloudant
Creating trigger
error: failed to create trigger feed vision-cloudant-trigger
{
    "activationId": "4447d0dc800041b5bb5b50d6f83f5131",
    "annotations": [],
    "end": 1468604389260,
    "logs": [],
    "name": "changes",
    "namespace": "[email protected]",
    "publish": false,
    "response": {
        "result": {
            "error": null
        },
        "status": "application error",
        "success": false
    },
    "start": 1468604388527,
    "subject": "[email protected]",
    "version": "0.0.52"
}
Creating actions
ok: created action extractor
ok: created action analysis
Creating change listener
ok: created action vision-cloudant-changelistener
Enabling change listener
error: [email protected]_dev/vision-cloudant-trigger does not exist (code 87972)
macs-mbp:processing aslom$

Similar to what has been done there:
IBM-Cloud/openwhisk-visionapp#2

Problem

Curent vertical spacing between headers is incorrect.

Solution

Problem

We should keep our Video Detail Header (above) consistent with how we display data in our new Processing Details menu (below).

Solution

Redlines:

Frame Detail:

Problem

Spacing between filter components is is incorrect.

Solution

Hello!

It appears that the repository property is not defined in package.json. Please add the following

  "repository": {
    "type": "git",
    "url": "https://github.com/IBM-Bluemix/openwhisk-darkvisionapp/"
  },

to identify your application and require "cf-deployment-tracker-client": "^0.x" to use the latest node Deployment Tracker client. Thank you!

Problem

The space between the words OpenWhisk and Dark Vision is too large.

Solution

As opposed to the current 10px spacing between words, there should just be a normal keyboard space.

The API "/api/videos/:id" in web/app.js performs the filtering of keywords/tags/faces to build the video summary. It ends up providing a static view for a video as the thresholds and minimum count are configured in app.js.

Instead we could return the full data to the client and have the threshold configurable in the client. This way one could play with knob-like controls to show more/less data.

as done here https://github.com/IBM-Bluemix/openwhisk-slackapp/blob/master/deploy.cmd

Problem

Image titles are missing the gray border-bottom that separates them.

Solution

• Try border-bottom: 1px solid #EAEAEA; and see how that looks.
• There should be 20px padding between the border-bottom and the image.

Once most of the UI changes has been done.

How can the Dark Vision web app can be improved to show the new insights coming from the audio stream?

Capabilities of Dark Vision V1 web user interface:

• Upload video
• Upload image
• View list of uploaded videos
• View list of uploaded images
• View results for a video
• View individual images extracted from the video
• View summary results
• View individual results for each image
• Relaunch an analysis of the video (extract + image process)
• Restart the analysis of all images (image process only)
• View results for an image
• Link to GitHub source code
• Link to Youtube video

Counterpart to #37 but to cache the calls made to Cloudant (retrieving videos, images and summary) as they can get quite consuming. However how to invalidate the cache when there is a new upload or when videos, images are being processed?

The implementation in the "audio" branch calls the STT synchronously, thus the OpenWhisk action waits doing nothing.

Instead we could use the asynchronous calls: https://www.ibm.com/watson/developercloud/speech-to-text/api/v1/?curl#async_methods

However this means registering a callback to be called by Watson when the STT process is completed.

• implement async API
• update toolchain
• update README

Instead of storing the attachments into Cloudant, put them in Object Storage and serve images/videos directly from Object Storage.

• abstract storage so that it can switch between putting media in Cloudant or in Object Storage
• implement media storage in Object Storage
• write instructions on how to use Object Storage with Dark Vision
• use Object Storage if configured, else default to Cloudant

I see that we have to upload a mp4 file.
But it will really nice if we could put a youtube link and openwhisk do the job from youtube.

Problem

The styling and behavior of the Filter frames by tag is slightly wonky.

Solution

Lets start by making the copy Filter frames by tag font-style: italic; and color: #B8C1C1.

Lets also extend the red timeline to the bottom of the screen when filtering.

If this ends up looking weird of clumsy we can work on a different solution.

Problem

• The icons are slightly larger than needed, 32px x 32px.
• Spacing between icons is too large, 40px.

Solution

• Icons should be a max of 28px by 28px.
• Icons should have a left margin of 20px (see below).
  
• The Cloud Upload icon should be 22px x 16px with a left margin of 8px(see below). This is the only icon that should have different dimensions.

Problem

Currently the Dark Vision app lacks hover-states for most link items.

Solution

Here's a first-pass at some initial hover-states. We may need to tweak some of them as we go.

1. Header Icons:

2. Processing Details:

3. Filter Bar:

4: Breadcrumb:

5: Scrubber:

6: Video Overview:

It would also be nice to play with some transition easing in areas we haven't yet. Lets try adding transition: all 0.25s ease; to 1,2, 3 & 4.

By doing so, we could have two triggers, one listening for changes of the video, one for images. This could remove the changelistener and call the actions directly. The actions would decide what to do with the event.

When the video frames are stored in Cloudant, analysis.js retrieves the image file without doing any retry.

If many frames are being analyzed, Dark Vision may hit Cloudant rate limit and fail to retrieve the image. It usually results in "not a jpeg file" error when analyzing the image.

Seems to come from an issue registering the Speech to Text callback in the toolchain.

The call to register fails with an error 406. It could be due to a concurrency issue where the speechtotext web action is not yet exposed by OpenWhisk when we try to call it during registration.

A re-run of the pipeline usually solves the issue.

{"code":400,"code_description":"Bad Request","error":"unable to register callback url 'https://openwhisk.ng.bluemix.net/api/v1/experimental/web/user_space/vision/speechtotext.http', callback server responded with status code 406"}

instead load the document from the change listener and look at it.

The Node.js part of the action can be made simpler. The current code was based on the first OpenWhisk Docker SDK. We could merge the client and server apps into one. See https://github.com/ibmets/openwhisk-jq for a clean implementation.

Problem

• Font-sizes are slightly off
• Icon and copy need to be vertically centered
• Nav bar is too tall (60px)

Solution

The follow up to #11. Once we have the transcript we can pass it through Tone Analyzer and display the results in the web/iOS app

After uploading a 30sec mp4 clip to my deployed Dark Vision app, it's getting stuck at 90% every time it tries processing audio - it's been stuck for 1hr+

Any known reasons?

The audio from the video is another dark data to process. First step is to get the text.

• extract audio
• store audio
• add Speech To Text service
• new action calling Speech To Text and persisting output to Cloudant when audio is added as attachment
• show Speech To Text in web ui

• Replace Object Storage Swift with the more recent Cloud Object Storage S3
• Offer the option in the toolchain to create a Cloud Object Storage or use an existing service instance (whether the instance if an actual Cloud Object Storage or a user-provided service with the right credentials set).

The issue seems to be in Speech to Text. It fails if the callback URL has a space in its name:

{"code":400,"code_description":"Bad Request","error":"unable to register callback url 'https://openwhisk.ng.bluemix.net/api/v1/experimental/web/[email protected]_Game of Bluemix/vision/speechtotext.http', callback server responded with status code 505"}

Another related issue fixed by PR #65 was with Cloud Functions:

  ___                __        ___     _     _    
 / _ \ _ __   ___ _ _\ \      / / |__ (_)___| | __
| | | | '_ \ / _ \ '_ \ \ /\ / /| '_ \| / __| |/ /
| |_| | |_) |  __/ | | \ V  V / | | | | \__ \   < 
 \___/| .__/ \___|_| |_|\_/\_/  |_| |_|_|___/_|\_\
      |_|                                         
Retrieving OpenWhisk authorization key...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  1757    0     0  100  1757      0   8514 --:--:-- --:--:-- --:--:--  8487
100  1757    0     0  100  1757      0    869  0:00:02  0:00:02 --:--:--   870
100  1757    0     0  100  1757      0    581  0:00:03  0:00:03 --:--:--   581
100  4067    0  2310  100  1757    737    561  0:00:03  0:00:03 --:--:--   738
error: syntax error, unexpected $end, expecting QQSTRING_TEXT or QQSTRING_INTERP_START or QQSTRING_END
.namespaces[] | select(.name == "[email protected]_Game
                                 ^^^^^^^^^^^^^^^^^^^^^
1 compile error
error: syntax error, unexpected $end, expecting QQSTRING_TEXT or QQSTRING_INTERP_START or QQSTRING_END
.namespaces[] | select(.name == "[email protected]_Game
                                 ^^^^^^^^^^^^^^^^^^^^^
1 compile error
Speech to Text OpenWhisk action is accessible at https://openwhisk.ng.bluemix.net/api/v1/experimental/web/[email protected]_Game of Bluemix/vision/speechtotext.http
 _   _      _      

https://www.ibm.com/blogs/bluemix/2016/12/openwhisk-api-keys-update/

• Use API keys per space
• Turn whisk.async/done into Promise
• Use openwhisk npm instead of whisk

• add AlchemyLanguage service
• new action to analyze the transcript
• query entities
• query concepts
• README update
• show results in UI

before merging #24

experimental/web has graduated.

Dark Vision should move to the new API.

https://medium.com/openwhisk/whats-new-web-actions-e0ffeef1bea3

Note that if you remove "experimental" today from the STT_CALLBACK_URL, it will lead to an error like:

{
  "error": "Response is not valid 'message/http'.",
  "code": 3104139
}

because the code of the speechtotext action needs to be updated.

there should be better logging to track what is going during deployment and execution

for deploying print more what is happening so more help when it fails

for operations print more in logs - for example currently my demo deployment processed two videos but stopped after that with no error i could find ...

The goal of the form will be to provide a field to set the video title, but also to select the video language.

The number of extracted frames is currently set in extract.js with an option to change it by rebuilding the Docker image.

Instead this number could be a property of the video and configured from the upload dialog.

Deleting a video should delete all the related images, audio and attached files (if not stored in Cloudant).

Per the documentation of the NLU service (https://www.ibm.com/watson/developercloud/doc/natural-language-understanding/#supported-languages) and the documentation of speech-to-text (https://www.ibm.com/watson/developercloud/doc/speech-to-text/input.html#models) there is support for multiple languages.

Is there any additional restriction to support them in OpenWhisk?

To ease the deployment of the app, add a toolchain. The toolchain will create the services, the webapp and deploy the OpenWhisk actions/triggers.

• build extractor Docker image with travis and push it to Docker Hub (only for master branch)
• create services
• create service keys
• set env variables from service keys
• initialize database
• deploy web app
• retrieve openwhisk key for space using accessToken from ~/.cf/config.json
• deploy openwhisk artifacts
• toolchain: prompt for GitHub repo
• toolchain: prompt for app name
• toolchain: prompt for login and password
• toolchain: allow customization of Cloudant db, Docker image name, Openwhisk host

As by default we use the rate limited plan of Cloudant, we should reduce the load on Cloudant when browsing the web app.

Caching comes to mind.

We could be caching images loaded from Cloudant by writing them on the local disk (we don't mind if the disk gets lost on restart) and having them served directly by node. Once uploaded images do not change - only the video thumbnail may change if the analysis is retriggered.

Need to make sure the Dark Vision code works with the Lite plan and correctly retries if it hits the rate limits.