ibm-cloud / openwhisk-darkvisionapp Goto Github PK
View Code? Open in Web Editor NEWDiscover dark data in videos with IBM Watson and IBM Cloud Functions
License: Other
Discover dark data in videos with IBM Watson and IBM Cloud Functions
License: Other
In the old UI design we had counters at the top showing the total number of videos/images vs. the number of videos/images to be processed. The new UI design is missing these information showing Dark Vision at work.
Take this image http://www.socialdemokraterna.se/upload/Stefan_Lofven/Bilder/Stefan%20Lofven.jpg
It is a 4MB file, 5616 x 3744. ImageMagick conversion fails - seems like it requires more memory than available to the action.
Activation: analysis (ccc)
[
"2017-04-03T08:35:50.39339173Z stdout: [ bbb ] Processing image.jpg from document",
"2017-04-03T08:35:51.875478614Z stdout: [ bbb ] KO ( 1.479 s) { Error: Command failed:",
"2017-04-03T08:35:51.875511335Z stdout: at ChildProcess.onExit (/nodejsAction/node_modules/gm/lib/command.js:301:17)",
"2017-04-03T08:35:51.875520462Z stdout: at emitTwo (events.js:106:13)",
"2017-04-03T08:35:51.875526287Z stdout: at ChildProcess.emit (events.js:191:7)",
"2017-04-03T08:35:51.875532128Z stdout: at maybeClose (internal/child_process.js:877:16)",
"2017-04-03T08:35:51.87553974Z stdout: at Process.ChildProcess._handle.onexit (internal/child_process.js:226:5) code: null, signal: 'SIGKILL' }"
]
cf create-service natural-language-understanding free <service-name>
for some reason creating trigger fails with cryptic error:
macs-mbp:processing aslom$ wsk trigger create vision-cloudant-trigger --feed vision-cloudant/changes -p dbname cloudant-a1 -p includeDoc true
error: failed to create trigger feed vision-cloudant-trigger
{
"activationId": "75e2430d583741768406f1f558653cbc",
"annotations": [],
"end": 1468607989929,
"logs": [],
"name": "changes",
"namespace": "[email protected]",
"publish": false,
"response": {
"result": {
"error": null
},
"status": "application error",
"success": false
},
"start": 1468607986701,
"subject": "[email protected]",
"version": "0.0.52"
}
no matter what parameters are passed or not ...
macs-mbp:processing aslom$ wsk trigger create vision-cloudant-trigger --feed vision-cloudant/changes
error: failed to create trigger feed vision-cloudant-trigger
{
"activationId": "a5719acbaef64a83802e92b7e6d0ef33",
"annotations": [],
"end": 1468605016622,
"logs": [],
"name": "changes",
"namespace": "[email protected]",
"publish": false,
"response": {
"result": {
"error": null
},
"status": "application error",
"success": false
},
"start": 1468605014490,
"subject": "[email protected]",
"version": "0.0.52"
}
and here is full script output:
macs-mbp:processing aslom$ ./deploy-darkvision.sh --install
Current namespace is [email protected]_dev.
Creating vision package
ok: created package vision
Adding service credentials as parameter
ok: updated package vision
Binding cloudant
ok: created binding vision-cloudant
Creating trigger
error: failed to create trigger feed vision-cloudant-trigger
{
"activationId": "4447d0dc800041b5bb5b50d6f83f5131",
"annotations": [],
"end": 1468604389260,
"logs": [],
"name": "changes",
"namespace": "[email protected]",
"publish": false,
"response": {
"result": {
"error": null
},
"status": "application error",
"success": false
},
"start": 1468604388527,
"subject": "[email protected]",
"version": "0.0.52"
}
Creating actions
ok: created action extractor
ok: created action analysis
Creating change listener
ok: created action vision-cloudant-changelistener
Enabling change listener
error: [email protected]_dev/vision-cloudant-trigger does not exist (code 87972)
macs-mbp:processing aslom$
Similar to what has been done there:
IBM-Cloud/openwhisk-visionapp#2
Hello!
It appears that the repository
property is not defined in package.json
. Please add the following
"repository": {
"type": "git",
"url": "https://github.com/IBM-Bluemix/openwhisk-darkvisionapp/"
},
to identify your application and require "cf-deployment-tracker-client": "^0.x"
to use the latest node Deployment Tracker client. Thank you!
The API "/api/videos/:id" in web/app.js performs the filtering of keywords/tags/faces to build the video summary. It ends up providing a static view for a video as the thresholds and minimum count are configured in app.js.
Instead we could return the full data to the client and have the threshold configurable in the client. This way one could play with knob-like controls to show more/less data.
Once most of the UI changes has been done.
How can the Dark Vision web app can be improved to show the new insights coming from the audio stream?
Capabilities of Dark Vision V1 web user interface:
Counterpart to #37 but to cache the calls made to Cloudant (retrieving videos, images and summary) as they can get quite consuming. However how to invalidate the cache when there is a new upload or when videos, images are being processed?
The implementation in the "audio" branch calls the STT synchronously, thus the OpenWhisk action waits doing nothing.
Instead we could use the asynchronous calls: https://www.ibm.com/watson/developercloud/speech-to-text/api/v1/?curl#async_methods
However this means registering a callback to be called by Watson when the STT process is completed.
Instead of storing the attachments into Cloudant, put them in Object Storage and serve images/videos directly from Object Storage.
I see that we have to upload a mp4 file.
But it will really nice if we could put a youtube link and openwhisk do the job from youtube.
The styling and behavior of the Filter frames by tag is slightly wonky.
Lets start by making the copy Filter frames by tag font-style: italic;
and color: #B8C1C1
.
Lets also extend the red timeline to the bottom of the screen when filtering.
If this ends up looking weird of clumsy we can work on a different solution.
32px
x 32px
.40px
.Currently the Dark Vision app lacks hover-states for most link items.
Here's a first-pass at some initial hover-states. We may need to tweak some of them as we go.
It would also be nice to play with some transition
easing in areas we haven't yet. Lets try adding transition: all 0.25s ease;
to 1,2, 3 & 4.
By doing so, we could have two triggers, one listening for changes of the video, one for images. This could remove the changelistener and call the actions directly. The actions would decide what to do with the event.
When the video frames are stored in Cloudant, analysis.js
retrieves the image file without doing any retry.
If many frames are being analyzed, Dark Vision may hit Cloudant rate limit and fail to retrieve the image. It usually results in "not a jpeg file" error when analyzing the image.
Seems to come from an issue registering the Speech to Text callback in the toolchain.
The call to register fails with an error 406. It could be due to a concurrency issue where the speechtotext web action is not yet exposed by OpenWhisk when we try to call it during registration.
A re-run of the pipeline usually solves the issue.
{"code":400,"code_description":"Bad Request","error":"unable to register callback url 'https://openwhisk.ng.bluemix.net/api/v1/experimental/web/user_space/vision/speechtotext.http', callback server responded with status code 406"}
instead load the document from the change listener and look at it.
The Node.js part of the action can be made simpler. The current code was based on the first OpenWhisk Docker SDK. We could merge the client and server apps into one. See https://github.com/ibmets/openwhisk-jq for a clean implementation.
The follow up to #11. Once we have the transcript we can pass it through Tone Analyzer and display the results in the web/iOS app
The audio from the video is another dark data to process. First step is to get the text.
The issue seems to be in Speech to Text. It fails if the callback URL has a space in its name:
{"code":400,"code_description":"Bad Request","error":"unable to register callback url 'https://openwhisk.ng.bluemix.net/api/v1/experimental/web/[email protected]_Game of Bluemix/vision/speechtotext.http', callback server responded with status code 505"}
Another related issue fixed by PR #65 was with Cloud Functions:
___ __ ___ _ _
/ _ \ _ __ ___ _ _\ \ / / |__ (_)___| | __
| | | | '_ \ / _ \ '_ \ \ /\ / /| '_ \| / __| |/ /
| |_| | |_) | __/ | | \ V V / | | | | \__ \ <
\___/| .__/ \___|_| |_|\_/\_/ |_| |_|_|___/_|\_\
|_|
Retrieving OpenWhisk authorization key...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 1757 0 0 100 1757 0 8514 --:--:-- --:--:-- --:--:-- 8487
100 1757 0 0 100 1757 0 869 0:00:02 0:00:02 --:--:-- 870
100 1757 0 0 100 1757 0 581 0:00:03 0:00:03 --:--:-- 581
100 4067 0 2310 100 1757 737 561 0:00:03 0:00:03 --:--:-- 738
error: syntax error, unexpected $end, expecting QQSTRING_TEXT or QQSTRING_INTERP_START or QQSTRING_END
.namespaces[] | select(.name == "[email protected]_Game
^^^^^^^^^^^^^^^^^^^^^
1 compile error
error: syntax error, unexpected $end, expecting QQSTRING_TEXT or QQSTRING_INTERP_START or QQSTRING_END
.namespaces[] | select(.name == "[email protected]_Game
^^^^^^^^^^^^^^^^^^^^^
1 compile error
Speech to Text OpenWhisk action is accessible at https://openwhisk.ng.bluemix.net/api/v1/experimental/web/[email protected]_Game of Bluemix/vision/speechtotext.http
_ _ _
https://www.ibm.com/blogs/bluemix/2016/12/openwhisk-api-keys-update/
before merging #24
experimental/web has graduated.
Dark Vision should move to the new API.
https://medium.com/openwhisk/whats-new-web-actions-e0ffeef1bea3
Note that if you remove "experimental" today from the STT_CALLBACK_URL, it will lead to an error like:
{
"error": "Response is not valid 'message/http'.",
"code": 3104139
}
because the code of the speechtotext action needs to be updated.
there should be better logging to track what is going during deployment and execution
for deploying print more what is happening so more help when it fails
for operations print more in logs - for example currently my demo deployment processed two videos but stopped after that with no error i could find ...
The goal of the form will be to provide a field to set the video title, but also to select the video language.
The number of extracted frames is currently set in extract.js with an option to change it by rebuilding the Docker image.
Instead this number could be a property of the video and configured from the upload dialog.
Deleting a video should delete all the related images, audio and attached files (if not stored in Cloudant).
Per the documentation of the NLU service (https://www.ibm.com/watson/developercloud/doc/natural-language-understanding/#supported-languages) and the documentation of speech-to-text (https://www.ibm.com/watson/developercloud/doc/speech-to-text/input.html#models) there is support for multiple languages.
Is there any additional restriction to support them in OpenWhisk?
To ease the deployment of the app, add a toolchain. The toolchain will create the services, the webapp and deploy the OpenWhisk actions/triggers.
As by default we use the rate limited plan of Cloudant, we should reduce the load on Cloudant when browsing the web app.
Caching comes to mind.
We could be caching images loaded from Cloudant by writing them on the local disk (we don't mind if the disk gets lost on restart) and having them served directly by node. Once uploaded images do not change - only the video thumbnail may change if the analysis is retriggered.
Need to make sure the Dark Vision code works with the Lite plan and correctly retries if it hits the rate limits.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.