ellavator / ellavator Goto Github PK

This is EllaVator project to build Ella the talking eleVator as part of a Saarland University software project class.

License: GNU General Public License v3.0

Java 94.75% Groovy 5.25%

ellavator's Introduction

EllaVator

How to build

Java 8 is required. If you have multiple Java installations, make sure you export JAVA_HOME to point to the correct version.

Get the source code

This project contains git submodules. Therefore, it should be cloned recursively, i.e.,

$ git clone --recursive https://github.com/EllaVator/EllaVator.git

If this is not done at clone time, it can of course be done later, by running

$ git submodule update --init

Build with Gradle

Simply run

$ ./gradlew shadowJar

to build the standalone application, which will be put into build/libs. The standalone application is a jar file, which can be run with

$ java -jar build/libs/EllaVator.jar

Run with Gradle

Instead of building a standalone jar file, it is also possible to just run the application directly, like this:

$ ./gradlew :run

known issues

After running the command $ ./gradlew :run, if it ouput error like:

repository_ellaVator/src/main/java/ModuleElla.java:17: error: package opendial does not exist
import opendial.DialogueState;
               ^
repository_ellaVator/src/main/java/ModuleElla.java:18: error: package opendial does not exist
import opendial.DialogueSystem;
               ^
repository_ellaVator/src/main/java/ModuleElla.java:19: error: package opendial.modules does not exist
import opendial.modules.Module;
                       ^
repository_ellaVator/src/main/java/ModuleElla.java:31: error: cannot find symbol
public class ModuleElla implements Module {
                                   ^
  symbol: class Module
repository_ellaVator/src/main/java/ModuleElla.java:48: error: cannot find symbol
	public ModuleElla(DialogueSystem system) {
	                  ^

May be one of the submodule is not installed correctly. Try to run the following command:

$ git submodule update --init

ellavator's People

Contributors

Stargazers

Watchers

Forkers

sirifarif psibre asayeed abentkamp hoffmannaj gitter-badger pat-carroll laurarf naszka tastyminerals stevenlol ravehe zhangxt

ellavator's Issues

Cannot run headlessly

Trying to run in a headless environment causes a headless exception:

$ ./gradlew run
Starting a new Gradle Daemon for this build (subsequent builds will be faster).
:sphinx4:sphinx4-data:compileJava UP-TO-DATE
:sphinx4:sphinx4-data:compileGroovy UP-TO-DATE
:sphinx4:sphinx4-data:processResources UP-TO-DATE
:sphinx4:sphinx4-data:classes UP-TO-DATE
:sphinx4:sphinx4-data:jar UP-TO-DATE
:sphinx4:sphinx4-core:compileJava UP-TO-DATE
:sphinx4:sphinx4-core:compileGroovy UP-TO-DATE
:sphinx4:sphinx4-core:processResources UP-TO-DATE
:sphinx4:sphinx4-core:classes UP-TO-DATE
:sphinx4:sphinx4-core:jar UP-TO-DATE
:compileJava UP-TO-DATE
:compileGroovy UP-TO-DATE
:processResources UP-TO-DATE
:classes UP-TO-DATE
:run
java.lang.UnsatisfiedLinkError: no rxtxSerial in java.library.path thrown while loading gnu.io.RXTXCommDriver
no rxtxSerial in java.library.path
[OpenDial] INFO: Module ModuleElla successfully attached 
[OpenDial] INFO: Module AudioModule successfully attached 
[OpenDial] INFO: Acoustic model: resource:/edu/cmu/sphinx/models/en-us/en-us 
[OpenDial] INFO: Dictionary: resource:/grammar_en/ella_en.dict 
[OpenDial] INFO: Recognition grammar: resource:/grammar_en/compact_gram.gram 
[OpenDial] WARNING: could not start module opendial.gui.GUIFrame: java.awt.HeadlessException 
java.awt.HeadlessException
    at java.awt.GraphicsEnvironment.checkHeadless(GraphicsEnvironment.java:204)
    at java.awt.Window.<init>(Window.java:536)
    at java.awt.Frame.<init>(Frame.java:420)
    at java.awt.Frame.<init>(Frame.java:385)
    at javax.swing.JFrame.<init>(JFrame.java:189)
    at opendial.gui.GUIFrame.start(GUIFrame.java:111)
    at opendial.DialogueSystem.startSystem(DialogueSystem.java:149)
    at OpenDialDemo.main(OpenDialDemo.java:44)
[OpenDial] WARNING: could not start module ModuleElla: java.awt.HeadlessException 
java.awt.HeadlessException
    at java.awt.GraphicsEnvironment.checkHeadless(GraphicsEnvironment.java:204)
    at java.awt.Window.<init>(Window.java:536)
    at java.awt.Frame.<init>(Frame.java:420)
    at java.awt.Frame.<init>(Frame.java:385)
    at javax.swing.JFrame.<init>(JFrame.java:189)
    at ModuleElla.start(ModuleElla.java:56)
    at opendial.DialogueSystem.startSystem(DialogueSystem.java:149)
    at OpenDialDemo.main(OpenDialDemo.java:44)
> Building 93% > :run

It's not clear if the system itself is functional at that point (and only the GUI monitor is failing), or if everything hangs in a broken state...

Serial and milk?

This is the issue for discussing the actual real true serial port interface stuff, from a hardware and a drivers perspective. We are using this:

http://www.lctech-inc.com/Hardware/Detail.aspx?id=67f4ee1a-7412-4d12-87f6-7d00d52566ca

The standard pinnage for connecting them is described here:

http://de.mathworks.com/help/supportpkg/raspberrypiio/ug/use-the-serial-interface-on-raspberry-pi-hardware.html

The issue of power levels is somewhat complicated. The way serial ports work is this:

http://lavalink.com/2012/04/rs-232-voltages/

The GPIO of the RPi runs at 3.3V, which is at the lowest range for serial port detection. 5V is common for PCs, although in theory 3.3V should be sufficient. We don't know how much the elevator will really require. The serial port board we have can take 5V and 3.3V.

There is a GPIO pin for 5V also. The use of it is a matter of considerable online discussion. We shall revisit the use of it in case the elevator cannot detect 3.3V.

Virtualize raspberry pi 2

Use qemu...

Multi input

Hi all,

I was thinking about whether we would like to allow multiple inputs.

At the moment the system is only able to handle one task.
One user enters, passes a command to the elevator and the order is executed.

But how would things look like if two people were to enter with each different destinations?
Would we like to have the elevator waiting for further speech input after each command?
Or do we just wand to ignore this possibility?

In case we'd like to also handle input from more than one person, we should also have this in mind, when thinking of a solution for switching languages, as both an English speaking person and a German speaking person might enter at the same time.

some assembly required

Most of the parts have arrived, so it's time for the issue for setting up the Pi. I will bring (note to self!!!) a tool set to Wednesday's class with the parts, and a spare monitor, mouse, and keyboard. I will also load a Raspbian image onto one of the SD cards. As I predicted, the serial port will probably come next week or later, but that's fine. This issue is opened just to keep track of anything we discover we didn't realize...

Enable sphinx recognition in our app

We currently run the OpenDialDemo from gradle (with the run task), but there is no integration with sphinx, so no ASR support.
We need to add ASR support, similar to the OpenDial Sphinx ASR plugin, and run it with our dictionary and grammar.

Serial Comm diagnostics

We should diagnose the serial communications between the elevator and any PC by creating a lightweight utility (in python or whatever) that can be run on a random laptop/PC with a serial port. This should be able to send the control and heartbeat commands to the elevator, and record the responses.

After we determine how to communicate with the elevator in this way, we can then invert the utility to emulate the elevator, and connect our Pi2 to it, testing that the EllaVator apps works in the same way and elicits the same responses. This can also be tested programmatically.

Present dialog frameworks

Resource cannot be found: resources//settings.xml

Currently when executing gradle run, this error is thrown:

    opendial.utils.XMLUtils extractMapping
    WARNUNG: error extracting the settings: java.lang.RuntimeException: Resource cannot be found: resources//settings.xml

There is a workaround for this problem, namely copying opendial/resources/* to resources/*.
But this is not a very nice thing to do, I think.

Here is why the error is thrown: the file path of settings.xml is defined in opendial/src/opendial/Settings.java:

    public static final String SETTINGS_FILE = "resources//settings.xml";

This will be hard to change without modifying the code, because it's static and final.

Then, in the Settings constructor we have a call to XMLUtils.extractMapping(SETTINGS_FILE) and finally the class opendial.utils.XMLUtils looks for this filename:

"/" + filename.replace("//", "/")

... which will transform resources//settings.xml to /resources/settings.xml. This works for the opendial project because the resources folder is located in the main directory. But in our case, the resources are located in opendial/resources as we embedded the opendial project in a subfolder.

Anybody any ideas how to solve this elegantly?

Finishing up the report

Hi guys. We decided to review the status of our report and see what else can be added to the table.
Below is a list of items that everyone is invited to contribute.

Testing -- Alex takes this.
Raspberri Pi (replace pi4j section), Serial cable connection setup (some undocumented stuff), Development tools (some info about serial port tools) - Pavel takes this.

ASR (Arif's presentation) -- ???
Acoustic model (ask grandmaster Ingmar) -- ???
Language model -- ???
TTS (add some general info) -- ???

Make your pick!

Prompt list

We still need a German prompt list, and we should let our imagination run wild with the English one. We want to have recordings of everything we want the system to understand, not limited by the current state.

Ellavator cannot find Domain_Opendial_Ella.xml file

When runnning ./gradlew :run.

[OpenDial] INFO: Module ModuleElla successfully attached
[OpenDial] INFO: Module AudioModule successfully attached
[OpenDial] INFO: Acoustic model: resource:/edu/cmu/sphinx/models/en-us/en-us
[OpenDial] INFO: Dictionary: resource:/grammar_en/ella_en.dict
[OpenDial] INFO: Recognition grammar: resource:/grammar_en/compact_gram.gram
[OpenDial] SEVERE: cannot read xml file: java.io.FileNotFoundException: resources/Domain_Opendial_Ella.xml (No such file or directory)

The file exists in the folder.

Running EllaVator with "detect voice activity" on by default

When running EllaVator with ./gradlew run, OpenDial starts the interaction UI without "detect voice activity" enabled.
We need to fix this so that when EllaVator is booted on startup it is ready to accept speaker's input.

InproTK prototype

no rxtxSerial in java.library.path thrown while loading gnu.io.RXTXCommDriver

I got the following error while doing a fresh run and following the steps as in README file.

I guess it doesn't find the serial port. Any idea?

 $ ./gradlew :run
:opendial:compileJava UP-TO-DATE
:opendial:compileGroovy UP-TO-DATE
:opendial:processResources UP-TO-DATE
:opendial:classes UP-TO-DATE
:opendial:jar
:sphinx4:sphinx4-data:compileJava UP-TO-DATE
:sphinx4:sphinx4-data:compileGroovy UP-TO-DATE
:sphinx4:sphinx4-data:processResources UP-TO-DATE
:sphinx4:sphinx4-data:classes UP-TO-DATE
:sphinx4:sphinx4-data:jar UP-TO-DATE
:sphinx4:sphinx4-core:compileJava UP-TO-DATE
:sphinx4:sphinx4-core:compileGroovy UP-TO-DATE
:sphinx4:sphinx4-core:processResources UP-TO-DATE
:sphinx4:sphinx4-core:classes UP-TO-DATE
:sphinx4:sphinx4-core:jar UP-TO-DATE
:compileJava
Note: /Users/arif/workspace/elevator2015/test/EllaVator/src/main/java/SerialPortController.java uses unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
:compileGroovy UP-TO-DATE
:processResources UP-TO-DATE
:classes
:copySettingsXml UP-TO-DATE
:run
java.lang.UnsatisfiedLinkError: no rxtxSerial in java.library.path thrown while loading gnu.io.RXTXCommDriver
Exception in thread "main" java.lang.UnsatisfiedLinkError: no rxtxSerial in java.library.path
    at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1865)
    at java.lang.Runtime.loadLibrary0(Runtime.java:870)
    at java.lang.System.loadLibrary(System.java:1122)
    at gnu.io.CommPortIdentifier.<clinit>(CommPortIdentifier.java:83)
    at SerialPortController.<init>(SerialPortController.java:33)
    at ElevatorController.<init>(ElevatorController.java:11)
    at ModuleElla.<clinit>(ModuleElla.java:33)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:264)
    at opendial.Settings.fillSettings(Settings.java:243)
    at opendial.readers.XMLDomainReader.extractPartialDomain(XMLDomainReader.java:147)
    at opendial.readers.XMLDomainReader.extractDomain(XMLDomainReader.java:109)
    at opendial.readers.XMLDomainReader.extractDomain(XMLDomainReader.java:64)
    at OpenDialDemo.main(OpenDialDemo.java:14)
> Building 95% > :run

Switching between English and German input

Looking for suggestions on how to switch between English and German input.
I have some vague ideas on how to do this in the jsgf grammar file, but (for those of us how looked into it) how would this work in Sphinx? Can it listen for two languages at the same time? Can Mary be activated for to languages at the same time?

Hot word / keyphrase listening

Since we would like to ignore when people are having a normal conversation in the elevator, it would be nice to use the keyphrase listening feature of sphinx. Here is what I found in their FAQ:
http://cmusphinx.sourceforge.net/wiki/faq#qhow_to_implement_hot_word_listening
In the end it says:

This feature is not yet implemented in sphinx4 decoder.

... but it is in pocketsphinx. Does that mean we should eventually try to use pocketsphinx or is there another workaround?

Collect prompts / model dialog flow

in German and English

OpenDial prototype

German speech recognition

we want Ella to understand German, so we need acoustic models for German.

OpenDial domain adaptation

Speech I/O testing with TTS and ASR feedback loop

Create pronunciation dictionary for our limited domain

Expand JSGF into a list of all possible utterances in domain

elevator mock to test the serial port signals coming from our application

Maybe it is not necessary to write our own mock program for the elevator. There is a linux program called minicom (http://www.cyberciti.biz/tips/connect-soekris-single-board-computer-using-minicom.html) that logs all signals coming from the serial port. We could run this program on an old computer with serial port, connect it to the rasberry pi, and see the output.

Project report

Hi everybody,

Let's try authorea for our project report:
https://www.authorea.com/users/50452/articles/58685/_show_article

I created this document and invited everybody I could get email addresses from. I will add everybody who sends me their email address.

The structure in the document is just a proposal. If you have better ideas go ahead and change everything or discuss it in the comments section...

yes, it's the shopping list issue

@psibre and I spent a little more time afterwards attempting to shop for parts. This is slightly harder than you might think, because we have some special accessory requirements. We are going to order most of @bryandeng's suggestions, which are the basic items:

Raspberry Pi 2 Model B:
http://de.rs-online.com/web/p/products/8326274/

Appropriate housing:
http://de.rs-online.com/web/p/products/8193646/

Power supply:
http://de.rs-online.com/web/p/products/8226373/

2 of these SD cards:
http://de.rs-online.com/web/p/products/7504795/

BUT, what we have left are the following:

RS232 level shifter (because the GPIO port is not run at enough power to behave like an old-school serial port)
Actual RS232 serial port (does the elevator require male or female? Need to check)
- We need to check whether or not we need a special case to accommodate the level shifter, etc, or an extension outside the case. It would look kind of silly to have another board hanging out of the case, but we may have no other option. This is selbstgebastelt electronics for you!
USB audio with appropriate audio input. The Adafruit website has a recommended audio card, but they seem to suggest that they haven't tested the input mic. However, there is a page for this (http://elinux.org/RPi_VerifiedPeripherals#USB_Sound_Cards) -- we may need to order it separately, since the RS web site doesn't seem to have these, or at least we couldn't find them.
Cheap HDMI cable. Does anyone know where to get it for cheaper? Anyway 6EUR is not a terrible sacrifice. We might be able to get this from Systems Group, since we only need it temporarily for testing.