just-ai / jaicf-kotlin Goto Github PK

CAILA NLU has intents tree, intent may be embedded into some context, for example, following intents
/FAQ/Weather
/FAQ/DeliveryHours
are in subtree /FAQ.
This is a useful feature in big projects with lots of intents and it makes users easier to navigate across intents.

This issue suggests:

1. Adding intent activation by full path (not only by name):

activators {
  intent("FAQ/Weather") // instead of current only-name activation
}

2. Adding intent activation by context path:

activators {
  intents("FAQ") // will activate all nested intents in FAQ
}

Develop barge-in to make available routes from state by intents, or play some another replica

Bolt-SDK development ramped up, there are tons of new releases with bugfixes and features.

This issue suggest bumping bolt-sdk version to some new version compatible with current classpath (compatible with current kotlin/ktor/serialization versions).

Caila NLU Provider (not sure about other nlu providers) support client entities. This issue suggests creating proper generic API for framework users to use client entities, allowing basic CRUD operations to manipulate entities and entity records.

What is client entity?
Client entity is some entity which synonyms and values are stored in NLU provider by clientId.

This issue suggest adding CAILA client entities support (http client + model + dsl methods).

Version 0.10.0 brought type-specific actions, e.g.:

action(telegram) { // safe-casts request and reactions type to TelegramBotRequest, TelegramReactions
    reactions.document("some-url") // use smartcasted method
}

Version 0.13.0 brought improved DSL, so we can set generic type for whole scenario.

val bot = Scenario(telegram) {
    state("state") {
        action {
            reactions.say("You said: ${request.input}", "button1", "button2")
        }
    }
}

This works great, but the only thing missing is type-casts in hooks. This issue suggests extend generic types and smart-casts to hooks. Expected DSL, generic types in handle:

handle<BeforeActionHook>(telegram) { // declare token same as in action
    reactions.document("...") // use smart-casted reactions/request/activator same as in action
}

, and generic types propagation from Scenario:

val bot = Scenario(telegram) {
    handle<BeforeActionHook> { // handle uses propagated request/reactions type
        reactions.document("...")
    }
}

Let’s think about making CatchAllActivator, BaseEventActivator and RegexActivator built into the framework, enabling the user skip these activators from the list of BotEngine instance.

NLU Providers can contain default answer for intent, which can be extracted from ActivatonContext in scenario and responded. Example usage:

state("handler") {
    activators {
        intent("HowAreYou")
    }
    action {
        activator.caila?.topIntent?.answer?.let {
            reactions.say(it)
        }
    }
}

It looks ok when there's not so many intents. But when intents number increases, usability decreases. User should list all project intents in single state and he cannot separate dialogue logic from scenario.

state("handler") {
    activators {
        intent("HowAreYou")
        intent("WhatCanYouDo")
        intent("Family")
        intent("WhatAreYouDoing")
        ...
    }
    action {
        activator.caila?.topIntent?.answer?.let {
            reactions.say(it)
        }
    }
}

This issue suggest adding something like:

state("handler") {
    activators {
        event(Activations.AnyActivation)
    }
    action {
        activator.caila?.topIntent?.answer?.let {
            reactions.say(it)
        }
    }
}

After #139 was merged JAICF obtained ability to merge sub-scenarios inside parent scenario.

Scenario example:

val HelperScenario = Scenario {
    state("ask4name") {
        activators {
            catchAll()
            intent("name")
        }
    ...
}

val HelloWorldScenario = Scenario {
    append(context = "helper", HelperScenario)
}

Here we can see that parent scenario provides context for subscenario. Thus means subscenario can never know absolute state path and should rely only on relative state paths (e.g. do reactions.go("../../someState") instead of reactions.go("/someState")).
This opens a major issue for many StateNotFound runtime exceptions in bot.

What can be done:
Create new symbol in statepath resolution - ~, which will hold a root path of current scenario object. This will allow to use absolute paths from scenario root (from ~) and will give more flexibility for appended sub-scenarios.
So the transition will look like reactions.go(~/someState)

CAILA NLU Engine can support client entities, stored by clientId for this client, we'd better to support it in JAICF.

Now it's possible to write Scenario like this:

state("") {
    activators { 
        action { 
            globalActivators { 
                
            }
        }
    }
}

Possible solution: use @DslMarker (doc).
But it requires to rework ScenarioBuilder architecture.

Why?
To track JAICF applications deployed in JAICP cloud

What?

• JAICF applications deployed in JAICP cloud should send logs to JAICP graylog
• Log messages must contain all fields required to show them in "Logs" page inside JAICP properly (accountId, projectId, botId, userId)

Definition of Done

• JAICF users can view application logs in Log page inside JAICP

Let's assume the following scenario:

state("main") {
	state("first") {
		activators {
			regex("first")
        }

		action {
			reactions.go("../second")
		}
	}

	state("second") {
	
	}
}

And the following test case:

withCurrentContext("/main")
query("first") goesToState "/main/first" endsWithState "/main/second"

Expected behavior:
The test succeeds

Actual behavior:
The test fails because of goesToState

Now it's impossible to use reactions.buttons() with Telegram. Because Telegram Telegram allows only to send keyboard attached to some message.

The idea is to use Telegram's editMessageReplyMarkup appending inline buttons to the previous message sent to the channel.

Problem:
In Reaction.kt we have method:

open fun buttons(vararg buttons: String) {}

This method adds buttons to response, but telegram does not support adding buttons by separate reaction, so when user reacts with buttons in telegram channel, nothing happens.

Solution:
We can modify buttons in TelegramReaction to edit last response and add buttons to last response. Thus allowing using following code in TelegramChannel like in synchronous channels:

action {
  reactions.say("What number do you pick?")
  reactions.buttons("1", "2", "3")
}

https://developer.amazon.com/en-US/docs/alexa/conversations/about-alexa-conversations.html

Multiple slashes in reaction paths are interpreted as multiple states with empty names, but they are allowed and equivalent to a single slash in Unix systems. There are also problems with slashes in the names of states.
I think we should parse as in Unix systems and forbid slashes in the names of states (except for one leading slash in top-level states)

context.clientId returns chat id for telegram

Example for bug reproduction.
Difference between IDs can be seen only in group chats, because in a private chats clientId == chatId

action {
                var userId: Long? = null
                request.telegram?.run {
                    userId = message.from?.id
                }
                if (userId != null) {
                    reactions.say(userId.toString())
                    reactions.say(context.clientId)
                }
}

Now changes in the scenario model will almost certainly break the next bot execution if persistent BotContextManager is used. And the only option scenario author has is to clear storage manually after every change.

This issue suggests implementing some mechanisms that will help the user in migrations.

Google Assistant (ActionsReactions) has it own rules for composing Response card, this leads to issues when we want to create a Scenario that will work mostly the same in all channels using default reactions.

Example:

action {
  reactions.image("url")
  reactions.say("text")
}

This will work in all channels but google.

I think we can and should make ActionsReactions work the same way as other channels.

Need to add MULTILINE flag to the RegexActivator rule matcher

Hi

When trying to integrate RASA NLU, an exception is thrown when the NLU training set contains entities that are resolved.

(From the console):
jaicf.activator.rasa.api.RasaApi - Cannot parse RasaParseMessageRequest(text=....., messageId=30eed22e-e5e8-4960-a979-6485d5295f45)
kotlinx.serialization.MissingFieldException: Field 'confidence' is required, but it was missing

From RASA output (using rasa train shell), we see a structure like this:

{
      "entity": "frequency",
      "start": 9,
      "end": 18,
      "confidence_entity": 0.9578643441200256,
      "value": "7",
      "extractor": "DIETClassifier",
      "processors": [
        "EntitySynonymMapper"
      ]
    }

But the mapping in RasaParseMessageResponse.kt contains:

@Serializable
data class Entity(
    val start: Int,
    val end: Int,
    val confidence: Float,
    val value: String,
    val entity: String
)

Maybe the confidence field should be renamed 'confidence_entity' to map correctly the output?

For now, the user must ensure by himself that there is no code after reactions.go, otherwise, this code will be executed, moreover, it will be executed before toState action block. Although it breaks reactions.go semantic and documentation that states the following: "Changes the state of scenario and executes it's action block immediately".
For example, such a code:

action {
	reactions.go("somewhere")
	reactions.say("Shouldn't get to this")
}

will produce the output: Shouldn't get to this.

I think, that it's not the desired behaviour and we should think about, for example, making reactions.go and other go-like methods return kotlin.Nothing. It will make user and compiler sure, that current action block execution interrupts after reactions.go

now there's no logging while using channels that are working by embeddedServer
example channels: Alice, Aimybox

This issue suggest a feature of filling slots from client context.

Let's say that we have an Intent, which has required slots like name and phoneNumber. These slots are required and must be filled to proceed dialog, otherwise bot won't be able to perform some action.

Once client answered to it, we can store it in client context, further eliminating need to ask it again. But right now this information will be stored only in our bot context, activator will have no knowledge of it and won't have any use of it. So, when user will fall into this same intent, he should be asked to fill name and phoneNumber again (this behaviour may vary from activator to activator, but still).
And this makes slot filling unusable for these kind of common scenarios.

As I think, resolving this issue must bring some interfaces/methods for framework users to pre-fill some slots (either via client context, or directly form ActionContext). Or just find way to re-use slots.

Is it possible to have a parent state with an activator which, when activated, starts evaluating the inner states straightaway without the user giving another utterance?

I've tried to visualise what I mean by conditional "gateways" below, so as mentioned in the initial question, when the parent activator is satisfied, JAICF doesn't reply with anything and instantly evaluates the inner states to compare the given intent with the activators.

Parent state with conditional activator which doesn't have an action (checking for whether a context variable is not null)
|
|-> Inner State with an intent activator and an action
|
|-> Inner State with intent activator and an action

The reason I am asking this question is that I've got experience in using Watson Assistant. This framework uses nodes (can be considered states) to group inner nodes together. This parent node can have a condition (same as an activator) which when satisfied, allows the chatbot access to instantly evaluate the child nodes (acting as a gateway essentially).

Any help/info is much appreciated!

Looks like it's common use-case when the user have to create dialogue scenario for only a single channel. In this case accessing a channel-specific interfaces like reactions.alexa? or request.alexa? could be annoying.
As I think we could provide some channel-specific builders to make it possible to automatically cast BotContext's interfaces to channel-specific and activator-specific implementations.

https://aws.amazon.com/lex/

Having multiple intents with different confidences, we have to select state for the most relevant intent, where relevance is defined by confidence + by context range (how close this new target state to current context).
This issue suggests that we should create that ranking function and apply it during intent activation.

DialogContext#transitions, that is used for strict (button?) transitions is never populated with new transitions and always remains empty.

ActionErrorHook is a hook invoked on action error. But use case for this hook looks a bit unclear.

Having error, user might want to handle this error, but hook does not allow error handling. It only allows to react in some way to error, while further processing in BotEngine will be aborted.

@morfeusys maybe we should create ActionErrorHandler instead of ActionErrorHook?

Hi!

JAICP has function Pushgate for outgoing mailing from chat-bot.
https://help.just-ai.com/docs/ru/JS_API/built_in_services/pushgate/pushgate

I didn't find this function in JAICF. How to make such the function?

HttpBotRequest uses read-once streams which are not so safe for public api. An error can easily happen if someone would want to log the content before it is passed next to BotChannel.

This issue suggests wrapping HttpBotRequest streams content to read it multiple times.

The idea is about to create a separate DSL for form-like dialogue scenarios. With this DSL the user could describe a simplified step-by-step dialogues that have an entry point and a resulting action. This action receives a resulting context filled by the previous steps.

Similar feature could be found in the Rasa's forms.

As far as I understand, Matcher#group(id) will return null, if a group hadn't matched any input.
And as we put groups in non-nullable RegexActicvatorContext#groups list, IllegalStateException occurres.

Now BotRequest doesn't provide an access to the initial raw request that was processed by the channel.

Всем привет! Возникла очень странная проблема по JAICF, буду рад любой помощи.
Мы используем стандартный шаблон проекта от jast AI https://github.com/just-ai/jaicf-template, он содержит в себе сценарий и проперти на эндпоинт mongoDB.

Суть проблемы в использовании переходов от стейта к стейту с помощью команд вида: reactions.buttons("текст" toState "state_name").
Проблему мы локализовали, и она связана именно с "toState".

Непонятный момент заключается в том, что проблемы появляются только после подключения базы данных (пробовали только дефолтную mongo). Это происходит из-за того, что после ее подключения на всех юзеров начинает вестись контекст (id, current state, transition history и т.п.). Как раз transition history содержит в себе список всех стейтов по порядку, как по ним переходил пользователь.

Проблема проявляется в том, что при использовании buttons через toState некоторые (не все почему-то) переходы не появляются в transition history, но их бизнес логика отрабатывает, выводятся все сообщения в диалог и т.п., однако current state не меняется в монге. Из-за этого возникает ситуация, когда пользователь видит уже следующий стейт пользователь жмет очередную кнопку чтобы пройти еще дальше, но в монге висит все равно предыдущий стейт, и она не может найти доступный переход (т.к. в предыдущем стейте это не возможно) и jaicf выдает fallback.

При отключении монги все работает отлично и никаких проблем не возникает ни в одном стейте.
Одно из решений сейчас - это убирать buttons toState и заменять их на обычные buttons, а в стейты класть регекс активаторы - это работает, однако по нашей бизнес логике необходимо использовать именно buttons to State.

Кто-то сталкивался с чем-то подобным? Возможно найти какие-то обходные пути? Есть ли какая-то альтернатива как можно перенаправить пользователя, чтобы по нажатии на какую-либо конкретную кнопку он переходил к какому-либо конкретному стейту?

Попытался заснять на видео:
https://www.youtube.com/watch?v=9aKExYql7Uo

Hi

Can you help me please.
I was trying to connect to mongodb from test.
My code is as an example https://github.com/just-ai/jaicf-kotlin/blob/master/managers/mongo/src/test/kotlin/com/justai/jaicf/context/manager/test/MongoBotContextManagerTest.kt
My test is dfferent only by connection string. In my case I use heroku:

mongodb://$login:$[email protected]:55762/heroku_9w2h7mvv

I get error wen rut test:
[Test worker] INFO org.mongodb.driver.cluster - Cluster description not yet available. Waiting for 30000 ms before timing out [cluster-ClusterId{value='5e9d4c755f914205c037beff', description='null'}-ds055762.mlab.com:55762] INFO org.mongodb.driver.cluster - Exception in monitor thread while connecting to server ds055762.mlab.com:55762 com.mongodb.MongoSocketOpenException: Exception opening socket at com.mongodb.connection.SocketStream.open(SocketStream.java:62) at com.mongodb.connection.InternalStreamConnection.open(InternalStreamConnection.java:126) at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:114) at java.base/java.lang.Thread.run(Thread.java:832) Caused by: java.net.SocketTimeoutException: Connect timed out

It's impossible to create non-JAICP tg channel and start processing updates

managers/mongo/src/main/kotlin/com/justai/jaicf/context/manager/mongo/MongoBotContextManager.kt

We have several problems when using MongoBotContextManager.

(1) it is not ready to support multiple simultaneous sessions with a one client -> sessions storage has to be a dictionary with the session id as a key
(2) ideally, i want to be able to define a strategy for BotContext._id , if BotRequest.clientid is not defined (in case of anonymous clients)

Here are SSML helper functions that should be extended to covers all SSML features supported by Alexa and Google Actions.

I hawe simple model

data class WordModel(
val word: String,
val length: Int,
val imageUrl: String
)

Whem I add item to MongoBD automaticly added field _id even if I remove this field

{
"_id": {
"$oid": "5ead47f27c213e5d2fa60237"
},
"word": "some_word",
"length": 9,
"imageUrl": "some_url"
}

And then when I try to get data I have an exeption
Unrecognized field "$oid" (class org.bson.types.ObjectId), not marked as ignorable (4 known properties: "machineIdentifier", "counter", "timestamp", "processIdentifier"]) at [Source: (String)"{ "_id" : { "$oid" : "5ea9c2837c213e2096467134" }, "word" : "some_word", "length" : 9, "imageUrl" : "some_url" }"; line: 1, column: 23] (through reference chain: com.justai.jaicf.template.scenario.WordModel["_id"]->org.bson.types.ObjectId["$oid"])

BotContextModel have _id field with type as String https://github.com/just-ai/jaicf-kotlin/blob/master/managers/mongo/src/main/kotlin/com/justai/jaicf/context/manager/mongo/BotContextModel.kt

If I add val _id: String to my model then I have an exeption
Cannot deserialize instance of java.lang.Stringout of START_OBJECT token at [Source: (String)"{ "_id" : { "$oid" : "5ead47f27c213e5d2fa60237" }, "word" : "some_word", "length" : 9, "imageUrl" : "some_url" }"; line: 1, column: 11] (through reference chain: com.justai.jaicf.template.scenario.WordModel["_id"]) com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot deserialize instance ofjava.lang.String out of START_OBJECT token at [Source: (String)"{ "_id" : { "$oid" : "5ead47f27c213e5d2fa60237" }, "word" : "some_word", "length" : 9, "imageUrl" : "some_url" }"; line: 1, column: 11] (through reference chain: com.justai.jaicf.template.scenario.WordModel["_id"])

What am I doing wrong?

Use official Bolt SDK instead of current in Slack channel.

Using third-party wrapped channel API libraries gives us a problem, that each channel provides dependencies which may interfere with other channel's (or activator) dependencies.

For example, i've counted 3 different versions of okhttp3:

facebook -> 3.14
slack -> 4.4.0
telegram -> 3.8

And now we have either TelegramChannel fully working, or Slack, or Facebook.