Kendra context summary encoding/decoding issues. about qnabot-on-aws HOT 2 OPEN

Hi @cboin1996
Thanks for reporting this.
Based on my initial read of this, it seems that the encoding issue is already at the Kendra level, therefore I do not see this as a bug in QnA but potentially as an enhancement. However we will look into this and revert back.

from qnabot-on-aws.

Yea it does look to be at the Kendra level, but I still wonder if it could be addressed in the bot fulfillment lambda.

One potential idea I had, is to force specific encoding's within the bot fulfillment lambda.

• Add a parameter to the qnabot designer called 'DecodeKendraResults'
• Decode the search results in bot fulfillment prior to passing it off to the client interface or passing to LLM if LLM plugin is enabled.

ex:

• EncodeKendraResults = 'cp1252'
• DecodeKendraResults = 'utf-8'

in the bot-fulfillment lambda, do something like

'•'.encode('cp1252').decode('utf-8')

but instead of just '•', you would have the kendra search results.

When I run the above code I get the correct character - '•'.

The caveat is this only would work for a single document encoding type.. so maybe I need to figure out if I can specify document encodings within Kendra so that when the fulfillment lambda call's kendra the document is forced to be encoded/decoded with the right encoding that matches the document.

from qnabot-on-aws.