No Working Examples in Documentation! about commai-env HOT 5 CLOSED

I managed to find examples in a presentation you gave at a workshop:

https://mainatnips.github.io/mainatnips.github.io/slides/allan-commai-env-workshop-talk.pdf

So apparently you use periods to mark end of statements.

That allows me to solve the noun phrase type questions.
I've achieved a score of 1, but no greater. possibly due to curses interface bugs.

However I can't reverse engineer your grammar as an English speaking human.

So how do you expect AI's to use it?

Typically languages require a grammar which the teacher and learner are both aware of.
Homo sapiens are predisposed to relying on adposition or affix systems,
which seems to be lacking from your language.

Do you have anyone with linguistics experience on your team?

from commai-env.

Hi @elspru,

Thanks for your feedback! Could you please specify which tasks you are referring to? We don't expect any AI can 'use' the environment yet, that would be quite remarkable :)

We do indeed have people with linguistics experience on our team... Would you like to chime in @mbaroni?

By the way, for more general discussion about the task and environment design (e.g. task grammar), it would be great if you could build your discussion here: https://www.facebook.com/groups/1329249007088140

Thanks again
Allan

from commai-env.

Hi @elspru ,

Thanks for your feedback also from me.

The simple grammars used for most tasks are documented here: https://github.com/facebookresearch/CommAI-env/blob/master/TASKS.md

The issue of whether humans can reverse-engineer our grammars is a very interesting one, and we plan to pursue it experimentally. Note that we don't expect a human to be able to do it very quickly, but possibly over time with pencil and paper, etc.

Also, we don't claim that the simplified language we use in our simulations is very much like natural-language. What we do hope is that some of the basic phenomena that an agent should possess in order to solve these tasks (ability to associate symbols and entities, memory, chunking...) will later be useful for the agent to learn proper language from a human teacher.

Regarding what children know about grammar before they start learning it, as you undoubtedly know, that's a very controversial and open issue in linguistics. We leave it to AI developers to decide how much pre-encoded knowledge of grammar they want to write into their systems.

As @ajabri suggested, it'd be great if you moved the discussion to the Users list, so other can participate.

Best,

Marco

from commai-env.

I've made a post in the facebook group, feel free to reply there.

"Regarding what children know about grammar before they start learning it, as you undoubtedly know, that's a very controversial and open issue in linguistics. "

I think only very uninformed people still believe in blank slate or anything of the sort. For instance universals archive lists over two thousand universals http://typo.uni-konstanz.de/archive/intro/

All human languages have nouns and verbs, and they also all use either adpositions or affixes for complex sentences (greater than ~3 noun phrases). And if they don't use adpositions/affixes for simple sentences then they have a well defined word-order.

I'd cite more research, but as someone that is supposed to know linguistics, I'm sure you know of all the research pointing to how SOV is easier for the brain to process, and that even deaf people developed an SOV language even though surrounded by SVO/VSO language.
Also from your research on language change you should know that SVO mostly became popular due to loss of nominal-accusative case distinction.

So I don't really know who in their right mind is still claiming that humans aren't predisposed to common forms of grammar.

If your language doesn't work with a human, then it's not a good test for an AI that is necessarily subhuman. Since the general-AI challenge isn't giving us 10-20 PFLOPs of processing power (human level) to work with, just 7-9 TFLOPs (more than a thousand times less).

from commai-env.

Hi @elspru ,
Thanks again for your feedback (and again, if you don't mind, it would be great to discuss such matters on the Users list, rather than here).
Sure, I totally agree that the fact that humans do have some predisposition for grammar is out of question, what is controversial is what that involves. E.g., are certain principles of grammar innate, or is the "language" that allows us to formulate such principles innate? Personally, I'd rather err on the nativist side, but I know that there is a wide spectrum of views.
Note that the GoodAI people are not using our more advanced tasks for their challenge. They are using their own version of the "CommAI-mini" tasks we documented here: https://arxiv.org/abs/1701.08954
It would be indeed interesting to see if humans would be able to solve those tasks, my hunch is that they would, and subject experiment is a strand of research we'd definitely like to pursue.

from commai-env.