dollabs / pamela Goto Github PK

With instaparse performing syntax checks and semantic check work in progress, pamela needs a 'check' action that will perform syntax and semantic checks of the given input file.

pamela -i biased-coin.pamela check # Check syntax and semantics of the input file

Calling a method with the wrong number of arguments fails silently when generating HTN and TPN. Both name and arity are needed for a method to be applicable for method expansion, but the user isn't informed of any failure.

In contrast, changing (two 1 2 3) to (dos 1 2 3) (an undefined method name) results in an error at parse-time. Although it's not important that both cases generate errors in the same way, an arity mismatch should result in some kind of warning at HTN generation time.

Note: I'll probably create an issue suggesting that multiple arities be supported for the same method (which will obviously impact the resolution to this issue).

(defpclass arg-test2 []
  :methods [(defpmethod one []
              (sequence
               (two 1 2 3)))  ;;wrong arity
            (defpmethod two [a1 a2]
              (three a1 a2))
            (defpmethod three [b1 b2])
            ])

There is a quoting/output problem.

When using the htn action without specifying a root task (-t), if the Pamela file contains a single 0-arg main pmethod, that pmethod should be used as the root task. It currently results in a cryptic error.

dcerys@nebbiolo:~/DOLL/github/pamela/test/pamela$ pamela -v -i statements.pamela  -o ../../dan/statements-nrt htn
verbosity level: 1
log level: :warn
construct-tpn: nil
file-format: json
input: [#object[java.io.File 0x2bb62414 /Users/dcerys/DOLL/github/pamela/test/pamela/statements.pamela]]
output: ../../dan/statements-nrt
magic: nil
output-magic: nil
root-task: nil
cmd: htn (valid)
17-05-26 14:51:05 nebbiolo WARN [pamela.parser:0] - unable to determine if this keyword value is valid in pclass main :illuminated
17-05-26 14:51:05 nebbiolo WARN [pamela.parser:0] - unable to determine if this keyword value is valid in pclass main :dead
17-05-26 14:51:05 nebbiolo WARN [pamela.parser:0] - unable to determine if this keyword value is valid in pclass main :open
17-05-26 14:51:05 nebbiolo ERROR [pamela.cli:0] - 
 ERROR caught exception:
cannot create arg-mappings with static-task-args: [] and dynamic-task-args: 

pamela script needs to use uberjar when available so that it does not incur compile penalty every time is it used.

lein clean; lein uberjar
...
Created /xxx/github/pamela/target/uberjar/pamela-0.3.0.jar
Created /xxx/github/pamela/target/uberjar/pamela.jar

~/p/g/pamela> pamela -h
Compiling pamela.core
Compiling ClojureScript...
Compiling "resources/public/js/compiled/app.js" from ["src/main/cljs"]...
Successfully compiled "resources/public/js/compiled/app.js" in 13.46 seconds.

Probabalistic Advanced Modeling and Execution Learning Architecture (PAMELA)

HTN methods in PAMELA must have exactly one of {sequence parallel choose}

The place to do this is in src/pamela/htn.clj #'make-htn-methods because that's where we convert the PAMELA IR into the Clojure HTN data structure.

We need to keep track of the number of times we've seen (#{:sequence :parallel :choose} type)
Which means "is the type of this subtask in the set of :sequence :parallel :choose ?
At the end of one run through the function we need to verify that the counter is exactly 1 or error out.

When using bin/pamela script, errors are logged to logs/pamela-service.log file. Errors in pamela log files are perfectly fine but it would be very helpful if they are also logged to stdout.

Ex:
pamela -v -v -i biased-coin.pamela -f json -o biased-coin.tpn.json tpn

Reports following in the log file

16-10-03 20:59:24 xxxxxx----yyyy ERROR [pamela.tpn:1] - unable to determine tpn plant, more than one pclass takes zero args
16-10-03 20:59:24 xxxxxx----yyyy ERROR [pamela.tpn:1] - unable to find the default TPN method: please specify --construct-tpn pclass:field:method

The Pamela parser doesn't do enough validation of the arguments within field constructors.

In the following case, the parser should warn (or error) about the field definitions of :field3 and :field4 since arg3 and arg4 aren't defined (e.g., by the pclass constructor arglist).

(defpclass plant [arg1 arg2]
  :meta {:doc "Plant with 2-arg constructor"}
  :fields {:field1 arg1
           :field2 {:observable true :initial arg2}
           :field3 arg3
           :field4 {:observable true :initial arg4}}
  :methods [(defpmethod do-it-now [a1 a2 a3 a4]
              )
            (defpmethod do-it []
              (do-it-now :field1 :field2 :field3 :field4))])


(defpclass constructor-arg-test []
  :meta {:doc "Test of constructor args"}
  :fields {:plant1 (plant 1 2)}
  :methods [(defpmethod main []
              (:plant1.do-it)
              )]
  )

constructor-arg-test.pamela.txt

During the execution of this HTN, the top-level and 2nd-level parents are colored dark green, even though the 3rd level parent is still in-progress. The top-level and 2nd-level parents should remain light green until all of the children (just the one 3rd level parent in this case) is complete.

This single-pclass Pamela file displays a spurious ERROR:
ERROR, not currently supported plant-fn type= nil

Using (arg-test1.one) as the root task

However, the generated HTN and TPN seem correct

(defpclass arg-test1 []
  :methods [(defpmethod one []
              (sequence
               (two 1 2)))
            (defpmethod two [a1 a2]
              (three a1 a2))
            (defpmethod three [b1 b2])
            ])

A few related modifications are needed in the HTN (HEM) args that we just added:

1. In cases of an empty args list, we should specify [] instead of nil
2. The args should be a vector, making it consistent with TPN activities and the proposed Plan Schemas
3. In cases of IR-expanded argument references, we should un-expand the references so that they're readable to the user. See Note below.

Example of IR-expanded references:

:args
  ({:type :pclass-ctor,
    :pclass qc,
    :args [],
    :id "qc3",
    :plant-part "pp3",
    :param qc3})

Instead, we should generate

:args [qc3]

Note: This assumes that all current consumers of the :args are interested in the non-IR-digested form of the argument list. In the future, there will be additional IR-digestion of references in the argument lists (see #81), and there likely will be users of that IR representation of those arguments (e.g., Dispatcher and BSM). When this happens, we may need to modify the schema such that we have 2 versions of the args: one for display, and one for computation. However, this isn't yet needed.

Reorganize the tests under the the test/pamela directory to include the following subdirectories:

• examples: This will include all of the current test/pamela/*.pamela files (moved now to a subdirectory). Automated testing will process this directory.
• regression (or micro-examples): These are small examples testing a single Pamela language feature. Automated testing will process this directory.
• errors: Expected errors for IR and HTN. Automated testing will process this directory.
• pending: Pamela files that may be promoted to examples or errors, as the implementation changes. Automated testing will ignore this directory.

Note that the subdirectory structure of examples and regression will be the same. (Although errors will also have the same structure, the interpretation of those files is different).

Currently, values of the arguments to plant functions are defined as:

argval   ::= symbol
           | boolean
           | string
           | number
           | safe-keyword

Extend this to also include vectors and maps.

Calling (foo [arg1]) as (foo "a") should result in an HEM of (foo "a"). Instead, we currently leave it unexpanded as (foo "arg1").

Planviz is complaining about end-nodes.

In pamela/test/simple-choice-parallel

pamela -i parallel.feasible.pamela -t "(parallel-feasible.main)" -o parallel.feasible htn
planviz -v -v -l debug -i parallel.feasible.tpn.json -i parallel.feasible.htn.json
# snip
Reading input from: parallel.feasible.tpn.json
17-Mar-31 15:56:32.924 pablo DEBUG [planviz.server:0] - END NODE :parallel.feasible.tpn:node-20 NOT FOUND?
17-Mar-31 15:56:32.925 pablo DEBUG [planviz.server:0] - END NODE :parallel.feasible.tpn:node-5 NOT FOUND?
17-Mar-31 15:56:32.926 pablo DEBUG [planviz.server:0] - END NODE :parallel.feasible.tpn:node-14 NOT FOUND?

• Why is planviz complaining about these nodes?

In the generated tpn file, state nodes have an end node. Only the state nodes that represent beginning of a actual sequence should have end node slot. i.e (sequence (act1) (act2)) and not otherwise

Currently we have two slots for plant methods

• args - vector of literal values passed to the method
• argsmap - map (or JSON object) of formal-arg keys to arg values

However argsmap drops the position of the args.

If, instead, we had a vector of vectors then we would have all the data:

argsmap: [[param0 val0] [param1 val1] ....]

for example

argsmap: [[loc1 "e1h1"] [loc2 "e2h1"] [loc3 "e3h1"] [loc4 "e4h1"]]

In fact.. with this argsmap we really wouldn't need the :args slot
because we could derive it with: (mapv second argsmap)

The goal of this ticket is to decide which format to use for argsmap.

We use :id in pamela class to route messages for a instance of the plant however we don't have a mechanism to uniquely identify observations from various parts of the plant. Currently, all observations are assumed to be coming from the plant.

So we need to add a :plant-part attribute to the language as below:

# Before
-           :bulb1 (bulb :source :drain :id "Bulb1")
-           :switchedpower (psw :source :drain :id "PSW1" :interface "RMQ")})
# After
+           :bulb1 (bulb :source :drain :id "switchedcircuit" :plant-part "Bulb1" :interface "RMQ")
+           :switchedpower (psw :source :drain :id "switchedcircuit" :plant-part "PSW1" :interface "RMQ")})

## In github/dollabs/pamela
pamela -v -v -i sequence.feasible.pamela -o sequence.feasible.cfm.tpn.json -c sequence-feasible:main:main -f json tpn

Generated TPN does not has any constraints.

HTN generation attempts to expand non-empty methods, despite being marked :primitive.

Instead it should simply treat the method body as if it were empty.

Prior to IR days, generated TPN state nodes had :sequence-end node pointing to the end node of the sequence.

Current TPN state nodes have :end-node pointing to the end-node of the activity instead of the end-node of the sequence. This is the case for TPNs generated by both htn and tpn task.

Also, the :end-node is not always present.

from first.tpn.json

  "node-28": {
    "tpn-type": "state",
    "uid": "node-28",
    "constraints": [],
    "activities": [
      "na-34"
    ],
    "incidence-set": [
      "act-46"
    ]
  },

See #101 for details. I am blocked on resolution of this issue

[Once agreed upon, this information is a candidate for inclusion in the PAMELA language manual]

When calling a method (non-primitive or primitive), each of the arguments can be a literal (i.e., boolean, string, number, map, vector, (and maybe a safe-keyword)) or a reference.

There are 1+ types of reference: symbols and (maybe) safe-keywords:

Symbols must have a value (through argument unification via args list) that is one of the supported literals (see above). In addition, the value of a symbol could be a pclass instance.

Question: Although it makes sense to pass a pclass instance to a non-primitive method, does it make sense for a primitive method? [Doing so would somehow be a way of passing a plant-id and plant-part to a primitive method]

Based on some recent clarifications, it may be the case that a keyword, when used as a reference is only used as a reference to a mode.
Question: Do we support modes (designated by keywords) as arguments (for both non-primitive and primitive methods)? [I think NO]

Question: Is any non-mode keyword a literal? Do we support keyword literals as arguments (for both non-primitive and primitive methods)?

When specifying a complex literal as an argument value (map or vector), any of the components of that complex literal can be a literal or a reference.

See issue #101 and use first.pamela for details. I am blocked on this issue

Currently, when -f option is not specified, pamela generates output in edn format. Whereas edn format is good debugging, our preferred output format is json and it should be the default.

-- JSON for activities should have argsmap filled. Note args are correctly filled.
-- add plantid slot where applicable

"act-92": {
    "args": [
      5,
      0.5
    ],
    "constraints": [],
    "uid": "act-92",
    "name": "move_takeoff({})",
    "tpn-type": "activity",
    "argsmap": {},
    "command": "move_takeoff",
    "htn-node": "hem-78",
    "controllable": false,
    "end-node": "node-98",
    "label": "TEMP-move_takeoff",
    "display-name": "moveTakeoff"
  },

Resolve compiler warnings and errors

pamela -i test/pamela/cannon.pamela -t "(game.main)" htn
17-01-26 19:45:40 pablo WARN [pamela.parser:0] - unable to determine if this keyword value is valid in pclass game :open
17-01-26 19:45:40 pablo WARN [pamela.parser:0] - unable to determine if this keyword value is valid in pclass game :open
17-01-26 19:45:40 pablo ERROR [pamela.htn:0] - unhandled function type in HTN: :whenever
17-01-26 19:45:40 pablo ERROR [pamela.htn:0] - unhandled function type in HTN: :whenever

pamela -i ir-test.pamela -t "(game.main)" htn
17-01-30 20:43:09 pablo WARN [pamela.parser:0] - unable to determine if this keyword value is valid in pclass box :high
17-01-30 20:43:09 pablo ERROR [pamela.cli:0] - unable to parse: [#object[java.io.File 0x3cc6d144 "/Users/prakash/projects/github/dollabs/pamela/test/pamela/ir-test.pamela"]]
error: Symbol argument to pclass constructor bob is neither a formal argument to, nor a field of the pclass game

Appears to be a bug as the error suggests. However, since there is a valid IR, I might be missing something.

time pamela -h gives
5.70 real 21.29 user 0.77 sys

time java -jar target/pamela.jar -h
2.42 real 7.90 user 0.29 sys

Why? What changed?

Pamela compiles a model perfectly fine but planviz refuses to load it.

TPN generated by following command does not load in planviz.
pamela -v -v -i biased-coin.pamela -f json -o biased-coin.tpn.json --construct-tpn main:necessary-evil:flip tpn

16-Oct-03 17:16:04.787 xxx-yyy ERROR [planviz.server:1014] - error parsing TPN plan: ../biased-coin.tpn.json
16-Oct-03 17:16:04.788 xxx-yyy ERROR [planviz.server:1015] - {:na-104 {:probability (not (instance? java.lang.Number a-clojure.lang.PersistentArrayMap))}, :na-95 {:probability (not (instance? java.lang.Number a-clojure.lang.PersistentArrayMap))}, :na-121 {:probability (not (instance? java.lang.Number a-clojure.lang.PersistentArrayMap))}, :na-130 {:probability (not (instance? java.lang.Number a-clojure.lang.PersistentArrayMap))}}

For first.pamela, When using htn task,

• We should be able to create HTN from pamela class first-tpn instead of first-tpn-htn
• in between constraints are not created
• Bounds are not created for plant activities
• Appearance of labels is unexpected and misleading.

TPN created using cfm method

TPN created using htn task

See pamela file for individual commands for compiling first.pamela file

Following command line used to work but not anymore.

ls -l biased-coin.pamela
-rw-r--r-- 1 ... staff 2286 Nov 9 14:00 biased-coin.pamela
~/p/p/demo-nov-2016-pamela> pamela -i biased-coin.pamela -o biased-coin.1000-flips.tpn.json --construct-tpn main:flip-sequence:flip-1000 -f json tpn
16-11-09 19:03:58 prakash-retina ERROR [pamela.cli:0] -
Failed to validate "-i biased-coin.pamela": INPUT file does not exist

Probabalistic Advanced Modeling and Execution Learning Architecture (PAMELA)

Usage: pamela [options] action

Commit ff32613 has relevant artifacts.

I expect tpn file as in this commit.
@dcerys Please verify and update htn file.
@tmarble Please verify both htn and tpn files.

Generated TPN's have empty incidence set. See tpn below.

{
"node-85": {
"tpn-type": "state",
"uid": "node-85",
"constraints": [],
"activities": [
"act-84"
],
"incidence-set": []
},
"net-74": {
"tpn-type": "network",
"uid": "net-74",
"begin-node": "node-85",
"end-node": "node-72"
},
"network-id": "net-74",
"act-100": {
"tpn-type": "activity",
"uid": "act-100",
"constraints": [],
"end-node": "node-72",
"controllable": false,
"plant": "this",
"command": "flip",
"args": [],
"argsmap": {}
},
"node-93": {
"tpn-type": "state",
"uid": "node-93",
"constraints": [],
"activities": [
"act-92"
],
"incidence-set": []
},
"act-92": {
"tpn-type": "activity",
"uid": "act-92",
"constraints": [],
"end-node": "node-101",
"controllable": false,
"plant": "this",
"command": "flip",
"args": [],
"argsmap": {}
},
"act-84": {
"tpn-type": "activity",
"uid": "act-84",
"constraints": [],
"end-node": "node-93",
"controllable": false,
"plant": "this",
"command": "flip",
"args": [],
"argsmap": {}
},
"node-72": {
"tpn-type": "state",
"uid": "node-72",
"constraints": [],
"activities": [],
"incidence-set": []
},
"node-101": {
"tpn-type": "state",
"uid": "node-101",
"constraints": [],
"activities": [
"act-100"
],
"incidence-set": []
}
}

Currently the "tpn" action in PAMELA will generate a TPN directly from PAMELA source (1). Instead we want to, in the medium term, use the existing HTN+TPN generation algorithm (2). In the future we'll use the new HTN->TPN algorithm (#41).

1. (This has been called TPN generation algorithm 1).
2. (This has been called TPN generation algorithm 2).

pamela fails to generate tpn for big-sequence

pamela -i big-sequence.pamela -o big-sequence.tpn.json -f json --construct-tpn main:flip-sequence:flip-sequence tpn
Compiling ClojureScript...
ERROR caught exception: nil

analyze reports success for a pamela file but pamela fails to generate json.

analyze sequence.infeasible.pamela 
Analyzing PAMELA file: /Users/prakash/projects/github/pamela/src/test/pamela/sequence.infeasible.pamela
Checking grammar for: sequence.infeasible.pamela
  Success! parse tree in:  /Users/prakash/projects/github/pamela/src/test/pamela/sequence.infeasible.txt
  parse diagram in:  /Users/prakash/projects/github/pamela/src/test/pamela/sequence.infeasible.png

pamela -f json -i sequence.infeasible.pamela -o sequence.infeasible.json tpn
Compiling ClojureScript...
16-05-27 21:28:49 prakash-retina DEBUG [io.netty.util.internal.logging.InternalLoggerFactory:71] - Using SLF4J as the default logging framework
16-05-27 21:28:49 prakash-retina DEBUG [io.netty.util.internal.PlatformDependent0:76] - java.nio.Buffer.address: available
16-05-27 21:28:49 prakash-retina DEBUG [io.netty.util.internal.PlatformDependent0:76] - sun.misc.Unsafe.theUnsafe: available
16-05-27 21:28:49 prakash-retina DEBUG [io.netty.util.internal.PlatformDependent0:71] - sun.misc.Unsafe.copyMemory: available
16-05-27 21:28:49 prakash-retina DEBUG [io.netty.util.internal.PlatformDependent:76] - sun.misc.Unsafe: available
16-05-27 21:28:49 prakash-retina DEBUG [io.netty.util.internal.PlatformDependent:71] - Javassist: available
ERROR caught exception: clojure.lang.ArityException: Wrong number of args (0) passed to: pclass/defpmethod, compiling:(null:1:1)

Argument values in method calls (i.e., argvals of plant-fn calls) that are symbols/keywords are kept as literal symbols, in contrast to other symbols (like the method name) which are parsed into context-ful structures (maps). This is true for: the generated IR, HTN, and TPN.
It might be useful (perhaps essential?) to do the same as for arguments to a Pamela conditional.

E.g.,
The method invocation:

(generator.generate-candidate-positions :gu1.next-lat :gu1.next-lon 
   :gu2.next-lat :gu2.next-lon :gu3.next-lat :gu3.next-lon)

generates the following in the IR:

{:type :plant-fn-field,
        :method generate-candidate-positions,
        :args
        [:gu1.next-lat
         :gu1.next-lon
         :gu2.next-lat
         :gu2.next-lon
         :gu3.next-lat
         :gu3.next-lon],
        :field :generator}

These literals carry through from the IR to the generated HTN and TPN. So, it's up to the user of the activities (e.g., the Dispatcher) to (re)construct the context and perform the correct dereferencing. The problem is that the complete context won't be available (unless we pull the IR along all the way). In the above example, there is no way that the field variable :gu1 corresponds to the ground-unit pclass instance with the :plant-id of "ground-unit1".

In contrast, the conditional (= :cannon-f.:ammunitions 0)
is expanded into

{:type :equal,
 :args [{:type :field-reference-field,
         :pclass this,
         :field :cannon-f,
         :value :ammunitions}
         {:type :literal, :value 0}]}

Still, this assumes some context that might not be available downstream, but it's a better situation. We would still need to be able to determine the appropriate pclass-ctor (in order to obtain the correct :plant-id and :plant-part). Of course, expanding all method args in this manner would require changes throughout htn.clj and in the Dispatcher.

Similar to Clojure's dotimes this should expand

(dotimes 3 (flip))

into

(sequence (flip) (flip) (flip))

This is a 'macro' in that dotimes will never appear in the IR, instead the expanded sequence will be put in it's place.

Currently there are certainly legacy "slots" in the TPN schema (e.g. name) and new slots in the HTN schema (e.g. label) that are either obsolete or need to be renamed for consistency. This issue will track this harmonization effort.

Add optional :primtive true meta data to defmethod such that
the method body is saved in the IR (but still treated like a primitive
method in generating the TPN).

Low Priority

At some point, it will be useful to have multiple methods of the same name, with different arities:

E.g., (foo 1 2 3) and (foo 1 2)

For now, this is Low Priority

This will allow us to do regression testing of the HTN/TPN generation process

As a first pass at supporting ask/tell/assert, we will treat them like 1-arg plant functions. Initially, the plant-id for these will refer to the Belief State Manager.

In the future, to handle all of Pamela constructs (e.g., unless, when), we’ll need to make some structural changes to the HTN and TPN representation.

In the following example, the :bounds declared for the primitive methods (e.g., do-a) aren't included in the generated TPN (generated using the htn action)

(defpclass plant []
           :meta {:doc "The Plant API"}
           :methods [(defpmethod do-a {:bounds [2 4]} [])
                     (defpmethod do-b {:bounds [3 6]} [])
                     (defpmethod do-c {:bounds [4 8]} [])
                     (defpmethod do-d [])                  
                     (defpmethod do-e {:bounds [5 10]} [])])

(defpclass infeasible-sequence [plnt]
           :meta {:doc "An example of infeasible sequence of activties"}
           :methods [(defpmethod start
                                 {:doc "Simple TPN with constraints"}
                                 []
                                 (sequence :bounds [10 30]
                                           (plnt.do-a)
                                           (plnt.do-b)
                                           (plnt.do-c)
                                           (plnt.do-d)
                                           (plnt.do-e)))])

PAMELA should be able to convert a well formed HTN with a root-task into the corresponding TPN. (This has been called TPN generation algorithm 3).

Recently the delay-activity TPN-TYPE was defined in plan-schema.

The TPN generation function in PAMELA should translate 'delay' functions into delay-activities.

For pamela/test/simple-choice/choice.feasible.pamela TPN in planviz does not render correctly.

TPN Choice View

TPN Parallel View looks better