we should probably add this?

EXTRA keeps track of the lineno, col_offset, end_lineno, end_col_offset. At the moment this is just a temporary empty list. To stop the compiler breaking.

Searching for EXTRA in cpython's c_generator.py/parse.c might with implementing this and including it in the generated code.
The python_generator that skulpt_generator descends from doesn't keep track of these variables since (afaik) it didn't get that far.

def f(*a, b):
    pass

This ends up with kw_defaults=[null]
but it should be kw_defaults=[None]

I think this is correct at the point of the ast
And it's just working out the best place to convert it to None

This particular path could be solved in

export class NameDefaultPair {
    arg: arg;
    value: expr | pyNoneType;
    constructor(arg: arg, value: expr | null) {
        this.arg = arg;
        this.value = value ?? pyNone;
    }
}

and then fixing all the typescript problems that result... but I don't know if that's a good idea
... might mean generating our own asdl patch

I vote 3...
so then the workflow for working with the grammar can be something like:

Generate patch

copy python.gram into skulpt_parser somewhere (maybe tools/grammar_patch)
git diff with skulpt.dev.gram and copy the output to the same folder
Then either as the apply_grammar_patch or just as step 3 in the above.
apply the patch to skulpt.gram somewhere (maybe src/grammar)

Then I think we can abandon #12 and it can just live in the tools/grammar_patch folder as a future helper/reference function.
(I'll adjust that pr after this pr)

search ASYNC in tokenizer.c for the implementation of this.

because @meredydd

Lib/test/test_peg_parser.py

similar to our ttests but will add more py 3 idioms

we should add some

Before running the symtable in compile.c there is an ast optimization phase that we should explore.

It seems to do some code folding of literal values.

see ast_opt.c

same issue we had with the constant types

Maybe we should move our error classes into a similar name space for now

when i turned off //@ts-nocheck there were 6 examples of this return statement that were complained about.

Seems strange.
It might be ok because from what I can tell a truthy ends up as a raised error.
But there's probably a better way to do the same thing without type script complaining.

here's the line in the python generator that we need to keep in mind

                action = node.action
                if not action:
                    if is_gather:
                        assert len(self.local_variable_names) == 2
                        action = (
                            f"[{self.local_variable_names[0]}] + {self.local_variable_names[1]}"
                        )
                    else:
                        action = f"[{', '.join(self.local_variable_names)}]"

see email from Kevin

this modal could also do with a new design update

Ultimately we should move the verbosity stuff out of the memoize functions because we probably don't need to ship a verbose parser outside of this repo.

Maybe we could instead generate a verbose parser like

vr gen_parser --verbosity=0 # no log statements
vr gen_parser --verbosity=1 # logs anytime we call a pegen function
vr gen_parser --verbosity=2 # logs anytime we try to find a cached token

Or something like that

we may never get round to this (adding wontfix) but currently the tokenizer isn't set up for interactive mode.
So parsing in interactive mode will fail for anything other than a single line statement.

Think this needs to be part of getting the EXTRA metadata

a list of functions we need to implement that are called from the generated_parser.ts

• make_module
• seq_append_to_end
• singleton_seq
• seq_flatten
• interactive_exit
• set_expr_context
• NEW_TYPE_COMMENT
• augoperator
• map_names_to_ids
• seq_count_dots
• alias_for_star
• join_names_with_dot
• function_def_decorators
• empty_arguments
• make_arguments
• slash_with_default
• star_etc
• add_type_comment_to_arg
• name_default_pair
• class_def_decorators
• seq_insert_in_front
• get_cmpops
• get_exprs
• cmpop_expr_pair
• concatenate_strings
• get_keys
• get_values
• key_value_pair
• collect_call_seqs
• dummy_name
• seq_extract_starred_exprs
• seq_delete_starred_exprs
• join_sequences
• keyword_or_starred
• nonparen_genexp_in_call
• arguments_parsing_error
• get_expr_name

It looks a little less intimidating here then it does when looking at the C file!

@s-cork what do you think?

The cache seems to be quite a slow point in the code

the python parser uses a dictionary cache - which is fine for generating the grammar
#79 improves the performance
But there's probably a better way to do this.

Code to explore in pegen is the is_memo, update_memo, insert_memo

Accessing and inserting into the cache seems to be a fairly performance consuming operation

typedef enum _expr_context { Load=1, Store=2, Del=3 } expr_context_ty;

typedef enum _boolop { And=1, Or=2 } boolop_ty;

typedef enum _operator { Add=1, Sub=2, Mult=3, MatMult=4, Div=5, Mod=6, Pow=7,
                         LShift=8, RShift=9, BitOr=10, BitXor=11, BitAnd=12,
                         FloorDiv=13 } operator_ty;

typedef enum _unaryop { Invert=1, Not=2, UAdd=3, USub=4 } unaryop_ty;

typedef enum _cmpop { Eq=1, NotEq=2, Lt=3, LtE=4, Gt=5, GtE=6, Is=7, IsNot=8,
                      In=9, NotIn=10 } cmpop_ty;

We generate constructors for each of these and call the constructor every time we need an instance. e.g.

export class expr_context extends AST {}
expr_context.prototype.tp$name = "expr_context";

export type expr_contextKind = typeof expr_context | typeof Load | typeof Store | typeof Del;

export class Load extends expr_context {}
Load.prototype.tp$name = "Load";
export class Store extends expr_context {}
Store.prototype.tp$name = "Store";
export class Del extends expr_context {}
Del.prototype.tp$name = "Del";

But we could just create const references to prevent having to instantiate these each time.

These are generated by TypeDefVisitor(f)

Todo

• work out what the best abstraction is

we need to work out where keyword checks get implemented

search ->keywords in the pegen directory for cpython

started to work on this and then realized that cpython is broken
https://bugs.python.org/issue35212
python/cpython#19398

We should just report the correct offsets if we can

something nicer than e.ctx.constructor._name

postpone making python types till in compiler

intermediate types can toString to the exact cpython value, while maintaining metadata to construct skulpt types.

drawbacks:

• our ast differs from cpythons because well use javascript types (with maybe annotation) we would need extra information to construct the right python type.

benefit:

• not tied to skulpts implementation

deno recently added ffi support

this package: https://github.com/denosaurs/deno_python
works great

proof of concept takes the python->deno->python peg_parser tests from minutes to seconds

we should re-write the tests to take advantage of this!

this is the reason we fail on all the string tests at the moment with the extra quotes in the ast output.

Before we concatenate we need to parse each one and determine the mode

Most/all of the skulpt code for parsing a string can be used here.

see parse_str.c for cpython's implementation

we need to not use bigints unless they're available. If they are available use them.
If not then use JSBI from the global scope which skulpt adds?

pseudo code

declare JSBI

class pyInt {
    constructor(v) {
        if ( v > Number.MAX_SAFE_INTEGER) {
            if (typeof BigInt !== "undefined") {
               // ...
            } else if (typeof JSBI !== "undefined") {
                // we're in skulpt world and in a browser that doesn't support BigInt so use JSBI

We'll need to work out how to handle these for the Constant node type.

It might be nice to use JavaScript objects so that we can completely isolate the parser from the Sk namespace and generate the real python objects at compile time.

Linking the comment from cpython that suggests this might be challenging

https://github.com/python/cpython/blob/v3.9.5/Include/asdl.h#L10-L15

We'll need to make sure the const enums get converted to their raw values in production
esbuild doesn't do that for us unfortunately
https://esbuild.github.io/content-types/#no-type-system

Issue arises in
"from .a import b"

the '.' gets tokenized as an OP but it's really a DOT
This is the correct output from tokenize.py too.

In the cpython version of the parser they don't pass the string to this.expect they pass the type

We need to work out how to fix OP's like this in our own parser.
I imagine a bit like how we changed the type of the Name when we found a keyword.

In order to generate the AST we need AST nodes

Not yet implemented
can probably be mostly ported over from skulpt

implement raise behaviour

How cpython tested the new parser:

• download the most popular pypi packages
• compare the asts

once we get through the run-tests directory we can try to implement this 😉

See: cpython/Tools/peg_generator/scripts/test_pypi_packages.py

I think it's time we make a test that runs the rus test 😄

This needs to be inserted somewhere for error reporting

"'中文🍕'"

using tokenize.py
TokenInfo(type=3 (STRING), string="'中文🍕'", start=(1, 0), end=(1, 5), line="'中文🍕'")
we get
TokenInfo(type=3 (STRING), string="'中文🍕'", start=(1, 0), end=(1, 6), line="'中文🍕'")
so the pizza emoji means we're out in the coloffset

but strangely this should result in the following ast

      Module(
        body=[
          Expr(
            value=Constant(
              value='中文🍕',
              lineno=1,
              col_offset=0,
              end_lineno=1,
-             end_col_offset=6), # we say 6
+             end_col_offset=12), # python says 12
            lineno=1,
            col_offset=0,
            end_lineno=1,
-           end_col_offset=6)],
+           end_col_offset=12)],
        type_ignores=[])

We say 6 but python says 12

No Astral codepoints
without the pizza emoji our tokenize matches python tokenize - (no astral codepoints) but our ast is still incorrect

"'中文'"

using tokenize.py
TokenInfo(type=3 (STRING), string="'中文'", start=(1, 0), end=(1, 4), line="'中文'")
we get
TokenInfo(type=3 (STRING), string="'中文'", start=(1, 0), end=(1, 4), line="'中文'")

but

      Module(
        body=[
          Expr(
            value=Constant(
              value='中文',
              lineno=1,
              col_offset=0,
              end_lineno=1,
-             end_col_offset=4), # we say 4 - this is correct no??
+             end_col_offset=8), # python says 8
            lineno=1,
            col_offset=0,
            end_lineno=1,
-           end_col_offset=4)],
+           end_col_offset=8)],
        type_ignores=[])

Is this a bug or a feature from cpython?!
see t542.py