php-rust-tools / parser Goto Github PK

View Code? Open in Web Editor NEW

313.0 8.0 17.0 4.2 MB

A handwritten fault-tolerant, recursive-descent parser for PHP written in Rust.

License: Apache License 2.0

Rust 99.88% Just 0.12%

lexer parser php-parser rust php-rust php-rust-tools fault-tolerant recursive-descent-parser

parser's Introduction

PHP-Parser

A handwritten fault-tolerant, recursive-descent parser for PHP written in Rust.

Warning - this is still alpha software and the public API is still subject to change. Please use at your own risk.

Usage

Add php-parser-rs in your Cargo.toml's dependencies section

[dependencies]
php-parser-rs = { git = "https://github.com/php-rust-tools/parser" }

or use cargo add

cargo add php-parser-rs --git https://github.com/php-rust-tools/parser

Example

use std::io::Result;

use php_parser_rs::parser;

const CODE: &str = r#"<?php

final class User {
    public function __construct(
        public readonly string $name,
        public readonly string $email,
        public readonly string $password,
    ) {
    }
}
"#;

fn main() -> Result<()> {
    match parser::parse(CODE) {
        Ok(ast) => {
            println!("{:#?}", ast);
        }
        Err(err) => {
            println!("{}", err.report(CODE, None, true, false)?);

            println!("parsed so far: {:#?}", err.partial);
        }
    }

    Ok(())
}

License

Licensed under either of

Apache License, Version 2.0 (LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0)
MIT license (LICENSE-MIT or http://opensource.org/licenses/MIT)

at your option.

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

Credits

parser's People

Contributors

Stargazers

Watchers

Forkers

jeffersonsimaogoncalves dohxis jobin-jas edsrzf doytsujin azjezz danog kennedytedesco isgasho frewsxcv cezar-popa tu6ge k-adam 4812571 zuruuh ulrichsg evandropoa

parser's Issues

Add support for heredocs / nowdocs

Not sure on the best approach yet. I think it'll be something like producing a start & end token, then parsing everything in the middle as either a constant string or an interpolated string.

Parse all type declarations

There are several varieties of type declaration that Trunk can't parse:

It also might be nice to have a richer type structure to deal with so that not everything is a string, but this is probably a separate issue.

Allow semi-reserved words in various places

PHP has semi-reserved words which can be used as names in some contexts. The Trunk parser appears to know about these already, but does not allow them to be used in all the places they should be allowed.

See test cases:

Also see uses of semi_reserved in the php-parser grammar: https://github.com/nikic/PHP-Parser/blob/master/grammar/php.y

I'll try to fill out this issue with more specific details in the future.

Parse close tags

The Trunk parser doesn't handle PHP close tags. eg:

<?php
// Doesn't parse:
?>
<html>

Parse do/while loops

Trunk can't parse do/while loops. eg:

do {
} while ($a);

Parse anonymous classes

$i = new class {}

$j = new class(1) {}

$k = new class extends Foo implements Bar {}

Parse backed enum cases without values

A backed enum that has a case with no value causes a runtime error, but is syntactically valid and can be parsed by nikic/php-parser. See this test case.

Parse goto and labels

See the docs.

<?php
goto a;
echo 'Foo';
 
a:
echo 'Bar';
?>

Add an `attributes` field to all statements

I want to introduce a new Attributes structure and relevant field to all statements so that we can store things like start_line, end_line and any comments on the statement.

Parse catch without a variable

The variable in a catch is optional.

try {
} catch (Exception) {
}

Parse array destructuring with `list`

list can be used to destructure arrays, but this doesn't parse:

<?php
list($a, $b, $c) = $array1;
list('foo' => $bar) = $array2;

Parse __halt_compiler

__halt_compiler is a pseudo-function that stops compilation past the point it's "called". Any arbitrary data can follow and it doesn't need to be valid PHP.

<?php

__halt_compiler();
We can put anything we want here, and it should still parse!

Support `try` statements

Move expressions to use a struct-type enum

The current setup on the Expression enum is a tuple/tagged union. This makes things a little hard to recognise in some cases, like anonymous classes. We should either store a single struct inside of the enum or use a structure on the variant directly and have named fields.

Parse shell execution operator (backticks)

Parse shell the shell execution operator (backticks):

<?php

``;
`test`;
`test $A`;
`test \``;
`test \"`;

Classes ending with comments don't parse

I tried running this test case from nikic/php-parser: https://github.com/nikic/PHP-Parser/blob/master/test/code/parser/commentAtEndOfClass.test

class MyClass {
    protected $a;
    // my comment
}

It currently doesn't parse at all due to the comment as the final line of the class body.

Parse references in loops and functions

These should parse:

<?php
function a(&$b) {}
function &a($b) {}
function &($a) {};
static fn(&$x) => $x;
fn&($x) => $x;
foreach ($a as &$b) {}
foreach ($a as $b => &$c) {}

Somewhat related to #19, since & can be a unary operator, but here we're talking about non-expression contexts.

Get rid of comment hacks

We just have loads of checks for comments at the moment to essentially make them noop statements, etc.

We should remove this and instead have some kind of store_comments() method that will store them on the struct and then another method that goes with #10 to clear them from the parser and stores them in the attributes.

Parse PHP attributes

Parse PHP attributes.

See test cases: https://github.com/nikic/PHP-Parser/blob/master/test/code/parser/stmt/attributes.test

Parse `use` statement modifications in classes

Use statements inside of classes can have modifications applied to change the precedence or visibility of a method.

https://www.php.net/manual/en/language.oop5.traits.php

Implement all of PHP's operators

https://www.php.net/manual/en/language.operators.precedence.php provides a list of operators ordered by their precedence. We support quite a few of these already but there's some things like bitwise ops and casts that still need to be implemented.

Unary

Binary

Assignment

Parse braced global namespace

This is a valid way to access the global namespace, but doesn't parse:

<?php
namespace {
    function globalFunc() {}
}

Parse array spread

These should parse:

<?php
[...[]];
[...[1, 2, 3]];
[...$array];
[...getArr()];
[...arrGen()];
[...new ArrayIterator(['a', 'b', 'c'])];
[0, ...$array, ...getArr(), 6, 7, 8, 9, 10, ...arrGen()];
[0, ...$array, ...$array, 'end'];

Lex binary strings

String literals can be prefixed with b or B:

<?php

b'';
b"";
b'Hi';
B'Hi';

// Heredocs and nowdocs too!
b<<<EOS
Binary
EOS;

I can't find any official documentation for this, but there are test cases for it and I found this Stackoverflow question. Apparently it was intended as a forward-compatibility feature for PHP 6, but PHP 6 was never released.

Closures trying to be parsed as function statements

function () {};

The parser thinks there should be an identifier after the function since it sees the function keyword. Just need to check if the peek isn't an identifier I think, it'll fall down into the expression logic then.

Parse `global`

<?php
$a = 1;
$b = 2;

function Sum()
{
    global $a, $b;

    $b = $a + $b;
}

Parse top-level constants

Constants inside classes will parse, but top-level constants currently don't:

<?php

const A = 0, B = 1.0;
const C = 'const';

Prevent named arguments being used before positional arguments

function foo($bar, $baz) {}


foo(baz: "Baz", "Bar");

nikic/php-parser produces a valid AST for this since it doesn't do anywork to validate the AST, but we can do one better and actually produce a ParseError for this instance since it's not allowed at runtime.

Move away from binding powers

Binding powers are kind of annoying given how complicated PHP's precedence rules can get. I think I should just move it all over to a regular Precedence enum or something and then have an Associativity rule too.

(This is where parser generators kind of dominate).

Parse first-class callables

These should all parse:

<?php
foo(...);
$this->foo(...);
A::foo(...);

nikic/php-parser also accepts these, even though they result in runtime errors:

<?php
new Foo(...);
$this?->foo(...);

Make regular assign expression an infix op

Trunk can't handle invalid UTF-8 in PHP source code

The Trunk lexer accepts a string and operates on chars. PHP source does not need to be valid UTF-8, and so in degenerate cases can't be represented as a Rust string or as individual chars. In particular, PHP docs state:

A valid variable name starts with a letter or underscore, followed by any number of letters, numbers, or underscores. As a regular expression, it would be expressed thus: ^[a-zA-Z_\x80-\xff][a-zA-Z0-9_\x80-\xff]*$

For instance, you can run this on the CLI:

$ printf '<?php $var\xff = 1; echo $var\xff . "\n";'|php
1

...and it works! This uses the character \xff as part of a variable name, which is not otherwise printable and is invalid UTF-8:

$ printf '<?php $var\xff = 1; echo $var\xff . "\n";'|iconv -f UTF-8
<?php $var
iconv: (stdin):1:10: cannot convert

This is admittedly a fairly bizarre corner of the PHP language, but if this toolchain wants to be compatible with the main PHP implementation, it should be fixed.

Parse grouped use statements

We already support basic use statements at the top of a file.

use Foo;
use Bar;

We should also support grouped statements though.

use App\{Foo, Bar, Baz as Car};

Parse readonly properties and classes

Readonly properties, which are required to have type declarations:

<?php

class C {
    readonly int $a;
    readonly protected string $b;
    readonly static stdClass $c;

    function __construct(readonly bool $d) {}
}

Readonly classes: (coming in PHP 8.2)

<?php

readonly class C {}

Nullsafe operator doesn't parse

Nullsafe parsing is implemented, but it doesn't seem to work correctly. A simple test case fails to parse at all:

<?php $a?->b;

Fix escape sequences in double-quoted strings

"Foo\n"

Is being parsed as Foon instead of inserting the line break.

Parse static closures

This doesn't parse:

<?php
static function() {};

Support spread/splat/rest parameters

We need to support parsing this:

function foo(...$bar) {}

function bar(string ...$baz) {}

function baz($foo, $bar, ...$baz) {}

https://phpast.com/?c=PD9waHAKCmZ1bmN0aW9uIGZvbyguLi4kYmFyKSB7fQoKZnVuY3Rpb24gYmFyKHN0cmluZyAuLi4kYmF6KSB7fQoKZnVuY3Rpb24gYmF6KCRmb28sICRiYXIsIC4uLiRiYXopIHt9

Parse `match` expressions

match ($expr) {
    'foo' => 'bar',
    'baz', 'car', 'bob' => 'foop',
    default => null
}

Parse `switch` statement

switch ($foo) {
    case 'bar':
        echo $foo;
    case 'baz':
    case 'car':
        break;
     default:
        echo 'foo';
}

Parse all valid `yield` expressions

yield with no expression
yield key-value (yield 'key' => 'value')
yield from

See the php-parser test cases.

Parse `declare`

See the docs and the php-parser test cases.

<?php

declare (X='Y');

declare (A='B', C='D') {}

declare (A='B', C='D'):
enddeclare;

Parse control flow statements without blocks

Trunk can't parse control flow statements without block bodies. See the blocklessStatement test cases.

Parse arrays with missing elements

Arrays can have missing elements:

<?php

[1, 2, , 4];

This also applies to destructuring assignments. (Somewhat related: #58)

Handle all types of numeric literals

The lexer doesn't handle various cases for numeric literals:

e-notation for floats (eg 0E0, 1e10000)
Hexadecimal literals (0xCAFEF00D)
Octal literals (0700)
Explicit octal literals (0o700)
Binary literals (0b01011111)
Large integer literals should overflow to floating-point (18446744073709551615) - currently they result in an error.

Ensure that identifiers also include non-reserved keywords

The current expectation in the parser is that all identifiers are non-keywords. This isn't quite right though since PHP has introduced new keywords in releases and wanted to preserve compatibility with method names etc.

This should be quite an easy change now that we dedicated methods for the expectations. If something is a keyword, we just need to return the string form instead of the captured identifier.

We could use nikic/php-parser as a source for the soft-reserved/non-reserved keywords.