Delphi string literals with '#' prefixed character parts - erroneous parsing about delphiast HOT 6 OPEN

@polyaklaci thank you for so detailed reports! I've fixed most of them. Please try the latest source.

By the way, what are you using DelphiAST for? It must be an extensive use, since you've got so many issues :)

from delphiast.

Thanks for the thank :) I'm happy if you can use my codes...

As I wrote in my 'newest' #63 issue :) I'm writing a 'Delphi to JavaScript' compiler, and I use your DelphiAST in this project (I generate JavaScript from the AST which is generated by DelphiAST).
This is a 'hard-test' for DelphiAST, because I need every 'details' in 'well-readable' form from the Delphi code (in the AST) to generate 'correct' JavaScript.
My 'Delphi to Javascript' project now can compile very complicated Delphi code to JavaScript. The JS syntax is 100% OK, the hardest part now is the full type-mapped symbol table (which is need for correct JS code). e.g. in Delphi the method calls without parameters doesn't require '()' at the end of method name, but JS requires. If we have a Delphi expression SomeDotpart1.SomeDotpart2.SomeDotpart3, in JavasScript it can be for example SomeDotpart1(1,2,3).SomeDotpart2().SomeDotpart('a','b','c') because of the parameter's default values, and the missing '()' in Delphi. And it's not so simple to 'find it out' at JS compiling...

I embedded the Mozzilla spidermonkey JS engine to Delphi, so I can run JS on it (where the JS objects and the Delphi objects are 'cross-usable', e.g.: a Delphi form property / method can be used in JS code). The aim of my efforts, to make a 'client framework' where the user-side functionality can be modified or can be extended without recompile the software. I just write and test the 'new functionality' in Delphi, then I compile it to JS and send it to the 'client framework' (which can run the JS code, and also can create forms from DFM file at runtime). When I sent the JS+DFM to the client, the new (or modified) user-side functionality is usable (don't need to recompile or change the client's compiled code)...

from delphiast.

This sounds very interesting!
Are you going to sell it, open source it or keep it to yourself?

from delphiast.

I try to find a solution for the problem. I rewrote TPasSyntaxTreeBuilder.StringConstSimple (to add 'char' type specifier for ptAsciiChar tokens) and TPasSyntaxTreeBuilder.StringConst (for generate expression with ADD operators from character constants without explicit ADD operators).

With this solution at the sample assignment above:

const
  SAMPLE = 'apple#20'#13#10;

now the RHS expression is parsed to:

<EXPRESSION line="13" col="11">
  <ADD line="13" col="27">
    <LITERAL value="apple#20" type="string"/>
    <ADD line="13" col="27">
      <LITERAL value="#13" type="char"/>
      <LITERAL value="#10" type="char"/>
    </ADD>
  </ADD>
</EXPRESSION>

The modified code of TPasSyntaxTreeBuilder.StringConstSimple is:

procedure TPasSyntaxTreeBuilder.StringConstSimple;
begin
  // Add childnode with value
  FStack.AddChild(sLITERAL).SetAttribute(sVALUE, AnsiDequotedStr(Lexer.Token, ''''));
  // Add type attribute 'char' for ptAsciiChar tokens
  if( Lexer.TokenID = ptAsciiChar ) then
    FStack.Peek.ChildNodes.Last.SetAttribute(sTYPE, 'char');
  inherited;
end;

And the modified code of TPasSyntaxTreeBuilder.StringConst is:

procedure TPasSyntaxTreeBuilder.StringConst;
var
  StrConst: TSyntaxNode;
  Literal: TSyntaxNode;
  Str: string;
  nodelist: array of TSyntaxNode;
  idx: Integer;

  // Some helper functions to handle internal nodelist

  function nodeLastIndex: Integer;
  begin
    result := High( nodelist );
  end;

  function nodeLast: TSyntaxNode;
  begin
    if( Length(nodelist) = 0 ) then
      result := nil
    else
      result := nodelist[nodeLastIndex];
  end;

  procedure nodeLiteralAdd(avalue, atype: String );
  begin
    SetLength( nodelist, Length(nodelist) + 1 );
    nodelist[nodeLastIndex] := TSyntaxNode.Create(sLITERAL);
    nodeLast.SetAttribute(sVALUE, avalue);
    nodeLast.SetAttribute(sTYPE, atype)
  end;

  procedure nodeExtendLastValue( avalue: String );
  begin
    nodeLast.SetAttribute(sVALUE, nodeLast.GetAttribute(sVALUE) + avalue);
  end;

begin
  StrConst := TSyntaxNode.Create('StringConst');
  try
    // Parse constant expression tag-Literals to StrConst
    FStack.Push(StrConst);
    try
      inherited;
    finally
      FStack.Pop;
    end;

    // Reset internal nodelist
    SetLength(nodelist, 0);

    // See through parsed tag-Literals
    for Literal in StrConst.ChildNodes do
    begin
      // String value of current literal ('string' or 'char')
      Str := Literal.GetAttribute(sValue);
      // Found a 'char' (without explicit ADD operator)
      if( Literal.GetAttribute(sTYPE) = 'char' ) then
        nodeLiteralAdd(Str, 'char')
      // Found 'string' literal, but no previous string value to 'join'
      else if( nodeLast = nil ) or ( nodeLast.GetAttribute(sTYPE) <> 'string' ) then
        nodeLiteralAdd(Str, 'string')
      // Found 'string' literal, has previous value to 'join'
      else
        nodeExtendLastValue(Str);
    end;
  finally
    StrConst.Free;
  end;

  // We have to 'join' created internal nodelist with ADD operators
  for idx := Low(nodelist) to nodeLastIndex do
  begin
    FStack.AddChild(nodelist[idx]);
    if( idx < nodeLastIndex ) then
      FStack.AddChild(sADD);
  end;
end;

from delphiast.

But technically it's not an expression...
What if we change this to something like:

<LITERAL type="string" value="'apple'#20#13#10"/>

?
(see quotes)

from delphiast.

Because Delphi has no string 'escaping' (like in C, C# or JavaScript), I think the string literals like this can be generated as expression, for most effective using of the generated AST. This literal equals with:

'apple' + #20 + #13 + #10   // String literal + char literals

Let's think about what if someone uses the generated AST? What will he do with the suggested "'apple'#20#13#10" value? I think that he has to starts to parse it, to generate an expression like above, to get the 'real' value...

from delphiast.