Keep inline and multi-line comments about fantomas HOT 11 CLOSED

I was going to say given an AST that is already type checked is it not possible to simply re-lex the input tokens given known formatting rules. But that would only give a canonical version of each line, and wouldn't take into account line separated tokens.

from fantomas.

On the google group Eric asks https://groups.google.com/forum/?hl=en&fromgroups#!topic/fsharp-opensource/fmPCrZLSQ4s:

Will the process you mention for reuniting the comments from the augmented and un-augmented token streams work for the compiler directives. That is still the other show stopper that needs to be addressed. Once past those problems, the next problem becomes one of speed. When last I looked a week ago, the process did one compilation step to ensure valid input, then anther compilation step to ensure valid output. This puts the process up to a few seconds for even a simple reformat and requires that code be valid. If this were to be put into an IDE such as VS, is there a way to identify when the code has no errors so as to skip the compilation steps.

Answers:

• yes, it deals with compiler directives, they are included in the augmented token stream
• compiler speed won't be a problem for VS/MD/XS since the code will be incorporated directly in the add-on and there will be no startup overhead.

from fantomas.

The F# lexer can already produce token streams that include all skipped tokens: // comments, (* *) comments, #if, #else etc., with ranges attached.

@dsyme I'm aware of that. I think it's the technique Tomas uses to augment F# source code with colorized tokens without changing formatting in https://github.com/tpetricek/FSharp.Formatting . Can you give a pointer on F# compiler source code to obtain these two token streams?

However, it should be possible to match up the non-augmented and augmented token streams in a post-processing step. That is, as text is produced in the output (based on formatting the AST from the non-augmented token steam), it should be possible to look in the augmented token stream for corresponding comments and other skipped text to emit. This means running two token streams: one non-augmented (to produce the AST) and one augmented.

I'm not sure how we can match up these two token streams. Comments can occur anywhere in the code, right now I only use the parser and walk through the ASTs. I assume that I can only make use of range information in order to do matching up.

In any case, I believe the correct technique is to use a combination of the the non-augmented formatted text (from the AST) and the augmented token stream to restore comments into the output. From talking to people in the Visual Studio team, this is (roughly speaking) how other compilers do it (though it may be worth checking with the C# commpiler in mono develop, for example).

I will also check scalariform, ocp-indent and PythonTidy for more insights on similar issues.

from fantomas.

I think this is a key thing to get resolved, I wouldn't want to integrate fully with MD/XS until then.

from fantomas.

@7sharp9 I agree this is a critical issue.

Do you have any comment on how to solve the problem?

from fantomas.

It would be good to raise this issue with the working group to try and get traction to resolve this issue and also get the feature into MonoDevelop. You should get an email from the Core Engineering group pretty soon.

from fantomas.

@7sharp9 Yes, I will do so. Thanks for the suggestion.

from fantomas.

@dsyme Did you have any more thoughts about an easy implementation for this? Im thinking that this is the start of documentation around some of the compiler internals and how to use them effectively in tools.

from fantomas.

@7sharp9 It has been resolved. See https://github.com/dungpa/fantomas/blob/master/src/Fantomas/TokenMatcher.fs for details.

It needs some improvements but current techniques are satisfactory.

from fantomas.

Excellent!

from fantomas.

Handling comments is in. If there are related bugs, they should be raised in separate issues.

from fantomas.