access_expr is partially implemented in #50, but various portions could not be implemented because the lexical tokens were not available.

Prereqs

• #50
• #62
• #63
• #64
• #65
• #66
• #67
• #68
• #69
• #71

Features

When all prereqs are complete, then accessExpression should completely match access_expr.

When I added sigils, I added support for modifiers at the end of the work and forgot to add the support to both the group format and heredoc format for sigils: currently only heredoc supports modifiers.

Group format such as ~r()i should be accepted, but it is not while

~r"""
"""i

is accepted.

Even a simple file like this is completely red and no sensible Syntax highlighting.

defmodule Chat do
  use Application

  # See http://elixir-lang.org/docs/stable/Application.Behaviour.html
  # for more information on OTP Applications
  def start(_type, _args) do
    Chat.Supervisor.start_link
  end
end

The error is something like "ElixirTokenType... or ...[..] expected, but got 'd' "

Support ? char tokens with literal characters and escape sequences as described here https://github.com/elixir-lang/elixir/blob/4b505286ec61b32cb791b79268ae6c58340c71f0/lib/elixir/src/elixir_tokenizer.erl#L166-L221.

Implement no_parens_op_expr to complete match_expr where it uses no_parens_expr.

Prereqs

• matchedExpression (#50)
• noParenthesesExpression (#58)

Features

matched_expr direct-dependencies

• matched_expr -> matched_expr no_parens_op_expr

no_parens_op_expr

This is matched_expr no_parens_op_expr should be rewritten as binary operations with matchedExpression as the left operand and noParenthesesExpression as the right operand. This can be accomplished with either new binary operations or by changing the matchedExpression*Operations from #50 to accept either a matchedExpression or a noParenthesesExpression as the right operand.

• Allow noParenthesesExpression as right operand to matchedExpressionMatchOperation
• Allow noParenthesesExpression as right operand to matchedExpressionAdditionOperation
• Allow noParenthesesExpression as right operand to matchedExpressionMultiplicationOperation
• Allow noParenthesesExpression as right operand to matchedExpressionHatOperation
• Allow noParenthesesExpression as right operand to matchedExpressionTwoOperation
• Allow noParenthesesExpression as right operand to matchedExpressionAndOperation
• Allow noParenthesesExpression as right operand to matchedExpressionOrOperation
• Allow noParenthesesExpression as right operand to matchedExpressionInOperation
• Allow noParenthesesExpression as right operand to matchedExpressionInMatchOperation
• Allow noParenthesesExpression as right operand to matchedExpressionTypeOperation
• Allow noParenthesesExpression as right operand to matchedExpressionPipeOperation
• Allow noParenthesesExpression as right operand to matchedExpressionComparisonOperation
• Allow noParenthesesExpression as right operand to matchedExpressionRelationalOperation
• Allow noParenthesesExpression as right operand to matchedExpressionArrowOperation
• Allow noParenthesesExpression as right operand to matchedExpressionWhenOperation

when handling for no_parens_op_expr has an additional right operand choice in addition to no_parens_expr: call_args_no_parens_kw

• Allow call_args_no_parens_kw as right operand to matcheExpressionWhenOperation

Allow the creation of Elixir projects from the New Project menu. This will include the use of mix to create the new project template. Feature request by @Claudiojlf in #6 (comment). Requested by @prodev42 in #77 and @andreacfm on stackoverflow.com

References

• ignatov/intellij-erlang@28e4f1e
• ignatov/intellij-erlang@b9d2eda

Tasks

• Setup SDK (#79)
• Add moduleType for Elixir Project
• Allow setting of mix new options using mix from Setup SDK.

backet_at_expr is used in access_expr

Prereqs

• #50
• #55

Features

access_expr direct-dependencies

• Add backet_at_expr to accessExpression

bracket_at_expr direct-dependencies

• dot_bracket_identifier
• bracket_arg

dot_bracket_identifier

• bracket_identifier

bracket_identifier

• Lex .[ as BRACKET_IDENTIFIER.

bracket_arg direct-depencies

• Use pre-existing rules from #55.

Handle dot, which needs the rest to be operators (so complete #29 first), ( for anonymous function calls and interpolated function names. (For example, Keyword."get"([a: 1], :a) is valid and the same as Keyword.get([a: 1], :a), so you can use a string or char list the after dot the same way you pass Symbols to send in Ruby.) Finally, there is normal . call syntax. All these forms need to accept multiple newlines and whitespace on either side of the ..

Right Operand

• .<three-token-operator> (#114)
• .<two-token-operator> (#114)
• .<one-token-operator> (#114)
• .(
  • parens_call
    • [ ] block_expr
      • [ ] unmatched_expr (Moved to #116)
        • [ ] expr
    • [ ] max_expr (Can't be in max_expr because use of matchedExpression requires it extending matchedExpression and being a direct child of matchedExpression
      • [ ] access_expr
  • call_args_parens
    • kw
    • call_args_parens_base
      • call_args_parens_expr
• .<string> (Already complete in #105)
• .<char-list> (Already completed in #105)

Call Identifier

• [ ] bracket_identifier
  • [ ] dot_bracket_identifier
  • [ ] bracket_arg
  • [ ] bracket_expr (Moved to #117)
    • [ ] Add to access_expr
  • [ ] bracket_at_expr (Moved to #117)
    • [ ] Add to access_expr
• [ ] paren_identifier (Moved to #115)
  • [ ] dot_paren_identifier
    • [ ] parens_call

no_parens_one_expr is part of matched_expr. The dot_identifier portion of it is currently implemented by matchedExpressionDotOperation and identifierExpression, but the call_args_no_parens_one still needs to be implemented.

I was able to trigger do_identifier in the tokenizer like so:

iex(37)> :elixir_tokenizer.tokenize('one do end', 1, [])
{:ok, 1, [{:do_identifier, 1, :one}, {:do, 1}, {:end, 1}]}

This held into the parser too:

iex(38)> Code.string_to_quoted("one do end")            
{:ok, {:one, [line: 1], [[do: nil]]}}

Prereqs

• #50
• #58

Features

matched_expr direct-dependencies

• matched_expr -> no_parens_one_expr

no_parens_one_expr direct-dependencies

• dot_op_identifier may be subsumed by dot_op_identifier in intellij-elixir if all op_identifiers are just lexed as IDENTIFIER.
• call_args_no_parens_one
• dot_do_identifier

call_args_no_parens_one direct-dependencies

• call_args_no_parens_kw should be implemented by #58
• matched_expr should be implemented by #50
• no_parens_expr should be implemented by #58

Implement as much of no_parens_expr as is possible given the pre-existing translation of elixir_parser.yrl

Prereqs

• matchedExpression (#50)

Features

no_parens_expr direct-dependencies

• Add to expression
• dot_op_identifier without overlapping matchedExpression as that would lead to left-recursion that isn't handled by Grammar Kit's Pratt parsing
• dot_identifer without overlapping matchedExpression as that would lead to left-recursion that isn't handled by Grammar Kit's Pratt parsing.
• call_args_no_parens_many_strict can be transliterated, in which case throw_no_parens_strict needs to be converted to IntelliJ-style error handling using and annotator. Otherwise, this will just be call_args_no_parens_many

call_args_no_parens_many_strict direct-dependencies

Without error handling

• call_args_no_parens_many

With error handling

• call_args_no_parens_kw
• Annotate parenthetical expressions as errors

call_args_no_parens_many

• Disentangle dependency on matchedExpression to not cause unresolvable left-recursion problems.
• call_args_no_parens_kw and 2
• call_args_no_parens_comma_expr
• call_args_no_parens_many -> matched_expr ',' call_args_no_parens_kw
• call_args_no_parens_many -> call_args_no_parens_comma_expr
• call_args_no_parens_many -> call_args_no_parens_comma_expr ',' call_args_no_parens_kw

call_args_no_parens_kw direct-dependencies

• call_args_no_parens_kw_expr

call_args_no_parens_kw_expr direct-dependencies

• call_args_no_parens_expr

call_args_no_parens_expr direct-dependencies

Without error handling

• call_args_no_parens_expr -> matched_expr
• call_args_no_parens_expr -> empty_paren

With error hanlding

• call_args_no_parens_expr -> no_parens_expr : throw_no_parens_many_strict('$1')

call_args_no_parens_comma_expr direct-dependencies

• Disentangle dependency on matchedExpression

Complete matchExpression

• Add as operand to matchedExpressionUnaryOperation
• Add as operand to matchedExpressionAtOperation
• Add as operand to matchedExpressionCaptureOperation
• #59

The current EOL token recognizes \r, \n, and \r\n as EOL. The elixir tokenizer also recognizes ; and EOL escapes. These additional capabilities need to be added.

container_args is used in tuple, used in bit_string. container_args_base is a direct dependency of container_args and container_args_base is used in list

Prereq

• #50
• #58

Features

list direct-dependencies

• list_args

list_args direct-dependencies

• Extract kw as keywords from list rule in Elixir.bnf.
• container_args_base

container_args_base direct-dependencies

• container_expr

container_expr

• Use pre-existing emptyParentheses
• Use pre-existing matchExpression

There is a bug about highlighting comments:

# This comment will to displayed correctly
IO.puts "All your #{inspect tuple} are belong to us"

The last part of the second line #{inspect tuple} are belong to us" will be displayed as a comment.

I dont know the name for the #{...} syntax in strings, but it needs to be handled.

(I know its a early version, but i would like to document my findings.)

The lexer breaks and can't match input if a EOL is encountered while parsing bad characters that don't match {SIGIL_NAME}.

There is already support for base integers, but not for normal decimal integers and floats which parsed here

max_expression is used in access_expr and used in map_expr.

Prereqs

• #50
• #58
• #68

Features

access_expr direct-dependencies

• Add max_expr to access_expr

max_expr direct-dependencies

• parens_call
• call_args_parens
• dot_alias

parens_call direct-dependencies

• dot_paren_identifier
• dot_call_op

dot_paren_identifer direct dependencies

• paren_identifier

dot_call_op

• Lex {DOT_OPERATOR} / {OPENING_PARENTHESES} as DOT_CALL_OPERATOR.

call_args_parens

• Use emptyParentheses
• Use noParenthesesExpression from #58.
• Use keywords from #68
• call_args_parens_base

call_args_parens_base

• Use containerExpression from #68

The current version of New > Elixir File, as so nicely provided by @abaire, doesn't differentiate between the File name and the Module name, so if you enter a Module name of Foo, the file is named Foo.ex, but Elixir convention is for the file name to be lowercase and underscored, so ExUnit Module name becomes ex_unit.ex. I tried debugging the Action and I'm not sure if using the CreateFileFromTemplateAction allows us to have a different File name and Module name since there is only one NAME attribute passed into the template. I may be possible to look at how the Package and Class creation actions work for Java and see if its possible to set the Module name as the PACKAGE_NAME or CLASS_NAME attribute and have the file name derived from that for the NAME attribute.

Empty parentheses are one form of valid expression. It will also entail adding support for open parenthesis, (, and close parenthesis, ) that will be needed for containers, #37.

Implementing commas as operators in #29 does not makes sense as unlike other operators, commas should parse a list of expressions (that don't themselves contain commas) instead of nested binary operations. elixir_parser.yrl is able to get away with parsing commas as binary expressions because the yrl format allows the elements of a given rule to be concatenated into a flattened list. To do this in the bnf format for Grammar Kit I need to actually write a list like expression, which means separating operations into a different rule than comma so that comma only contains comma-less direct children to prevent left-recursion.

The map association operator, => is not used anywhere else, so it can be parsed by itself

map is used in access_expr

Prereqs

• #50
• #71

Features

access_expr direct-dependencies

• Add map to accessExpression

map direct-dependencies

• Lex % instead of %{} as MAP_OPERATOR. It will be used for both map_op and struct_op.
• map_args
• map_expr

map_args direct-dependencies

• Lex { as OPENING_CURLY
• Lex } as OPENING_CURLY
• map_close
• assoc_update
• assoc_update_kw

map_close direct-dependencies

• Extract kw ((keywordPair (COMMA EOL* keywordPair)* COMMA?)?) from list in Elixir.bnf
• assoc
• assoc_base

assoc direct-dependencies

• assoc_base

assoc_base

• assoc_expr

assoc_expr

• Use pre-existing rules

assoc_update direct-dependencies

assoc_update_kw

• Use existing rules.

map_expr

• Use existing rules

Be able to recovery from invalid digits in numbers so that parsing in pygment example doesn't stop at first invalid number example.

• Detect obsolete 0B and parse without PsiErrorElement
• Detect obsolete 0X and parse without PsiErrorElement
• Detect invalid binary digits for 0b and 0B
• Detect invalid digits for decimal numbers
• Detect invalid octal digits for 0o.
• Detect invalid hexadecimal digits for 0x and 0X

eoe or End Of Expression is EOL or ;. Right now EOL includes ; in Elixir.flex, so that does not match the usage in Elixir's parser

backet_expr is used in access_expr

Prereqs

• #50
• #55

Features

access_expr direct-dependencies

• Add backet_expr to accessExpression

Any token that starts with a lowercase letter or underscore (_) and does not match any other token type is an identifier

Lex regular keywords as distinct tokens from IDENTIFIER.

Elixir.flex

• Lex fn as FN
• Lex end as END
• Lex true as TRUE
• Lex false as FALSE
• Lex nil as NIL

Elixir.bnf

• Add TRUE to accessExpression
• Add FALSE to accessExpression
• Add NIL to accessExpression

The bit string syntax is used to declare and match binaries.

dot_op is used for the call syntax. In Elixir's implementation, the parser matches . directly, but in IntelliJ, I'll need to parse it in the lexer, Elixir.flex, as DOT_OPERATOR and then use it in Elixir.bnf for the parser.

tuple is used in access_expr

Prereqs

• #50
• #68

Features

access_expr

• Add tuple to access_expr

tuple

• CLOSING_CURLY and OPENING_CURLY from #66.
• container_args

container_args direct-dependencies

• kw from #68
• container_args_base from #68

Comments currently consume the EOL to make combining multiline comments together easier, but this leads to trailing comments to eat the EOL, which means the required EOL between expressions is not there leading to invalid parses.

0x1 # does not consume EOL to separate 0x1 and 0x2
0x2 # does not have EOL at EOF

Expected:

Elixir File(0,81)
  PsiElement(ElixirTokenType.NUMBER)('0x1')(0,3)
  PsiWhiteSpace(' ')(3,4)
  PsiComment(ElixirTokenType.COMMENT)('# does not consume EOL to separate 0x1 and 0x2')(4,50)
  PsiElement(ElixirTokenType.EOL)('\n')(50,51)
  PsiElement(ElixirTokenType.NUMBER)('0x2')(51,54)
  PsiWhiteSpace(' ')(54,55)
  PsiComment(ElixirTokenType.COMMENT)('# does not have EOL at EOF')(55,81)

Actual:

Elixir File(0,81)
  PsiElement(ElixirTokenType.NUMBER)('0x1')(0,3)
  PsiWhiteSpace(' ')(3,4)
  PsiComment(ElixirTokenType.COMMENT)('# does not consume EOL to separate 0x1 and 0x2\n')(4,51)
  PsiErrorElement:ElixirTokenType.EOL expected, got '0x2'(51,54)
    PsiElement(ElixirTokenType.NUMBER)('0x2')(51,54)
  PsiWhiteSpace(' ')(54,55)
  PsiComment(ElixirTokenType.COMMENT)('# does not have EOL at EOF')(55,81)

Recognize aliases as module names. Note: Not all tokens with leading capital letters are aliases/Module names if they're followed by : (<colon><space>) then they should just be parsed as the atom keyword identifier for a Keyword list or map.

This builds on top of #19.

If a Module name has multiple levels, such as ExUnit.AssertionError, then the file should have a nested path of ex_unit/assertion_error.ex. It may be possible to determine how to do this by treating the ExUnit has a PACKAGE_NAME and AssertionError as a CLASS_NAME and seeing how the Java Class action work in the IDEA Community Edition source.

Blank lines, such between comments are erroneously marked as bad characters because only single EOL are expected to separate expressions. Blank lines should be treated as whitespace, so they aren't consumed by the parser.

The current set of VALID_ESCAPE_SEQUENCE

VALID_ESCAPE_SEQUENCE = {ESCAPED_DOUBLE_QUOTES} |
                        {ESCAPED_SINGLE_QUOTE} |
                        {ESCAPED_INTERPOLATION_START}

is too limited and misses escapes of sigil terminators.

Not only should the sigil terminators be escapable, but VALID_ESCAPE_SEQUENCE needs to be updated to cover all escapes that Elixir's lexer handles.

As a next smaller step, visual support for atoms might be cool.

I think i might not be abled to help developing, but testing for sure :)

Hi, I have installed the plugin successfully in IntellJ, however it doesn't not let me create elixir modules. What do I have to do to enable this?

At the lexer level \x is valid because it allows for uniform handling of character escapes, but iex raised an argument error, so \x must report an error

in is keyword for a binary operator used as an "include" operator as in

x = 1
x in [1, 2, 3] # true

kw is a prereq for list_args which is a prereq for list, but list_args and kw don't make sense as independent expressions, so to implement kw, implement list_args -> kw and list -> open_bracket list_args close_bracket to form a minimal list. list should be parsed as a value in Elixir.bnf.

• Empty Lists
• Keyword Pairs
• Disallow Space Between Alias and :
• Disallow Space Between Identifier and :
• Disallow Space Between quote and :

bit_string is used in access_expr.

Prereqs

• #50
• #68

Features

access_expr

• Add bit_string to access_expr

bit_string

• Lex << as OPENING_BIT
• Lex >> as CLOSING_BIT
• container_args from #66

Part of the atom lexing overlaps with the keyword identifier when it uses the JSON syntax. <atom><colon> should be parsed as <keyword_identifier>.

stab is used in access_expr (and used in do_block)

Prereqs

• #50
• #64

Features

access_expr direct-dependencies

• Add fn_eoe stab end_eoe
• Add open_paren stab close_paren
• Add open_paren stab ';' close_paren
• Add open_paren ';' stab ';' close_paren

stab

• stab_expr

stab_expr direct-dependencies

• stab_maybe_expr
• call_args_no_parens_all
• stab_parens_many

stab_maybe_expr direct-dependencies

• expression?

call_args_no_parens_all direct-dependencies

• Use pre-existing rules

stab_parens_many

• Use pre-existing rules.

The pygment example has escape sequences with too many digits. These extra digits are currently lexed as a separate decimal number immediately adjacent to the valid escape sequences. This problem can be generalized to expressions being adjacent to each other with missing end-of-expressions. When end-of-expressions are missing, then the parser should recover. There should also be quick fixes to insert end-of-expressions.

Char Tokens make use of Char Escapes, so it makes sense to support Char Escapes prior to Char Tokens. This feature should take care of #14.

Matched expressions are calls with parentheses. They are the second type of expression after empty parentheses, #48.

Prerequirements

• In Operators
• Dot Operators

Todo

• Fix left-recursion in dotAlias
• Fix left-recursion in dotIdentifier

To Be Implemented Separately

• max_expr -> parens_call call_args_parens
• max_expr -> parens_call call_args_parens call_args_parens
• max_expr -> dot_alias

Hi guys,

Here is the screenshot of what I had after installing new version.

http://cl.ly/image/0R1q0g1H0a29

The elixir tokenizer contains a 'Stand-alone token' for "...", which needs to be parsed as an identifier.

Handle 3 token, 2 token, and 1 token operators.

Special handling is needed to distinguish % and %{}

Use the pygment's example_elixir.ex that @alco developed to ensure feature parity with pygment's highligher.

I tried opening this file, but PHPStorm seem to freeze:

https://github.com/marcioj/elixir_blog/blob/master/lib/blog.ex

No Error message on the console :(