roger-luo / expronicon.jl Goto Github PK

View Code? Open in Web Editor NEW

67.0 3.0 6.0 3.22 MB

Collective tools for metaprogramming on Julia Expr

Home Page: https://expronicon.rogerluo.dev/

License: MIT License

Julia 99.43% Shell 0.57%

metaprogramming julia julia-language

expronicon.jl's Introduction

Expronicon

Collective tools for metaprogramming on Julia Expr. See also ExproniconLite

Installation

Expronicon is a Julia Language package. To install Expronicon, please open Julia's interactive session (known as REPL) and press ] key in the REPL to use the package mode, then type the following command

pkg> add Expronicon

License

MIT License

expronicon.jl's People

Contributors

Stargazers

Watchers

Forkers

thautwarm domluna 0x0f0f0f thofma pitmonticone walterl

expronicon.jl's Issues

LineInfo still prefixes kwdef after `prettify`

ref #4

julia> types[1]
quote
    #= /home/domluna/.julia/dev/GraphQL/src/jltype.jl:177 =#
    #= /home/domluna/.julia/dev/GraphQL/src/jltype.jl:177 =# Base.@kwdef mutable struct A
            #= /home/domluna/.julia/dev/GraphQL/src/jltype.jl:178 =#
            field1::Union{C, Missing, Nothing} = nothing
            field2::Union{Int32, Missing, Nothing} = nothing
        end
    #= /home/domluna/.julia/dev/GraphQL/src/jltype.jl:181 =#
    StructTypes.StructType(::Type{A}) = begin
            #= /home/domluna/.julia/dev/GraphQL/src/jltype.jl:181 =#
            StructTypes.Mutable()
        end
    #= /home/domluna/.julia/dev/GraphQL/src/jltype.jl:182 =#
    StructTypes.omitempties(::Type{A}) = begin
            #= /home/domluna/.julia/dev/GraphQL/src/jltype.jl:182 =#
            true
        end
end

julia> Expronicon.prettify(types[1])
quote
    #= /home/domluna/.julia/dev/GraphQL/src/jltype.jl:177 =# Base.@kwdef mutable struct A
            field1::Union{C, Missing, Nothing} = nothing
            field2::Union{Int32, Missing, Nothing} = nothing
        end
    StructTypes.StructType(::Type{A}) = begin
            StructTypes.Mutable()
        end
    StructTypes.omitempties(::Type{A}) = begin
            true
        end
end

(GraphQL) pkg> st
     Project GraphQL v0.1.0
      Status `~/.julia/dev/GraphQL/Project.toml`
  [6b7a57c9] Expronicon v0.5.5
  [cd3eb016] HTTP v0.9.5
  [d8e11817] MLStyle v0.4.9
  [83ef0002] RBNF v0.2.2
  [856f2bd8] StructTypes v1.5.2

Error parsing function with return type annotation

julia> ex2 = @expr function f(x; a=10)::Int
           return x
       end
:(function f(x; a = 10)::Int
      #= REPL[148]:1 =#
      #= REPL[148]:2 =#
      return x
  end)

julia> JLFunction(ex2)
ERROR: expect function head expr expression, got f(x; a = 10)::Int.
Stacktrace:
 [1] anlys_error(expect::String, got::Expr)
   @ Expronicon ~/.julia/packages/Expronicon/RauF7/src/analysis.jl:89
 [2] split_function_head(ex::Expr)
   @ Expronicon ~/.julia/packages/Expronicon/RauF7/src/analysis.jl:391
 [3] JLFunction(ex::Expr)
   @ Expronicon ~/.julia/packages/Expronicon/RauF7/src/analysis.jl:504
 [4] top-level scope
   @ REPL[149]:1

Errors from `@adt public`

In the latest v0.9.1, the public keyword doesn't work.

julia> using Expronicon.ADT: @adt

julia> @adt public Foo begin
           A(::Int)
           B(::Float64)
       end
ERROR: LoadError: MethodError: no method matching var"@adt"(::LineNumberNode, ::Module, ::Symbol, ::Symbol, ::Expr)
Closest candidates are:
  var"@adt"(::LineNumberNode, ::Module, ::Any, ::Any) at ~/.julia/dev/Expronicon/src/adt/emit.jl:227
in expression starting at REPL[5]:1

ADT cannot be navigated in VS Code

I actually cannot think why this happens.

If I define a @config (from Configurations.jl) in a file, I can include the file and navigate to the definition from anywhere.
However, this does not happen with ADTs generated with Expronicon.ADT. Any idea why?

@Roger-luo

documentation for variant_show_inline

Support isbits/isbitstype for variants?

using Expronicon.ADT: @adt


julia> @adt D begin
           D1
           D2(::Int)
       end

julia> isbits(D.D1)
ERROR: ArgumentError: invalid variant type

The issue comes from:

julia> @less isbits(D.D1)
isbits(@nospecialize x) = (@_pure_meta; typeof(x).flags & 0x8 == 0x8)

# this works:
# getfield(D, :flags) & 0x8

Julia base might not consider much about overloading getproperty for DataTypes (this is also generally
not that good). Incompatibility like above could happen unexpectedly. Maybe patch this in Expronicon.jl?

TagBot trigger issue

This issue is used to trigger TagBot; feel free to unsubscribe.

If you haven't already, you should update your TagBot.yml to include issue comment triggers.
Please see this post on Discourse for instructions and more details.

If you'd like for me to do this for you, comment TagBot fix on this issue.
I'll open a PR within a few hours, please be patient!

Error removing line annotations

julia> typs[1]
quote
    #= /home/domluna/.julia/dev/GraphQL/src/jltype.jl:166 =#
    #= /home/domluna/.julia/dev/GraphQL/src/jltype.jl:166 =# Base.@kwdef mutable struct D
            #= /home/domluna/.julia/dev/GraphQL/src/jltype.jl:167 =#
            field1::Union{ID, Missing, Nothing} = nothing
        end
    #= /home/domluna/.julia/dev/GraphQL/src/jltype.jl:170 =#
    StructTypes.StructType(::Type{D}) = begin
            #= /home/domluna/.julia/dev/GraphQL/src/jltype.jl:170 =#
            StructTypes.Mutable()
        end
    #= /home/domluna/.julia/dev/GraphQL/src/jltype.jl:171 =#
    StructTypes.omitempties(::Type{D}) = begin
            #= /home/domluna/.julia/dev/GraphQL/src/jltype.jl:171 =#
            true
        end
end

julia> Expronicon.Printings.println(stdout, Expronicon.Transform.prettify(typs[1]))
begin
    Base.@kwdef
    StructTypes.StructType(::Type{D}) = begin
            StructTypes.Mutable()
        end
    StructTypes.omitempties(::Type{D}) = begin
            true
        end
end

The struct def is removed from the output

Using julia keywords as type fields

This is a corner case but it can come up when you're generating code from a language that's end not Julia, such as GraphQL and a field is called "end".

julia> JLKwStruct(;name=:ABC, fields = [JLKwField(name=:end)])
begin
    #= /home/dom/.julia/packages/Expronicon/trhCs/src/codegen.jl:101 =#
    struct ABC
        end
    end
    #= /home/dom/.julia/packages/Expronicon/trhCs/src/codegen.jl:102 =#
    begin
        #= /home/dom/.julia/packages/Expronicon/trhCs/src/codegen.jl:156 =#
        function ABC(; end)
            ABC(end)
        end
        #= /home/dom/.julia/packages/Expronicon/trhCs/src/codegen.jl:157 =#
        nothing
    end
end

This will be unable to be parsed but it can work if we replace uses of end with the following:

julia> JLKwStruct(;name=:ABC, fields = [JLKwField(name=:end)])
begin
    #= /home/dom/.julia/packages/Expronicon/trhCs/src/codegen.jl:101 =#
    struct ABC
        var"end"
    end
    #= /home/dom/.julia/packages/Expronicon/trhCs/src/codegen.jl:102 =#
    begin
        #= /home/dom/.julia/packages/Expronicon/trhCs/src/codegen.jl:156 =#
        function ABC(; var"end")
            ABC(var"end")
        end
        #= /home/dom/.julia/packages/Expronicon/trhCs/src/codegen.jl:157 =#
        nothing
    end
end

so I guess the idea is to detect Julia keywords and wrap them with @var_str.

It doesn't look like this can be done on input but maybe during codegen?

scope rule for ADT

currently we expose all the variant name in the scope, which is not a good idea, we should instead support

@adt Name begin
   A
   B
end

Name.A
Name.B

using Name: A, B

I'm not sure if we can emulate this by having a fake module, but probably no possible without compiler support because it seems impossible to hack using here.

oops?

julia> @adt Food begin
               Apple
               Orange
               Banana
           end
ERROR: LoadError: InexactError: check_top_bit(UInt64, -1)
Stacktrace:
  [1] throw_inexacterror(f::Symbol, #unused#::Type{UInt64}, val::Int64)    @ Core .\boot.jl:635
  [2] check_top_bit
    @ .\boot.jl:649 [inlined]
  [3] toUInt64
    @ .\boot.jl:760 [inlined]
  [4] UInt64
    @ .\boot.jl:790 [inlined]
  [5] convert
    @ .\number.jl:7 [inlined]
  [6] cconvert
    @ .\essentials.jl:492 [inlined]
  [7] sizehint!
    @ .\array.jl:1285 [inlined]
  [8] scan_fields!(info::EmitInfo, def::ADTTypeDef)
    @ Expronicon.ADT C:\Users\MrJSa\.julia\packages\Expronicon\TSFIu\src\adt\emit.jl:123
  [9] EmitInfo(def::ADTTypeDef)
    @ Expronicon.ADT C:\Users\MrJSa\.julia\packages\Expronicon\TSFIu\src\adt\emit.jl:180
 [10] emit(def::ADTTypeDef)
    @ Expronicon.ADT C:\Users\MrJSa\.julia\packages\Expronicon\TSFIu\src\adt\emit.jl:240
 [11] var"@adt"(__source__::LineNumberNode, __module__::Module, head::Any, body::Any)
    @ Expronicon.ADT C:\Users\MrJSa\.julia\packages\Expronicon\TSFIu\src\adt\emit.jl:230
in expression starting at REPL[4]:1

function keyword arguments default values are not generated

julia> JLFunction(;name="foo", kwargs=[JLKwField(;name=:arg1, default=10)])^C

julia> kw = JLKwField(;name=:arg1, default=10, type=:Int)
arg1::Int

julia> f = JLFunction(;name="foo", kwargs=[kw])
function "foo"(; arg1::Int)
end

julia> codegen_ast(a)
:(arg1::Int)

julia> codegen_ast(f)
:(function ("foo")(; arg1::Int)
  end)

This should be arg1::Int=10. Changing 10 to a Symbol has the same result.

change `@use` to non-const binding and create `@const_use`

[Feature Request] Support subtyping for ADT

Is it possible to support

abstract type Language end
@adt public Foo <: Language begin
    ...
end

request more examples of use

As you have time, or some inspiration, additional examples and different kinds of use would go a long way to getting people more comfortable with this.

potential performance issue caused by ADT namespace

from here:

JuliaLang/julia#48883 (reply in thread)

I don't quite understand why, but this seems to be an issue for us overloading getproperty on Type. I think either we need to find a way to use a different syntax or we have to bare the cause of this.

JLFunction does not support x -> x

Using [6b7a57c9] Expronicon v0.10.3 on Julia 1.9.0, JLFunction cannot construct a function from an anonymous function with only one input. Somewhat strangely, JLFunction(:((x, y) -> x)) does work.

julia> using Expronicon

julia> JLFunction(:(x -> x))
ERROR: MethodError: no method matching split_function_head(::Symbol; source::Nothing)

Closest candidates are:
  split_function_head(::Expr; source)
   @ Expronicon ~/.julia/packages/Expronicon/7EBrJ/src/analysis/split.jl:43

Stacktrace:
 [1] JLFunction(ex::Expr; source::Nothing)
   @ Expronicon ~/.julia/packages/Expronicon/7EBrJ/src/analysis/cons.jl:30
 [2] JLFunction(ex::Expr)
   @ Expronicon ~/.julia/packages/Expronicon/7EBrJ/src/analysis/cons.jl:16
 [3] top-level scope
   @ REPL[2]:1

This probably just needs split_function_head to have a new case for symbol, maybe.

[Feature Request] Auto generate `hash` and `(==)` for ADT.

I think this would greatly decrease the amount of boiler plate code that is needed to start using an ADT.

As long as the semantics tree is inferable from the ADT it should be possible to auto-generate this functionality. I guess the question is how to export those functions but perhaps you can put the implementations in the ADT namespace like the variant types. Then later all one would have to do is wrap them with Base.hash(x::MyADT, h::UInt) = MyADT.hash(x, h) and similar for ==

What do you think?

mention functions/types not available in ExproniconLite

there are some features not available in ExproniconLite at the moment, we need to mention that in docs.

generate ExproniconLite

currently I manually write if else in ExproniconLite to get rid of MLStyle/MatchCore, but this is not necessary since the generated code from @match/@switch is actually dependency free.

In principle we just need to implement a print_expr function to print Julia Expr to a file after macroexpand the MLStyle macros. (the default Julia show for Expr does not guarantee its printing can be unparsed)

request for `JLNamedTuple <: JLExpr`

I am as yet too unfamiliar with this sort of metaprogramming to offer a PR.

Wherever I notice special treatment of Structs and that flexibility has not extended to NamedTuples or where Structs benefit from dedicated dispatch specialization not yet supported for NamedTuples, there is an opportunity to ask for playing-field leveling. Usually this is not difficult for the person who has written the code in support of Structs, as there are similarities in structural design.

switch to enum_matcher

thautwarm/MLStyle.jl#155 (comment)

Function to parse a whole file in structured form.

In 0.7.3 there was Expronicon.parse_file. Now it has been moved in Expronicon.ZhanKai. What is the purpose of that module?

[Feature request] generating BNF from ADTs

It will be very useful if we can generate BNF from user-defined ADTs automatically.

Macro to import all variant names except the listed.

Typically there should only be a small number of name collisions when defining multiple ADT's hence a better semantic for the import macros would be to import all except a list which you specify.

Take the Scalar and Waveform ADT's I am using.

@adt ScalarLang begin
    Scalar(::RealLang)
    Negative(::ScalarLang)

    struct Default
        var::RealLang
        value::Float64
    end

    struct Reduce
        head::Symbol
        literal::Float64
        args::Dict{ScalarLang,Int}
    end

    struct Slice
        duration::ScalarLang
        Interval::ScalarLang
    end

    struct Interval
        first::ScalarLang
        last::ScalarLang
    end
end


@adt WaveformLang begin
    Negative(::WaveformLang) # Always try to move outside of over operations
    Append(::Vector{WaveformLang})

    # throw error when composing with this variant type
    struct Pad
        waveform::WaveformLang
        alignment::AlignmentOption
        value::PaddingValue
    end

    struct Name
        waveform::WaveformLang
        name::String
    end

    struct Add
        lhs::WaveformLang
        rhs::WaveformLang
    end

    struct Scale
        waveform::WaveformLang
        scale::ScalarLang
    end

    struct Slice
        waveform::WaveformLang
        interval::ScalarLang
    end

    struct Smooth
        waveform::WaveformLang
        kernel::KernelType
    end

    struct Instruction
        shape::ShapeType
        duration::ScalarLang
    end
end

The only collisions I have are Slice and Negative Hence a semantic like:

@use_all_except WaveformLang: Slice, Negative
@use_all_except ScalarLang: Slice, Negative

Support ADT type in MLStyle

for the newly added ADT syntax

https://github.com/Roger-luo/Expronicon.jl/blob/master/src/adt.jl

one thing current is not consistent is the following pattern is not supported

@adt begin
   @type struct MyADT end
   struct TypeA
      name::String
   end
   
   struct TypeB
      age::Int
   end
end


@match MyADT begin
   ::TypeA => 1
   ::TypeB => 2
   _ => nothing
end

This is mainly because TypeA is currently a fake Julia type, and thus this hierarchy is not compatible with Julia's own type system, we need either change the syntax of ADT or make MLStyle
work for this pattern.

printing missing parathesis

ex = :(var"##step_1_l#259" = 1 << (var"##plain_locs#258"[1] - 1))
print_expr(ex)

BUG: Incorrect anonymous functions are still parsed

julia> JLFunction(:(foo(bar) -> x))
foo(bar) -> x

julia> JLFunction(:(foo(bar) -> x)) |> dump
JLFunction
  head: Symbol ->
  name: Symbol foo
  args: Array{Any}((1,))
    1: Symbol bar
  kwargs: Nothing nothing
  rettype: Nothing nothing
  generated: Bool false
  whereparams: Nothing nothing
  body: Expr
    head: Symbol block
    args: Array{Any}((2,))
      1: LineNumberNode
        line: Int64 1
        file: Symbol REPL[10]
      2: Symbol x
  line: Nothing nothing
  doc: Nothing nothing

While

julia> foo(bar) -> x
ERROR: syntax: "foo(bar)" is not a valid function argument name around REPL[1]:1
Stacktrace:
[1] top-level scope
 @ REPL[1]:1

cc @walterl