dave / dst Goto Github PK

This is problematic because only *ast.File stores comments. If we convert a dst.Expr to an ast.Expr, there's nowhere to put the comments so they will be lost. Workarounds? For Decl perhaps we could create an empty File with one Decl, but what for Stmt and Expr?

The docs should be easier to find. Would you accept to add a badge or a link in the README?

I'm trying to load a package and walk the resulting tree. I'm noticing that all of the dst.Ident Path values are set to an empty string if the type is in the package being loaded but the Path values are set correctly if the type refers to another package. I pasted a simple program below that loads the net/http package and finds the CookieJar interface for which it prints an identity of just CookieJar (I expected net/http.CookieJar). It also prints the identity of the type of the first parameter to the SetCookies method as net/url.URL.

Is there some way to have the path info filled in?

package main

import (
	"github.com/dave/dst"
	"github.com/dave/dst/decorator"
	"golang.org/x/tools/go/packages"
)

func main() {
	pkgs, err := decorator.Load(&packages.Config{
		Mode: packages.NeedName |
			packages.NeedFiles |
			packages.NeedCompiledGoFiles |
			packages.NeedImports |
			packages.NeedTypes |
			packages.NeedTypesSizes |
			packages.NeedSyntax |
			packages.NeedTypesInfo,
	}, "net/http")
	if err != nil {
		panic(err)
	}

	for _, pkg := range pkgs {
		for _, file := range pkg.Syntax {
			for _, decl := range file.Decls {
				if genDecl, ok := decl.(*dst.GenDecl); ok {
					for _, spec := range genDecl.Specs {
						if typeSpec, ok := spec.(*dst.TypeSpec); ok {
							if typeSpec.Name.Name == "CookieJar" {
								println("CookieJar ident:", typeSpec.Name.String())
								for _, method := range typeSpec.Type.(*dst.InterfaceType).Methods.List {
									if method.Names[0].Name == "SetCookies" {
										param1Type := method.Type.(*dst.FuncType).Params.List[0].Type.(*dst.StarExpr)
										println("  Param1 StarExpr X ident:", param1Type.X.(*dst.Ident).String())
									}
								}
							}
						}
					}
				}
			}
		}
	}
}

Output:

CookieJar ident: CookieJar
  Param1 StarExpr X ident: net/url.URL

What do we do with the current Doc (Field, File, FuncDecl, GenDecl, ImportSpec, TypeSpec, ValueSpec) and Comment (Field, ImportSpec, TypeSpec, ValueSpec) fields?

Is there any alternative to go/format's format.Node function? I couldn't find any. Any help is greatly appreciated, thank you!

Thanks for starting this package, the problem it solves is one I encountered a very long time ago!

I just started off with a very simple test:

package b

func B() {
	println("B")
}

package a

import "github.com/pwaller/example/testdata/a/b"

func A() {
	b.B()
}

With the following program I get unexpected output:

package main

import (
	"golang.org/x/tools/go/packages"

	"github.com/dave/dst/decorator"
)

func main() {
	pkgs, err := decorator.Load(&packages.Config{Mode: packages.LoadAllSyntax}, ".")
	if err != nil {
		panic(err)
	}

	for _, p := range pkgs {
		for _, s := range p.Syntax {
			decorator.Print(s)
		}
	}
}

Here is the output:

package a

import "github.com/pwaller/example/testdata/a/b"

func A() {
	B()
}

I expected that I should see a call to b.B(), not B(), so it seems the package qualifier is lost.

This is using github.com/dave/dst v0.23.0.

I'm a bit surprised by the bug, I would have thought this case should work - have I done something wrong? Thanks!

Hi dave~

package main

// Typo is blabla...
// more comments here, lines not fixed...
// ...
// finally, comments are done...
type Demo struct {
}
func main() {
	println(a, b, c)
}

In the above example, does dst support explicitly change the comment attached to Demo structure ? Like change the word Typo to Demo.

I'm trying to generate some code using dst package.
Part of the code involves multi-line string literal.
I'd like to keep the code prettier by propagating the indentation level into the string literal

... // node :=
&dst.CallExpr{
	Fun: &dst.SelectorExpr{
		X:   &dst.Ident{Name: "mypackage"},
		Sel: &dst.Ident{Name: "MyFunc"},
	},
	Args: []dst.Expr{
		&dst.BasicLit{
			Kind:  token.STRING,
			Value: fmt.Sprintf("`\n%v\n`", "multi\nline\nliteral"),
		},
	},
},

The generated code will look like

func GeneratedFunc() {
	node := mypackage.MyFunc(`
multi
line
literal
`) // <-- This non-indent looks rather bothering
}

Is it possible that I can make the string literal indent aligned with the caller, or somehow retrieve the 'indentation level' via the dst package, so that I can manually Tweak the literal? e.g.

func GeneratedFunc() {
	node := mypackage.MyFunc(`
		multi
		line
		literal
		`)
}

Hello,

This might be a real edge case, but just wanting to share.
When for example, missing a package main you will get a panic:
panic: runtime error: invalid memory address or nil pointer dereference

func TestDecorator(t *testing.T) {
	code := `import "github.com/MyPackage/Name/package1"


func main() {
	fmt.Println("Hello!")
}`
	decorator.Parse(code)
}

I think earlier versions did not fail this test, and feel like it should gracefully return an error.

Hi! I'm very new to your project, and I'm still learning on it.
When I use Load function from your project, it only allows packages.LoadAllSyntax or packages.LoadSyntax which is deprecated in ast.
https://pkg.go.dev/golang.org/x/tools/go/packages#pkg-constants

Adding or removing nodes that contain Qualified Imports (SelectorExpr) from the tree can leave the import block out of sync. There are several problems:

1. The mapping between package path and package name is undefined until type checking is performed.
2. Identifiers aren't portable - e.g. if a node is copied from one file to another, the package name may change if the import blocks define different aliases for the path.
3. There's no way to detect that a node has been added or removed from the tree, so the imports block can't be updated as the tree is modified.

Do we make dst more opinionated? Or create another package to handle this? I think this is a common enough problem that dst should handle it internally.

Solutions:

Ident.Path

Perhaps add Path string to dst.Ident. Decorator will convert all qualified ident ast.SelectorExpr to dst.Ident with Path set to the the package path. Restorer will convert idents that need a selector into ast.SelectorExpr. All dst.Ident will have Path set - even local? That would ensure nodes are portable - e.g. could be copied to a new package. But would this be the preferred behaviour? Perhaps not - copying nodes that expect local variables would probably expect the local variables to stay local in the new package. Perhaps local idents simply use blank Path.

Resolver

For this to work, both Decorator and Restorer will have to be able to reliably determine the mapping between package name and package path. Perhaps we can have a Resolver interface that could be plugged in as the developer prefers the context.

• SimpleResolver might use a developer provided map.
• FilesystemResolver might query the local filesystem using a specified base directory.
• LoaderResolver might use the loader package.

The restorer will have to scan the tree and reconstruct the imports block, making sure any existing decorations aren't broken.

Thanks for the very useful dst package, I have used it a few times for rewriting code with great success! :)

Now that the Go language proposal golang/go#43651 has been accepted, I was wondering what your thoughts are on updating the dst package when generics land?

Do you plan to make the required changes so that dst can read, modify and write code that uses generics? (I’m assuming changes will be required.)

Thanks,

Nodes can't be re-used in the tree because restoreNode both restores the node (creates an ast node) and also applies the node (creates the comments, increments the cursor). When a duplicate node is detected, it exits early so the apply is skipped.

We should separate these so restoring the node can be performed before applying. That way if a duplicate node is detected, the restore function will exit early, but the apply function will run.

This will enable us to simplify restoreObject. All the nodes can be restored without being applied, which will mean the restore step won't need to be deferred until after the file is restored.

I plan to add DecorationDecl and perhaps DecorationStmt that hold nothing apart from a set of decorations.

Pros?
These comments aren't attached to nodes, so it makes sense to have them as their own node.

Cons?
The dst package becomes more opinionated - I'd prefer it to stay as close to go/ast as possible so it's more familiar.

I've assumed that the format package will handle all indenting, so dst doesn't bother reconstructing the byte offsets for indent characters. However, there seems to be one exception:

switch a {
case 1:
	// Comment about case 1
case 2:
}

switch a {
case 1:
// Comment about case 2
case 2:
}

Both are valid and will not be modified by gofmt. Since the AST nodes created by the Restorer have no indenting, the format package will always choose the second version.

How to fix this?

Are these needed? I've ignored adding spaces between tokens, and it seems to render fine - but is this always the case?

package main

import (
	"github.com/dave/dst/decorator"
	"github.com/dave/dst/decorator/resolver/goast"
	"github.com/dave/dst/decorator/resolver/guess"
	"go/token"
)

func main() {
	code := `package main

import (
	"github.com/opentracing/opentracing-go"
)

var a opentracing.Tracer
`

	dec := decorator.NewDecoratorWithImports(token.NewFileSet(), "main", goast.New())

	f, err := dec.Parse(code)
	if err != nil {
		panic(err)
	}
	res := decorator.NewRestorerWithImports("main", guess.New())
	if err := res.Print(f); err != nil {
		panic(err)
	}
}

get:

package main

var a opentracing.Tracer

expect:

package main

import (
	"github.com/opentracing/opentracing-go"
)

var a opentracing.Tracer

Hello dave, I've got one question about the rewrite API for you:)

I think the Apply(root dst.Node, pre, post ApplyFunc) (result dst.Node) API is not as useful as it can be, one will have to do many ast node cast to do useful things, which essentially duplicates the work done in ast.Walk etc.

The most tricky part when rewrite is finding and replacing all its references, not just renaming the declare side, is that right? IMHO a useful API should consider references automatically.

UPDATE

I find dst can automatically handle references for renaming import, but not type, var, const etc.

Would be nice to have support for go's line directives.

The simplest way I believe would be to avoid indenting line comments Decorations starting with //line

Thank you!

How should we handle these?

I can't seem to figure out how to make an empty struct without a new line. I've tried the struct decorator and the field list decorator. It feels like I should be altering the field decorator but there are no fields. I'm fighting with a new linter that is removing the empty line from empty structs and I don't want generated code to be affected by the linter. Here's what I get:

type MyEmptyStruct struct {
}

Here's what I want:

type MyEmptyStruct struct {}

Thanks in advance!

Comments that are placed inside empty function, loop or if statement bodies are not found in the respective node's decorations. I would expect them to be placed somewhere in the e.g. dst.FuncDeclDecorations of the function or similarly the dst.ForStmtDecorations of the loop declaration. This snipped demonstrates the issue:

package main

func main() {
}

func empty() {
	// inside empty function, not found
}

func foo() {
	i := 1
	for i < 2 {
		// inside empty for statement, not found
	}
	// after empty for statement, found and associated with for statement

	if i == 3 {
		// inside empty if statement, not found
	}
	i = 4
}

And here you can find code to reproduce the issue: https://play.golang.org/p/Yg9OOkoP6IF [Disclaimer: sometimes the playground needs multiple tries to import all packages, so if you encounter a timeout running go build error, just execute the code again.]

I tried using dst in order to fix the issue golang/go#39753 I had with go/ast. As I am a fairly new user of this module, I might also have overlooked something - Please let me know if this is the case and how I could get the correct comment associations :) Thanks!

$ go version
go version go1.14.2 darwin/amd64

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/.../Library/Caches/go-build"
GOENV="/Users/.../Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/.../go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/.../go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/yb/hqncynqs0b5_3hyxcjpdsyr00000gr/T/go-build447564702=/tmp/go-build -gno-record-gcc-switches -fno-common"

it is hard to move the existed code based on go/ast to dst .

Hi,

I wrote code with the standard ast, astutil and inspector packages, which gets some code from a file and adds some new blocks to a specific position, plus some imports.
Worked well, I had only one problem, wasn't able to figure out how to add a newline before the inserted block. :D
That's how I found your library.
Currently, I have a working code, which is:

package main

import (
	"go/ast"
	"go/format"
	"go/parser"
	"go/token"
	"os"
	"strings"

	"golang.org/x/tools/go/ast/astutil"

	"github.com/dave/dst"
	"github.com/dave/dst/decorator"
	"github.com/dave/dst/dstutil"
	"golang.org/x/tools/go/ast/inspector"
)

func main() {
	code := `package main

	import "github.com/prometheus/client_golang/prometheus/promhttp"

	func main() {
		c.echoEngine.GET("metrics", echo.WrapHandler(promhttp.Handler()))
	}
`

	newFile, err := decorator.Parse(code)
	if err != nil {
		panic(err)
	}

	var newBlockStmt *dst.BlockStmt
	for _, d := range newFile.Decls {
		f, ok := d.(*dst.FuncDecl)
		if !ok {
			continue
		}

		if f.Name.Name == "main" {
			newBlockStmt = f.Body
		}
	}

	fset := token.NewFileSet()
	origFile, err := parser.ParseFile(fset, "controller.go.source", nil, 0)
	if err != nil {
		panic(err)
	}

	blockStmt, exprStmt := healthCheckParentNodes(origFile)

	dec := decorator.NewDecorator(fset)
	dstFile, err := dec.DecorateFile(origFile)
	if err != nil {
		panic(err)
	}

	dstBlockStmt := dec.Dst.Nodes[blockStmt].(*dst.BlockStmt)
	dstExprStmt := dec.Dst.Nodes[exprStmt].(*dst.ExprStmt)

	dstutil.Apply(dstBlockStmt, func(c *dstutil.Cursor) bool {
		if c.Node() != dstExprStmt {
			return true
		}

		for i := len(newBlockStmt.List) - 1; i >= 0; i-- {
			c.InsertAfter(newBlockStmt.List[i])
		}
		return false
	}, nil)

	newSource, err := os.Create("controller_new.go.source")
	if err != nil {
		panic(err)
	}
	defer newSource.Close()

	fset, origFile, err = decorator.RestoreFile(dstFile)
	if err != nil {
		panic(err)
	}

	for _, i := range newFile.Imports {
		astutil.AddImport(fset, origFile, strings.Trim(i.Path.Value, `"`))
	}

	if err := format.Node(newSource, fset, origFile); err != nil {
		panic(err)
	}

}

func healthCheckParentNodes(f *ast.File) (blockStmt ast.Node, exprStmt ast.Node) {
	types := []ast.Node{new(ast.BasicLit)}

	ins := inspector.New([]*ast.File{f})
	ins.WithStack(types, func(n ast.Node, push bool, stack []ast.Node) (prune bool) {
		if !push || blockStmt != nil {
			return false
		}

		lit := n.(*ast.BasicLit)
		if !strings.Contains(lit.Value, "healthcheck") {
			return true
		}

		for i := len(stack) - 1; i >= 0; i-- {
			if s, ok := stack[i].(*ast.ExprStmt); ok {
				exprStmt = s
				continue
			}

			if s, ok := stack[i].(*ast.BlockStmt); ok {
				blockStmt = s
				break
			}
		}

		return false
	})

	return blockStmt, exprStmt
}

The input file looks like this:

package rest

import (
	"net/http"
	"net/http/pprof"
	"os"
	"runtime"

	"github.com/labstack/echo"
)

type controller struct {
	echoEngine *echo.Echo
}

func NewController(echoEngine *echo.Echo) *controller {
	return &controller{echoEngine: echoEngine}
}

func (c *controller) Start() {
	if os.Getenv("DEBUG") == "true" {
		runtime.SetBlockProfileRate(1)
		runtime.SetMutexProfileFraction(5)

		c.echoEngine.GET("/debug/pprof/*", echo.WrapHandler(http.HandlerFunc(pprof.Index)))
		c.echoEngine.GET("/debug/pprof/cmdline", echo.WrapHandler(http.HandlerFunc(pprof.Cmdline)))
		c.echoEngine.GET("/debug/pprof/profile", echo.WrapHandler(http.HandlerFunc(pprof.Profile)))
		c.echoEngine.GET("/debug/pprof/symbol", echo.WrapHandler(http.HandlerFunc(pprof.Symbol)))
		c.echoEngine.GET("/debug/pprof/trace", echo.WrapHandler(http.HandlerFunc(pprof.Trace)))
	}

	c.echoEngine.Add(http.MethodGet, "/healthcheck", func(eCtx echo.Context) error {
		return eCtx.String(http.StatusOK, "ok")
	})

	c.echoEngine.Add(http.MethodGet, "/something-else", func(eCtx echo.Context) error {
		return eCtx.String(http.StatusOK, "ok")
	})
}

The code is way too complicated and using both the ast and dst packages, plus the inspector, so I have some questions.

First, I wasn't able to figure out how to add the import from the new code snippet to the old one. Checked the readme about resolvers, but it feels way too complicated and mainly developed for automatic import management. I just wanted to add an import manually without any magic, like the astutil.AddImport.
Tried to do something like dstFile.Imports = append(dstFile.Imports, newFile.Imports...), but when printed out the source code with decorator.Print(dstFile), the import was missing.
Is it possible to do this with dst?

Second thing, that there is a decorator.Decorate function, which returns a dst.Node, first I used this to add the newline, but wasn't able to figure out how to convert back to an ast.Node, I only saw a method to restore a whole ast.file.
Is it possible? Would this work? I mean, would that keep the newline in the ast.Node too?

Third, I wasn't able to figure out how to replace the inspector WithStack method with dst related code, because I don't know how to build a stack.
I can't see any method which tells about parent nodes during Inspect or Walk, so I don't know how the current node relates to the previous one.
Is it possible with dst?

It's a little long, sorry, checked the documentation, but didn't find the answers.

`
func TestPosition(t *testing.T) {
path := "../internal/transports/bulletin/router_bulletin.go"

//===============================================================
// 1: Use ast directly 
astFset := token.NewFileSet()
astFile, err := parser.ParseFile(astFset, path, nil, parser.ParseComments)
if err != nil {
	log.Fatalln(err)
}
fmt.Print(astFile.Decls[3].Pos(), ",")
fmt.Println(astFile.Decls[3].End())

//===============================================================
// 2: parse to ast, then use decorator
dec := decorator.NewDecorator(astFset)    //包装器
dstFile, err := dec.DecorateFile(astFile) //对astFile进行包装
if err != nil {
	log.Fatalln(err)
}
fmt.Print(dec.Ast.Nodes[dstFile.Decls[3]].Pos(), ",")
fmt.Println(dec.Ast.Nodes[dstFile.Decls[3]].End())

//===============================================================
// 3:  parse to dst, then use restorer
dstFset := token.NewFileSet()
file, err := decorator.ParseFile(dstFset, path, nil, parser.ParseComments)
if err != nil {
	log.Fatalln(err)
}
restorer := decorator.NewRestorer()
_, err = restorer.RestoreFile(file)
if err != nil {
	log.Fatalln(err)
}
a := restorer.Ast.Nodes[file.Decls[3]]
fmt.Print(a.Pos(), ",")
fmt.Println(a.End())

}`

Result:
1942,2357
1942,2357
1845,2210 //wrong

Is this normal?

Some comments are mis-aligned after format.Node... They get the correct alignment after a second run through gofmt. I suspect we need to add spacing for separator characters?

Hi dave~

ast.MergePackageFiles can be used to merge multiple files, but the resultant file will have multiple import declarations which is very ugly and maybe not even valid syntax, like this:

package demo

import "fmt"

func hi() {
	fmt.Println("hi")
}

import (
	"fmt"
	"math"
)

func hello() {
	fmt.Println(math.Abs(-1))
}

Does dst support merging multiple import declarations into one while reserving comments?

I'm trying to generate code similar to this:

type something struct {
	here int
}

func newSomething() *something {
	return &something{
		here: 42,
	}
}

I've got the struct in place and a start on the func but the body of the func is giving me an error:

panic: duplicate node: &dst.Ident{Name:"something", Obj:(*dst.Object)(nil), Path:"", Decs:dst.IdentDecorations{NodeDecs:dst.NodeDecs{Before:0, Start:dst.Decorations(nil), End:dst.Decorations(nil), After:0}, X:dst.Decorations(nil)}}

goroutine 1 [running]:
github.com/dave/dst/decorator.(*FileRestorer).restoreNode(0xc000626e10, 0x1457e60, 0xc000625a80, 0x13dd1ff, 0xc, 0x13db130, 0x4, 0x13db0c0, 0x4, 0x0, ...)
	/Users/dschultz/workspace/go/pkg/mod/github.com/dave/[email protected]/decorator/restorer-generated.go:15 +0x14356
github.com/dave/dst/decorator.(*FileRestorer).restoreNode(0xc000626e10, 0x1457c80, 0xc000559c20, 0x13dc48c, 0x9, 0x13dacc7, 0x1, 0x13db0c0, 0x4, 0x0, ...)
	/Users/dschultz/workspace/go/pkg/mod/github.com/dave/[email protected]/decorator/restorer-generated.go:459 +0x7fe
github.com/dave/dst/decorator.(*FileRestorer).restoreNode(0xc000626e10, 0x1458140, 0xc000619730, 0x13dc8fe, 0xa, 0x13dbaef, 0x7, 0x13db0c0, 0x4, 0x0, ...)
	/Users/dschultz/workspace/go/pkg/mod/github.com/dave/[email protected]/decorator/restorer-generated.go:1839 +0x1b0d

I dumped the DST of a go file similar to what I want to create:

  1280  .  .  .  Body: *dst.BlockStmt {
  1281  .  .  .  .  List: []dst.Stmt (len = 1) {
  1282  .  .  .  .  .  0: *dst.ReturnStmt {
  1283  .  .  .  .  .  .  Results: []dst.Expr (len = 1) {
  1284  .  .  .  .  .  .  .  0: *dst.UnaryExpr {
  1285  .  .  .  .  .  .  .  .  Op: &
  1286  .  .  .  .  .  .  .  .  X: *dst.CompositeLit {
  1287  .  .  .  .  .  .  .  .  .  Type: *dst.Ident {
  1288  .  .  .  .  .  .  .  .  .  .  Name: "something"
  1289  .  .  .  .  .  .  .  .  .  .  Obj: *(obj @ 50)
  1290  .  .  .  .  .  .  .  .  .  .  Path: ""

My code is creating an identical structure but it looks like the restorer code thinks that something is duplicate when defined on line 1288 above. Also not sure what to do about the Op field. I'm setting it to token.AND but not sure if that's right.

Any help is greatly appreciated!

Right now we have very fine-grained low-level control of the imports block. It would be better to have some convenience functions so that adding a qualified identifier automatically adds the import spec. This is problematic because we don't know the mapping between package name and package path.

I have no need for import management, and wanted to use Load to get a DST of a package; but there is no way to configure the load and opt-out. My (messy) workaround was to use packages to load the package, then decorate each *ast.File syntax separately, and pass around the associated decorator.

Perhaps a custom Load config is needed, that might also address #53?

I want to do the following to my Go project

1. Remove all private const / var / type / function / method
2. Replace the implement of all public functions with zero return values (for example: "" for string, nil for pointer)

How to do this with dst ?

Investigating...

This code is a bit of a mess. I will remove the fragger type and combine with Decorator

This is very similar to #9 but a more general case.

A statement that flows onto multiple lines will be indented. This means that comments that follow will be able to have two indent levels:

package a

const a = 1 +
	1
	// a1

	// a2
const b = 1

const c = 1 +
	1

// d1

// d2
const d = 1

This is not a small change, and I will be hiking in Patagonia from December 2021 until April or May 2022 without my laptop, so I won't be able to get this done until I get back.

When I decorate an AST, and then walk the DST, I encouter a *dst.FuncType node, but there is no mapping back to the corresponding *ast.FuncType node.

In decorator/decorator-node-generated.go, it appears that the mapping is not made in the *ast.FuncDecl case.

could you please help me to figure out one thing?
in the docs here

// All nodes contain position information marking the beginning of
// the corresponding source text segment; it is accessible via the
// Pos accessor method. Nodes may contain additional position info
// for language constructs where comments may be found between parts
// of the construct (typically any larger, parenthesized subpart).
// That position information is needed to properly position comments
// when printing the construct.

but at the same time the interface and it's implementations do not contain this method at all.
I have a doubt if dst nodes contain positions at all

If there're no positions provided, what approach would you suggest in implementing them?

The code that generates the conversion functions should be simpler. FragmentInfo should contain all the info needed to do the generation so the generation functions don't need to use go/types. In addition we should generate a file with all the FragmentInfo objects as literals, to serve as documentation for how the generation code works.

Currently each Comment is added to it's own CommentGroup. This renders ok, but it would be more correct to group sequences of Comments into CommentGroups.

Hi there! Thank you so much for this awesome library!

I want to write a small refactoring tool. The idea is to load a package, move some things around using dst, and then write the changes back to disk. I am able to load my package, do changes and print the result of each file to stdout with code like this:

func featureRequest() {
	dir := "/tmp/t"
	pkgs, err := decorator.Load(&packages.Config{Dir: dir, Mode: packages.LoadSyntax}, "root")
	if err != nil {
		panic(err)
	}
	p := pkgs[0]
	fmt.Printf("parsed %d files\n", len(p.Syntax))
	for _, f := range p.Syntax {
		b := f.Decls[0].(*dst.FuncDecl).Body
		b.List = append(b.List, &dst.ExprStmt{
			X: &dst.CallExpr{
				Fun: &dst.Ident{Path: "fmt", Name: "Println"},
				Args: []dst.Expr{
					&dst.BasicLit{Kind: token.STRING, Value: strconv.Quote("Hello, World!")},
				},
			},
		})

		fmt.Println("=====================================")
		r := decorator.NewRestorerWithImports("root", gopackages.New(dir))
		if err := r.Print(f); err != nil {
			panic(err)
		}
	}
}

This successfully loads the files, applies changes and resolves the imports (great!). Here's the output of calling the function above (I have these files in my /tmp/t directory: main.go, sum.go and go.mod):

$ go run ./cmd/refactorer/main.go 
parsed 2 files
=====================================
package main

import "fmt"

func sum(int x, int y) {
	_ = x + y
	fmt.Println("Hello, World!")
}
=====================================
package main

import "fmt"

func main() { fmt.Println("Hello, World!") }

However, as far as I could see, the paths of .go files that were loaded are not exposed anywhere. This means at the moment I can't write the changes back to disk because I don't know which parsed file inside p.Syntax corresponds to which path.

It would be awesome if we could expose the file names in the struct returned by decorator.Load, perhaps by converting Syntax:

Hi there, can dst parse a single go file with it's all imported dependencies to a AST tree? I noticed "NewDecoratorWithImports" may help but when I dig into it, seems like not meet my expectation. I have an example below:

File request.go:
import Team type Request struct { team Team json:"team" }

File team.go
type Request struct { name string json:"name" }

What I'd like to have is to get the whole AST tree of Request with all the information of Team. Thank you.

This is minor, but curious whether you'd take a patch to grant users of the library control over import grouping.

Some organizations (including my own) tend to group imports beyond stdlib/non-stdlib. I've seen two permutations at different orgs:

1. stdlib, public, private
2. stdlib, public, private (shared), private (local)

goimports supports the former via its -local flag, such that:

import (
	"net/http"
	"github.com/public/pkg"
	"github.com/private/pkg"
)

when processed via goimports -local github.com/private will result in:

import (
	"net/http"

	"github.com/public/pkg"

	"github.com/private/pkg"
)

Further, goimports does not remove pre-existing groups.

I'd like dst's import management to provide the ability to retain the original grouping of imports - though this does introduce the problem of needing to find an appropriate place to insert any new imports that aren't stdlib.

Alternately, perhaps the best bet is to avoid using an import-aware FileRestorer and implement similar logic by hand.

When dst automatically manages the imports block, the imports are internally keyed by path. This incorrectly assumes that only one import for each package path is possible. Although very rare, it is possible to add the same package multiple times with different aliases.

See: golang/go@3409ce3

In TestLoadStdLibAll we test that the entire standard library is able to be parsed and rebuilt including management of the imports block. This test fails for the two files server.go and request.go in net/http because of this.

In order to handle this gracefully would involve a major rewrite of the import management code which is the most complex part of dst. For now I've skipped the TestLoadStdLibAll for these two files.

Re: @dmitshur

Perhaps we should stop generating the code and hand-craft the conversion functions. It would make handling special cases simpler... and the code generation scripts aren't simple to follow.

Decoration points that come after lists of nodes are not needed - there will always have a node to attach to, and usually this will be a better attachment point.

Unfortunately we can't remove the End decorations that come directly after lists - this would interfere with the detection of hanging indents (#18).

We should remove:

Field.Names
CallExpr.Args
AssignStmt.Lhs
CaseClause.List
ValueSpec.Names

SliceExpr

Two colons may be present if Slice3 == true even if Max == nil?

ChanType

No arrow when Dir == 0

EmptyStmt

Semicolon length = 1/0 when ???

TypeAssertExpr

When Type == nil, then .(type) is rendered

CommClause

"case" token changes to "default" if Comm == nil

Range

"range" token rendered before X?

FuncDecl

"func" token from Type rendered before "Recv" and "Name"

my code:

dec := decorator.NewDecoratorWithImports(token.NewFileSet(), "main", goast.New())
f, err := dec.Parse("package test\nfunc init() {}")
if err != nil {
	panic(err)
}

b := f.Decls[0].(*dst.FuncDecl).Body
b.List = append(b.List, &dst.ExprStmt{
	X: &dst.CallExpr{
		Fun: &dst.SelectorExpr{
			X: &dst.Ident{
				Name: "Gin",
			},
			Sel: &dst.Ident{
				Name: "GET",
			},
		},
		Args: []dst.Expr{
			&dst.BasicLit{
				Kind:  token.STRING,
				Value: strconv.Quote("url"),
			},
			&dst.SelectorExpr{
				X: &dst.Ident{
					Name: "packageName",
					Path: "module/packageName",
				},
				Sel: &dst.Ident{
					Name: "functionName",
				},
			},
		},
	},
})

res := decorator.NewRestorerWithImports("main", guess.New())
res.Print(f)

result：

package test

import "module/packageName"

func init() { Gin.Get("url", packageName.packageName.functionName) }

====
I want Gin.Get("url", packageName.functionName)

import is not work