mmcloughlin / avo Goto Github PK
View Code? Open in Web Editor NEWGenerate x86 Assembly with Go
License: BSD 3-Clause "New" or "Revised" License
Generate x86 Assembly with Go
License: BSD 3-Clause "New" or "Revised" License
Provide access to interface arguments. See this asmdecl
code for the memory layout and expected parameter naming.
Should also extend examples/args
to demonstrate how to use it.
We should support TEXT
and GLOBL
attributes as defined in textflag.h
.
textflag.h
?It would be good to support comments in the output. At the moment it is natural to have comments in the generator:
Lines 21 to 28 in 022d24d
This could just as well be replaced with Comment("Load initial hash")
and it would serve to document the generator and the output.
It would be good to generate a massive test case that covers all the instruction builder functions.
The asmtest
generator works by building a huge assembly function with one line for every instruction form.
Lines 22 to 28 in 27cea3b
We could do the same but for the avo
builders instead of going straight to assembly.
Is there a golangci-lint
configuration option or another linter that will ensure all packages have package-level doc?
Related to #9
Run the asmdecl
analysis pass on all generated assembly to ensure we are referencing parameters correctly.
https://godoc.org/golang.org/x/tools/go/analysis/passes/asmdecl
Currently the build
package consists of a Context
struct with methods for incrementally building a program. Then for convenience, we have a global Context
object and package-level functions operating on it. The two have to be kept in sync manually. Ideally this would be automated.
The following gist contains a start of what this could look like
https://gist.github.com/mmcloughlin/2f6ff496978ef57efce13eab84c69cd5
@myitcv suggested that godoc
could be used to get most of the way:
I'd say writing a code generator in that case would be overkill. Use 'go doc` on the package to get the methods (via grep) then sed the output to generate the functions.
https://gophers.slack.com/archives/C0VPK4Z5E/p1544342455029800
Instruction builders do not have documentation.
Lines 109 to 116 in f4c5957
We should autogenerate it (same as for constructors).
Creating an issue to record thoughts on moving to Intel XED for the instruction database (started in #22).
Add golint
to list of enabled linters.
Some instructions require fixed registers for certain operands. We need to incorporate this into register allocation.
See PeachPy implementation:
GLOBL
should not construct a static symbol.
Lines 28 to 32 in f4c5957
We should use go modules for dependencies.
In the past I have used preprocessor #define
s to effectively name registers. See for example:
I am wondering if the same could be done in avo
with virtual registers (or even physical registers too). This would help make the output more comprehensible, and ease debugging #6.
It would be great to support a "debug" print mode. This would display all metadata from the AST, including the results of passes. See PeachPy format_instructions
for example.
The v
in GP64v()
and others is ugly. I no longer remember why this was necessary. Consider removing it.
Support compiler directives such as go:noescape
.
https://groups.google.com/forum/#!topic/golang-nuts/_gKfyReaPmY
https://golang.org/cmd/compile/
For example, it is sometimes necessary to access the X
sub-register of a Y
register. PeachPy supports this through the as_xmm
property (see dot product example). avo
should have support for this feature.
TEXT
and GLOBL
symbols are currently too restrictive. I think we do not correctly support:
See:
Line 106 in 49b1691
Line 78 in 49b1691
Related to #37
Symbols have:
.
is a convenience for current package.
and /
are replaced with unicode in asm source codeName is required. The others are optional and we see all 4 possibilities in the standard library:
no pkg, no static: p256SubInternal
pkg, no static: syncโatomicยทLoadInt32
no pkg, static: _expand_key_192b<>
pkg, static: runtimeโinternalโatomicยทkernelcas<>
The following TEXT
symbol names do occur in the standard library:
_expand_key_192b<> (static, no dot)
cmpbody<> (static, no dot)
runtimeยทsigprofNonGoWrapper<> (static, package qualified)
runtimeโinternalโatomicยทkernelcas<> (static, package qualified)
syncโatomicยทLoadInt32 (package qualified)
p256SubInternal (no dot)
TEXT
Symbolshttps://github.com/golang/go/blob/fd752d5ede482cdf52a920c75486677cbcb441b0/src/crypto/aes/asm_amd64.s#L243
https://github.com/golang/go/blob/master/src/internal/bytealg/compare_amd64.s#L30
Almost always these do not include the dot symbol, but there are some weird exceptions.
https://github.com/golang/go/blob/14560da7e469aff46a6f1270ce84204bbd6ffdb3/src/runtime/sys_linux_ppc64x.s#L420
https://github.com/golang/go/blob/14560da7e469aff46a6f1270ce84204bbd6ffdb3/src/runtime/internal/atomic/sys_linux_arm.s#L34
Note this is the unicode "division slash U+2215"
Assembler lexer:
https://github.com/golang/go/blob/fd752d5ede482cdf52a920c75486677cbcb441b0/src/cmd/asm/internal/lex/lex.go#L104-L114
https://github.com/golang/go/blob/fd752d5ede482cdf52a920c75486677cbcb441b0/src/cmd/asm/internal/lex/tokenizer.go#L52-L67
.
and /
"".
Assembler:
https://github.com/golang/go/blob/fd752d5ede482cdf52a920c75486677cbcb441b0/src/cmd/asm/internal/asm/asm.go#L79-L90
https://github.com/golang/go/blob/fd752d5ede482cdf52a920c75486677cbcb441b0/src/cmd/internal/obj/plist.go#L80-L82
NAME_EXTERN
or NAME_STATIC
asmdecl
pass
<>
must have Go declarationsklauspost/asmfmt is the de facto standard for Go assembly formatting. It would be good to produce output that conforms to asmfmt
.
I have a preference for avoiding non-Go dependencies (stdlib and sub-repos only). Therefore:
asmfmt
explicitly, that would be preferred. This may actually be possible since (at the time of writing) many of the asmfmt
rules simply wouldn't apply to avo
output.asmfmt
are too complicated, then we can accept the additional dependency.Either way, it would be good to have a check in CI to confirm that all generated files are formatted correctly. Something like find . -name '*.s' | xargs asmfmt -w
and check the git repo is clean.
It should be possible to add build tags to generated files. Questions/thoughts:
amd64
?Related: Maratyszcza/PeachPy#75
Measuring unit test coverage is trivial. We should do this anyway.
A lot of the testing at the moment is integration testing (see examples
and tests
directories). Can we measure the coverage of this testing too?
In a previous life, I did a z80 disassembler that generated almost correct c for the "Small C" compiler. The combination could round-trip with a little human help, and I found it far clearer than disassembling to raw assembler notation.
This language might be an elegant disassembler target.
Our linter is passing but we're getting errors on goreportcard at the moment:
https://goreportcard.com/report/github.com/mmcloughlin/avo
These are all golint
errors:
$ golint ./...
examples/args/args.go:3:6: exported type Struct should have comment or be unexported
examples/args/args.go:18:6: exported type Sub should have comment or be unexported
examples/returns/returns.go:3:6: exported type Struct should have comment or be unexported
examples/sha1/sha1.go:8:2: exported const Size should have comment (or a comment on this block) or be unexported
examples/sha1/sha1.go:12:1: exported function Sum should have comment or be unexported
examples/stadtx/stadtx.go:30:1: exported function SeedState should have comment or be unexported
examples/stadtx/stadtx.go:64:6: exported type State should have comment or be unexported
It appears that golangci-lint
excludes examples
directories by default.
I would like to enable linting of examples
directory and fix any issues.
The go assembler supports both of the following instructions but avo does not.
I see that x86/zctors.go is autogenerated, but I'm a bit lost on how to update the generator to resolve this. To make sure that valid output would be generated I modified it by hand and the generated code was properly assembled:
diff --git a/x86/zctors.go b/x86/zctors.go
index 6d03480..f52ba55 100644
--- a/x86/zctors.go
+++ b/x86/zctors.go
@@ -1280,6 +1280,7 @@ func ANDPS(mx, x operand.Op) (*intrep.Instruction, error) {
// ANDQ imm32 rax
// ANDQ imm8 r64
// ANDQ imm32 r64
+// ANDQ imm64 r64
// ANDQ r64 r64
// ANDQ m64 r64
// ANDQ imm8 m64
@@ -1308,6 +1309,13 @@ func ANDQ(imr, mr operand.Op) (*intrep.Instruction, error) {
Inputs: []operand.Op{mr},
Outputs: []operand.Op{mr},
}, nil
+ case operand.IsIMM64(imr) && operand.IsR64(mr):
+ return &intrep.Instruction{
+ Opcode: "ANDQ",
+ Operands: []operand.Op{imr, mr},
+ Inputs: []operand.Op{mr},
+ Outputs: []operand.Op{mr},
+ }, nil
case operand.IsR64(imr) && operand.IsR64(mr):
return &intrep.Instruction{
Opcode: "ANDQ",
@@ -8450,6 +8458,7 @@ func MOVBWZX(mr, r operand.Op) (*intrep.Instruction, error) {
// MOVD imm32 m64
// MOVD r64 m64
// MOVD xmm r64
+// MOVD r32 xmm
// MOVD r64 xmm
// MOVD xmm xmm
// MOVD m64 xmm
@@ -8505,6 +8514,13 @@ func MOVD(imrx, mrx operand.Op) (*intrep.Instruction, error) {
Inputs: []operand.Op{imrx},
Outputs: []operand.Op{mrx},
}, nil
+ case operand.IsR32(imrx) && operand.IsXMM(mrx):
+ return &intrep.Instruction{
+ Opcode: "MOVD",
+ Operands: []operand.Op{imrx, mrx},
+ Inputs: []operand.Op{imrx},
+ Outputs: []operand.Op{mrx},
+ }, nil
case operand.IsR64(imrx) && operand.IsXMM(mrx):
return &intrep.Instruction{
Opcode: "MOVD",
Need a full README with all the usual sections.
Once done, enable public doc linting. Unfortunately this will require a workaround for golangci/golangci-lint#21.
Consider porting existing PeachPy Go projects to avo
. At a minimum this would be really valuable feedback and system-level testing for avo
. These could be committed to the avo
examples directory or potentially committed back to the original repos if their maintainers are interested.
Repository | Description | Stars |
---|---|---|
Yawning/chacha20 |
ChaCha20 cryptographic cipher. | 33 |
Yawning/aez |
AEZ authenticated-encryption scheme. | 6 |
robskie/bp128 |
SIMD-BP128 integer encoding and decoding. | 22 |
dgryski/go-marvin32 |
Microsoft's Marvin32 hash function. | 7 |
dgryski/go-highway |
Google's Highway hash function. | 55 |
dgryski/go-metro |
MetroHash function. | 66 |
dgryski/go-stadtx |
Stadtx hash function. See examples/stadtx |
7 |
dgryski/go-sip13 |
SipHash 1-3 function. | 17 |
dgryski/go-chaskey |
Chaskey MAC. | 5 |
dgryski/go-speck |
SPECK cipher. | 7 |
dgryski/go-bloomindex |
Bloom-filter based search index. | 79 |
dgryski/go-groupvariant |
SSE-optimized group varint integer encoding. | 25 |
bwesterb/go-sha256x8 |
Eight-way SHA256 | 0 |
gtank/ed25519 |
radix51 sub-package originally generated with PeachPy |
8 |
We are reliant primarily on the examples for testing at this point. It would be good to "stress" the allocator because I would be (pleasantly) surprised if it's bug-free.
tests/
that (for example) uses the max number of registers of a given kind and confirms the register allocator doesn't fall overUnit tests right now are trivial:
Lines 9 to 46 in 7752262
For complexity reasons AVX-512 was not initially considered. We should add support.
Lines 130 to 133 in 9fbb71b
We currently have U8
, U16
, ... as well as Imm
. I am concerned that Imm
is confusing, so perhaps should be removed.
Lines 20 to 32 in 9fbb71b
I don't think the ast.go
types make much sense in the root avo
directory. Consider moving them to a sub-package github.com/mmcloughlin/avo/ir
.
Preference for ir
(intermediate representation) since these types contain much more than a representation of the syntax, for example the results of liveness analysis or register allocation. Moreover I have a half-baked plan to write a parser for a superset of Go assembly into avo formats, so in that world ast
may make more sense as a target for the parser (which is then transformed to ir
).
Add support for referencing symbols in other packages. Include an example of calling an external function or (potentially easier) reading a variable from another package.
Related: #39
Remove leftover comments from porting the Stadtx example
Lines 110 to 118 in 9fbb71b
Replace for example:
Lines 5 to 8 in 672b77a
Use function/data attributes #2 in a new or existing example.
Change this to Label
and make it match the corresponding function in Context
.
Related to #33 (we should keep the context and package-level functions in sync)
Implement a dot product example. This is both an excellent example and helps text feature parity with the same example in PeachPy.
Depends on #1
Other test suites likely provide useful edge cases:
asmjit_test_x86_cc.cpp
Lines 182 to 184 in f4c5957
I think it would be better to return something that implements error
. Some "multi error" type wrapper similar to https://golang.org/pkg/go/scanner/#ErrorList.
Liveness analysis should process instructions in topological sort order. Performance is acceptable for now, but this should be addressed at some point.
Lines 18 to 19 in 9fbb71b
Some common use cases are a bit ugly right now. Consider adding helpers to make this nicer.
Lines 14 to 15 in 7752262
Perhaps offer LoadPointer
that wraps in a Mem
operand for you.
Lines 13 to 14 in 7752262
Perhaps offer a LoadSlice
that returns a Slice
object with registers in it for base, len, cap? But, then what about the common case where cap
is ignored? Seems you might need some mechanism for lazy loading the parameters, depending on which of the subcomponents are actually accessed? Needs more thought.
It's more annoying that it should be
Lines 20 to 34 in 7752262
"loop"
above): if Label
returned a LabelRef
you would not need to repeat the string literal "loop"
"done"
above)... not sure yetAllocating an array of registers is fairly common.
Line 22 in 7752262
In cases like these it could be nice to have a convenience to block allocate registers. So this would be something like GP32s(5)
.
Provide README
s for examples:
examples/
folder: links and summary for each example (maybe auto-generated)avo
errors are currently very hard to understand due to lack of file:line
references. We should fix this.
At the same time, it may be good to consider how to make position information more widely available to other components (#6 for example).
(Reuse go/token.Position?)
Using shippable while the repo is private. Prefer to switch to Travis CI once public.
We depend on Maratyszcza/Opcodes for our instruction database, and indeed include a copy in this repo. Need to check the license and include a copy if necessary.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.