Giter Club home page Giter Club logo

go-internals's Introduction

go-internals

go-internals is a work-in-progress book about the internals of the Go (1.10+) programming language.


Table of Contents


Click here for the GitBook version.

Goals

  • Concise: The book aims to be as concise as possible, encouraging code and diagrams over lengthy prose.
  • Community-effort: I myself am learning as I go through the writing of this book. I will make mistakes along the way. Hopefully the community can help in pointing out and correcting these mistakes.
  • Beyond the theory: The book will not just cover the theory, but the actual implementation too. Assumptions will be proven or invalidated via experiments and measurements.
  • Up-to-Date: The book will try to keep up-to-date with new Go versions being released.
  • Experienced audience: The Go community has created loads of great introductory material for newcomers. Unfortunately we're still lacking good resources when it comes to the more advanced stuff. This books hopes to help solve this issue.

Translations

Contributing

All kinds of contributions are very much welcome.

Don't hesitate to open an issue for e.g...:

  • pointing out technical or english mistakes
  • suggesting improvements and/or additions to existing chapters
  • suggesting external links that might be of interest
  • ..and pretty much anything else you can think of, really!

Author

Clement Rey <[email protected]> (@teh_cmc)

License

Licensed under the BY-NC-SA Creative Commons 4.0 International Public License

go-internals's People

Contributors

aarzilli avatar antekresic avatar cch123 avatar teh-cmc avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

go-internals's Issues

Request for update(if possible)

After so many years, seems go compiler optimization have changed a lot. One problem is in chapter 2, iface.go, chapter 2. To make the assembly code call the function through itab func pointer,

func main() {
    m := Mather(Adder{id: 6754})
    // This call just makes sure that the interface is actually used.
    // Without this call, the linker would see that the interface defined above
    // is in fact never used, and thus would optimize it out of the final
    // executable.
    m.Add(10, 32)
}

should be changed to

func main() {
	var m Mather
	m = Adder{id: 6754}
	// This call just makes sure that the interface is actually used.
	// Without this call, the linker would see that the interface defined above
	// is in fact never used, and thus would optimize it out of the final
	// executable.
	m.Add(10, 32)
}

Or the assembly code would CALL "".Adder.Add(SB) directly.

chapter2: Interface conversions. Type assertions where the destination type is itself an interface.

It's quite possible this is covered in Chapter 2 or elsewhere, and I missed it, because I am lazy. If so, apologies!

Based on my C++ background, I believe I completely understand how Go handles your type-assertion example here. It can just, as you say, compare Eface._type with type.uint32.

But I cannot fathom how the following code works!

type Craft interface { Float() }

type Car struct {}
type Boat struct {}

func (c Boat) Drive() {}
func (c Boat) Float() {}
func (c Car) Drive() {}

var vface interface { Drive() }
var cface interface { Float() }

func main() {
  b := Boat{}
  vface = b
  cface = vface.(Craft)
}

On the line cface = vface.(Craft), all the compiler knows about vface is that it is some kind of Vehicle. (Incidentally, its _type compares equal to type.Boat; but nobody said anything about Boat on this line, so that can't be relevant to anything.) So how does the runtime know how to convert an arbitrary Vehicle into a Craft?

If you initialize b := Car{} instead, then you get panic: interface conversion: main.Car is not main.Craft: missing method Float at runtime — which is exactly what I would expect — but, I don't understand how the runtime figured that out. Does the _type structure for Car contain a list of the names and signatures of every one of its methods, and then the interface-conversion code walks through that whole list at runtime to collect the needed methods to populate a Craft itab (or panic)?

In C++ terms, your Eface.(uint32) example is simply a std::any_cast — relatively cheap — but the vface.(Craft) example seems much wilder, much more dynamic and costly — so costly that it can't be done at all in C++. Is that right?

meta: chinese translation?

Hello, i am a native chinese speaker, and i am devoted to studying Golang also, and found that there is
little information about golang internals. your project seems like get this point. So let me work with you.
Let more chinese programmer understanding golang better.

chapter2: about symbol addr and size

We can also use nm in binutils to dump symbol information

nm  --print-size ./iface |grep "go.itab.main.Adder,main.Mather"
00000000004767a0 0000000000000028 R go.itab.main.Adder,main.Mather

Though no difference ...

chapter1: Clarify "nop before call" paragraph

The section

The NOP instruction just before the CALL exists so that the prologue doesn't jump directly onto a
CALL instruction. On some platforms, doing so can lead to very dark places; it's common pratice to
set-up a noop instruction right before the actual call and land on this NOP instead.

has generated some questions on reddit and #performance on the Gophers Slack. Could you please expand on this more?

chapter1: objdump

Question: Whats the difference between the output provided by the -S compile switch to the one that go tool objdump provides?

chapter1: digits of 137438953482 in base 2

$ echo 'obase=2;137438953482' | bc
10000000000000000000000000000000001010
\_____/\_____________________________/
   32                             10

should be

$ echo 'obase=2;137438953482' | bc
10000000000000000000000000000000001010
\____/\______________________________/
  32                              10

chapter1: Frame pointer

Like most recent compilers, the Go tool suite always references argument and locals using offsets from the stack-pointer directly in the code it generates. This allows for the frame-pointer to be used as an extra general-purpose register on platform with fewer registers (e.g. x86).

But it doesn't, starting with 1.8 (or was it 1.7 or 1.9?) the compiler emits instructions to use the frame pointer for its intended purpose, to help dtrace-like debugging tools. Just like you say later in the chapter. Maybe I'm misunderstanding.

meta: set up gitbook

If required I would even like to help with a few PR's so that the book's structure conforms to that of gitbook. Although I think markdown should work right-away there too

chapter2 : interface Type-switches have sorted

That this reordering is a net win for us in this particular case is nothing but mere luck, AFAICT. In fact, if you take the time to experiment a bit with type-switches, especially ones with more than two cases, you'll find that the compiler always shuffles the cases using some kind of deterministic heuristics.
What those heuristics are, I don't know (but as always, I'd love to if you do).

Type-switches have sorted and runtime use a Binary search to find the result , so the time complexity is log2(n).
if necessary, i am glad to pull request to this repo about how the golang compile do

chapter1: about SP register

The Go compiler never generates instructions from the PUSH/POP family: the stack is grown or shrunk by respectively decrementing or incrementing the virtual stack pointer SP.

The asm code in this chapter is generated by the compiler, I think Go compiler(1.10) will not generate code that refer to virtual register SP, it should be the hardware SP.

Clarify "top of the stack"

It would be helpful to use different term than "top" to refer to the first "element" in the stack that would be popped off. Maybe, tip or head?

Here's a sentence that felt confusing:

"".b+12(SP) and "".a+8(SP) respectively refer to the addresses 12 bytes and 8 bytes below the top of the stack (remember: it grows downwards!).

From this sentence, I thought the stack looked like as follows (here the stack is growing downwards):

RETURN ADDRESS 0(SP)
Argument A 8(SP)
Argument B 12(SP)

However, the diagram in the book is as follows:

   |    +-------------------------+ <-- 32(SP)              
   |    |                         |                         
 G |    |                         |                         
 R |    |                         |                         
 O |    | main.main's saved       |                         
 W |    |     frame-pointer (BP)  |                         
 S |    |-------------------------| <-- 24(SP)              
   |    |      [alignment]        |                         
 D |    | "".~r3 (bool) = 1/true  | <-- 21(SP)              
 O |    |-------------------------| <-- 20(SP)              
 W |    |                         |                         
 N |    | "".~r2 (int32) = 42     |                         
 W |    |-------------------------| <-- 16(SP)              
 A |    |                         |                         
 R |    | "".b (int32) = 32       |                         
 D |    |-------------------------| <-- 12(SP)              
 S |    |                         |                         
   |    | "".a (int32) = 10       |                         
   |    |-------------------------| <-- 8(SP)               
   |    |                         |                         
   |    |                         |                         
   |    |                         |                         
 \ | /  | return address to       |                         
  \|/   |     main.main + 0x30    |                         
   -    +-------------------------+ <-- 0(SP) (TOP OF STACK)

Looking at this diagram I see that the top of the stack refers to the bottom of the stack, but it might be nice to use different terminology.

Arguments passing conventions

First of all, great work @teh-cmc . I really enjoy your book, it help me to understand Go better.
Since version 1.17, Go has changed its argument passing convention from solely stack based to register based Go 1.17 release note.
Could you consider update your book or insert a warning to notice readers about this change?
Thanks.

chapter2: switch O(n)

I actually got surprised that type switches are O(N).

"how else could it work?"

I would assume that since the list of hashes is known at compile time, the compiler could find a perfect hashing function and just do a few arithmetic ops to get the address on which to jump1. At least if the size of switch exceeds some threshold.

If finding perfect hash would be too time-consuming, it could at least sort the hashes and do a binary search.

I guess the answer is that this is one of the optimizations Go is still awaiting...

EDIT: 1 Actually it would need to get just an index and lookup the target address; compiler knows the list of target hashes but not all input hashes.

chapter2: How you can get duplicated go.itab interface definitions

You asked why go.itabs are dupok. I believe the fundamental reason is that Go can be statically creating iface structures outside of either the package that defines the type or the package that defines the interface.

To make this concrete, here's a scenario where I believe that Go has to create duplicate ones. Suppose that we have four packages, A, B, C, D, and a main package. A defines a concrete type T, B defines an interface I (and perhaps some functions that take it), and both C and D import A and B and statically create B.I instances from A.T instances. Now we use both C and D in our main program.

Package A and B don't know about each other, so neither can define the iface<B.I, A.T> structure. Since C and D may be used by themselves, each must define this separately; they both need it and they have no guaranteed source of it outside themselves. Then when we use both of them together in our main program, we have duplicate definitions of iface<B.I, A.T>, one from each package, so Go must make such duplicates harmless.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.