MDB support for Go
Last week was Joyent Engineering’s inaugural hackathon, where we spent a couple of days together in San Francisco working in small teams on projects which interested us and were at least in some way relevant to our business.
Whilst the obvious choice for me would have been to work on something which used Manta for pkgsrc analysis (no shortage of ideas or potential there), I wanted to take advantage of being in the same room as my illustrious co-workers and work on something a bit more low-level and personally challenging.
For the past few months Aram Hăvărneanu has been doing some amazing work porting Go to SunOS, and we wanted to help out. So, while Bryan and Josh got stuck in with Go core and Brendan started cooking up some Go-related DTrace, I ventured into MDB land to add support for Go stack traces.
C Stack Traces
Let’s start by looking at a contrived example in C. Here is the code:
Compile it with no optimisations to ensure we do not lose any functions to
-g isn’t strictly necessary for these examples, but is good
Now we can show a stack trace using MDB. Comments inline:
From the output we can match up a few things:
The instruction pointer
%ripis set to our break point, and you can see both the real (
0x401078) and the symbolic (
The arguments to
printf()are passed in the registers, with
%rsicontaining the integers we are printing, and
%rdicontaining a pointer to the string “
%d -> %d” which we can see using
The stack contains arguments (
6), instruction pointers (
4010cc) and frame pointers (
If you are wondering where I am getting those addresses from, note that this is on x86 which is little-endian, and so you need to read each byte backwards - that is, if you have a dump line which contains a 64-bit value, a 32-bit value, a 16-bit value and two 8-bit values, then:
- 0xfffffd7fffdff378 (64-bit)
- 0xffdff378 (32-bit)
- 0x0702 (16-bit)
- 0x41 (8-bit)
- 0x0d (8-bit)
Playing around with MDB like this was super helpful for me to get a visual representation of memory, and how functions and arguments are passed around.
Thanks to the frame pointers being saved on the stack, we are able to easily get useful stack traces by simply following them and printing each frame in turn.
At this point it’s worth mentioning the infamous GCC argument
-fomit-frame-pointer. If you are of a certain age, you may remember some
Linux distributions using this flag in an attempt to make programs faster.
What happens is that instead of using the frame pointer to record valuable
information about where we came from, it is instead used as a general purpose
register, and the stack information is lost.
This can be verified pretty easily:
That’s all we have.
%rbp is no longer used to record the previous
and so after printing the current instruction, we have nowhere else to go, and
the stack is useless.
Friends don’t let friends go without their frame pointers. There is very little to suggest that the extra register provides any performance benefits, and the cost is way too high. Just Say No.
Go Stack Traces
So, let’s look at Go. One of the nice things about Go is that it compiles programs down to a self-contained binary, which can then be copied around and executed. This may lead you to think that stack traces will simply work, as there is no dynamic runtime stuff going on. Well, let’s see.
Let’s start with a comparative program to the one above, written in Go.
Compile it and start MDB.
The first thing we notice is that we do have symbols, which is good! We can
also show the first few instructions of our
However, note that the frame pointer is never saved - the function just goes right ahead and tries to allocate more stack. And, indeed, if we set a breakpoint and run the program, we do not get a useful stack trace.
The problem is that Go uses a completely different calling convention, so the usual method of following frame pointers does not work. At this point we would normally be stuck, however Go does provide enough information in the binary for us to calculate the stack in a different way. We just need to dig it out.
First, let’s take a look to see what the stack does contain. I deliberately
set a breakpoint to the start of the
printout() function so that the stack
did not contain any local variables for that function.
A few things stood out here. Knowing the function arguments in advance allows
us to spot the various
1 integers being passed, so we can take a
guess that the other addresses refer to functions, and we can confirm this with
So, we can dig through the stack and find functions and arguments. However, because functions can have differing argument lengths, we can’t step over a fixed size - we need to know how many arguments a function takes, so that we know how much of the stack to skip in order to get to the next function.
Thankfully, Go provides us with the answer, in the form of the pclntab.
The pclntab is written into the binary by the Go linker, and contains useful information about each function. The format is described in this document for Go 1.2, which is the version we are targetting.
Here is the beginning of the pclntab:
If you read the document above you can see and compare the pclntab header
format - we start with the magic number
0xfffffffb, followed by
0x01 for the instruction size quantum and
0x08 for the size of a
pointer (both accurate for 64-bit x86). After that we have
0x064d as the
64-bit size of the function symbol table.
After the header we have pairs of function addresses to function offsets within
the pclntab. So, for example, we start with
0x400c00, for which the
information is stored at offset
0x64f0 within the pclntab. Decoding the data
at this offset gives us the details we need.
At this point we need to dig through the Go source code to find how this information is stored, and thankfully we can mostly lift it directly into our MDB module.
The current version of the MDB module is here, you may want to follow along there.
We can start by transplanting the layouts for the pclntab and Go function information.
Once we have that, we can do the following:
Start by loading the pclntab and checking it is valid in
configure(). If it is good, we store its location and size.
Get the previous instruction from the top of the stack. In our example this is
0x400cff, and is done using
load_current_context()which reads the current value of
Perform a binary search through the
go_functbl_tentries for that address. If our address is larger than the current lookup but less than the next one, we have a match. This is implemented in
Read in the function information into our
pcvalue()and the information stored at our function offset.
This is implemented with the
::goframe dcmd, which we can use after loading
Note that we have some additional useful information here, including the
filename and the line number. I implemented a
-p name argument which pretty
prints this information:
The final part of the puzzle is to dig out the argument length and skip over
that size, so that we can get to the next function. This is pretty
straight-forward, we just skip over the size of the
With this in place we can implement an MDB “walker”, which walks our stack and
prints each address. MDB makes this really easy, and is implemented in
walk_goframes_step, which simply describe how to
find the first frame and then how to find each subsequent frame.
And, for the finale, we show the beautiful modularity of MDB by simply piping
those addresses to the
And hey presto, we have a stack trace.
So, as you may have noticed, I cheated a little by setting a break point to the
start of the
printout() function. This allows us to print a full stack
trace, including the current instruction, using the
However, if we set the break point to the point where we call
We are unable to walk the stack, and the arguments to
incorrect, due to
main.printout() allocating a bunch of the stack for its
local variables which messes up our offsets.
I need to figure out how to calculate these reliably, plus there are a bunch of cleanups to do in the module. It’s also likely that there are lots of edge cases where things will break.
It’s also likely that the format of the symbol tables will change, so we will need to track changes upstream.
Finally, the most useful work we can do based on this initial implementation
would be to translate the same information into a DTrace ustack helper. This
would allow us to dynamically instrument running Go programs, and do all sorts
of useful performance and debugging analysis. The information is all there,
and we can in theory get to it using
uregs, but there are a number of
challenges to overcome first, most notably trying to hook into the Plan9 linker
which currently rejects DTrace ELF sections. Help in this area would be
Hopefully this was a useful introduction to MDB and how to implement support for esoteric languages. I welcome any feedback and improvements.
- 16 Jul 2015 » Reducing RAM usage in pkgin
- 03 Mar 2015 » pkgsrc-2014Q4: LTS, signed packages, and more
- 06 Oct 2014 » Building packages at scale
- 04 Dec 2013 » A node.js-powered 8-bit CPU - part four
- 03 Dec 2013 » A node.js-powered 8-bit CPU - part three
- 02 Dec 2013 » A node.js-powered 8-bit CPU - part two
- 01 Dec 2013 » A node.js-powered 8-bit CPU - part one
- 21 Nov 2013 » MDB support for Go
- 30 Jul 2013 » What's new in pkgsrc-2013Q2
- 24 Jul 2013 » Distributed chrooted pkgsrc bulk builds
- 07 Jun 2013 » pkgsrc on SmartOS - creating new packages
- 15 Apr 2013 » What's new in pkgsrc-2013Q1
- 19 Mar 2013 » Installing SVR4 packages on SmartOS
- 27 Feb 2013 » SmartOS is Not GNU/Linux
- 18 Feb 2013 » SmartOS development preview dataset
- 17 Jan 2013 » pkgsrc on SmartOS - fixing broken builds
- 15 Jan 2013 » pkgsrc on SmartOS - zone creation and basic builds
- 10 Jan 2013 » Multi-architecture package support in SmartOS
- 09 Jan 2013 » Solaris portability - cfmakeraw()
- 08 Jan 2013 » Solaris portability - flock()
- 06 Jan 2013 » pkgsrc-2012Q4 illumos packages now available
- 23 Nov 2012 » SmartOS and the global zone
- 24 Oct 2012 » Setting up Samba on SmartOS
- 10 Oct 2012 » pkgsrc-2012Q3 packages for illumos
- 23 Aug 2012 » Creating local SmartOS packages
- 10 Jul 2012 » 7,000 binary packages for OSX Lion
- 09 Jul 2012 » 9,000 packages for SmartOS and illumos
- 07 May 2012 » Goodbye Oracle, Hello Joyent!
- 13 Apr 2012 » SmartOS global zone tweaks
- 12 Apr 2012 » Automated VirtualBox SmartOS installs
- 30 Mar 2012 » iptables script for Debian / Ubuntu
- 20 Feb 2012 » New site design
- 11 Jan 2012 » Set up anonymous FTP upload on Oracle Linux
- 09 Jan 2012 » Kickstart Oracle Linux in VirtualBox
- 09 Jan 2012 » Kickstart Oracle Linux from Ubuntu
- 22 Dec 2011 » Last day at MySQL
- 15 Dec 2011 » Installing OpenBSD with softraid
- 21 Sep 2011 » Create VirtualBox VM from the command line
- 14 Sep 2011 » Creating chroots for fun and MySQL testing
- 30 Jun 2011 » Graphing memory usage during an MTR run
- 29 Jun 2011 » Fix input box keybindings in Firefox
- 24 Jun 2011 » How to lose weight
- 23 Jun 2011 » How to fix stdio buffering
- 13 Jun 2011 » Serving multiple DNS search domains in IOS DHCP
- 13 Jun 2011 » Fix Firefox URL double click behaviour
- 20 Apr 2011 » SSH via HTTP proxy in OSX
- 09 Nov 2010 » How to build MySQL releases
- 29 Apr 2010 » 'apt-get' and 5,000 packages for Solaris10/x86
- 16 Sep 2009 » ZFS and NFS vs OSX
- 12 Sep 2009 » pkgsrc on Solaris
- 09 Dec 2008 » Jumpstart from OSX
- 31 Dec 2007 » Set up local caching DNS server on OSX 10.4