Skip to Content

From the Virtyx Blog

Cross Compiling Go in Docker

By Ethan Mick • January 11, 2018

At Virtyx, our Agent is written in Golang and runs on Linux, Windows, and macOS. We’re really happy with Go’s small footprint, cross compatibility, and standard library. While there are a few things we’d love to see (better error handling!), we’re excited to keep writing our client agent in Go.

From day one, we’ve been cross compiling the Agent to run on all three major OS’s, 32 and 64-bit. To start, we used gox, which gave us easy cross compilation with simple pass through options. For example, to compile our Agent:

gox -os="linux windows darwin" -arch="amd64 386" -output="agent_{{.OS}}_{{.Arch}}" .

The {{.OS}} and {{.Arch}} are automatically replaced with the compiled values. A nice benefit of gox is also that the binaries are compiled in parallel!

However, recently we’ve expanded our cross compilation to include ARM (Hello, Raspberry Pi!), and this complicated the process. gox doesn’t support the ARM options, so we’d be falling back to go build. Rather than just tack on additional build commands (to the agent and all our plugins), we took this opportunity to refactor our Makefile.

We have been using Docker extensively for our tests and local builds. Some of our more complicated integration tests require external files to be present. Running the tests in a Docker container ensures we have complete control over the environment and those files are present. Docker also helps with stability and reliability as we can ensure the exact same environment between engineers. Since we’ve already been using Docker as such a major part of our development environment, we decided to cross compile our code inside a docker container as well!

That’s when we ran into an issue.

The cross compiling appeared to work perfectly, all binaries built successfully. But upon actually trying to run them, the Linux-amd64 binary would not run. The 32-bit version worked fine, and all other builds worked. Only the Linux-amd64 binary was broken. The error was:

-su: agent: No such file or directory

(Note: I was running the command as root, and the name of the binary was agent).

Okay – that doesn’t make any sense. I’m executing the binary, it’s clearly present. The error message isn’t clearly explaining what is going on. And that makes searching for a solution tricky. What’s going on?

After analyzing the new binary, we realized that it was actually missing a dependency:

brian@local:~/agent$ ldd previous-version/agent => (0x00007ffe8a7ce000) => /lib/x86_64-linux-gnu/ (0x00007f9065445000) => /lib/x86_64-linux-gnu/ (0x00007f906507b000)
  /lib64/ (0x00007f9065662000)

brian@local:~/agent$ ldd new-version/agent => (0x00007fff14bc5000) => not found

Wait a second… I thought Go built a static binary! You know, the kind where all dependencies are bundled in! That’s why deploying Go is so easy, just toss a binary on a host and start it up. Well, not quite. Go still builds binaries that depend on system libraries that are present. You can build perfect static binaries, but some of Go’s functionality requires cgo, and that requires the system libraries. However, these libraries are (almost always) present on modern systems. And in many cases, the only way to use the libraries is by dynamically linking against the system library. So why is our binary freaking out? It must have something to do with the changes we made in our compilation step, since we only just started having this issue. And the build command barely changed, the only other thing we changed was…. Docker.

Ah yes. We moved our compilation inside a Docker container. And not just any container – Alpine Linux. By default, we use Alpine Linux as our base image for all Docker containers. It’s small, efficient, lightweight, and for most applications, does a great job. In this case though, Alpine Linux has an important difference from other distributions. It uses a lightweight version of libc called musl-libc. This turned out to be the crux of the issue – the resulting binary was built against musl-libc, but on Ubuntu/Debian it found the standard glibc, and would crash with that cryptic error message.

We fixed this by changing our build Docker container to use Debian instead of Alpine Linux as it’s root image. Since only our build server needed to hold the image, we decided this was the fastest and best solution. It also better matches the environment in which the Agent most frequently runs.

In the end, we are still very happy with our usage of Docker and Go. It’s frustrating when seemingly innocuous changes cause bugs, but it’s a good reminder to test everything. We’re always looking to improve our development practices and help out others in the community!

If you’re interested, sign up for Virtyx today, or follow us on Twitter for more great posts!

Come join our team - we're hiring!

Get Started for Free
Read More

By Ben Burwell • October 22, 2018

By Ben Burwell • October 19, 2018

By Jim Maniscalco • September 27, 2018

By Ethan Mick • September 20, 2018

By Jim Maniscalco • September 14, 2018

Start using Virtyx today.

If you manage computers or servers, Virtyx can make your life easier.