26 March 2018

Introduction

Eight years ago, the Go team introduced goinstall (which led to go get) and with it the decentralized, URL-like import paths that Go developers are familiar with today. After we released goinstall, one of the first questions people asked was how to incorporate version information. We admitted we didn’t know. For a long time, we believed that the problem of package versioning would be best solved by an add-on tool, and we encouraged people to create one. The Go community created many tools with different approaches. Each one helped us all better understand the problem, but by mid-2016 it was clear that there were now too many solutions. We needed to adopt a single, official tool.

After a community discussion started at GopherCon in July 2016 and continuing into the fall, we all believed the answer would be to follow the package versioning approach exemplified by Rust’s Cargo, with tagged semantic versions, a manifest, a lock file, and a

SAT solver to decide which versions to use. Sam Boyer led a team to create Dep, which followed this rough plan, and which we intended to serve as the model for go command integration. But as we learned more about the implications of the Cargo/Dep approach, it became clear to me that Go would benefit from changing some of the details, especially concerning backwards compatibility.

The Impact of Compatibility

The most important new feature of Go 1 was not a language feature. It was Go 1’s emphasis on backwards compatibility. Until that point we’d issued stable release snapshots approximately monthly, each with significant incompatible changes. We observed significant acceleration in interest and adoption immediately after the release of Go 1. We believe that the promise of compatibility made developers feel much more comfortable relying on Go for production use and is a key reason that Go is popular today. Since 2013 the Go FAQ has encouraged package developers to provide their own users with similar expectations of compatibility. We call this the import compatibility rule: “If an old package and a new package have the same import path, the new package must be backwards compatible with the old package.”

Independently, semantic versioning has become the de facto standard for describing software versions in many language communities, including the Go community. Using semantic versioning, later versions are expected to be backwards-compatible with earlier versions, but only within a single major version: v1.2.3 must be compatible with v1.2.1 and v1.1.5, but v2.3.4 need not be compatible with any of those.

If we adopt semantic versioning for Go packages, as most Go developers expect, then the import compatibility rule requires that different major versions must use different import paths. This observation led us to semantic import versioning, in which versions starting at v2.0.0 include the major version in the import path: my/thing/v2/sub/pkg.

A year ago I strongly believed that whether to include version numbers in import paths was largely a matter of taste, and I was skeptical that having them was particularly elegant. But the decision turns out to be a matter not of taste but of logic: import compatibility and semantic versioning together require semantic import versioning. When I realized this, the logical necessity surprised me.

I was also surprised to realize that there is a second, independent logical route to semantic import versioning: gradual code repair or partial code upgrades. In a large program, it’s unrealistic to expect all packages in the program to update from v1 to v2 of a particular dependency at the same time. Instead, it must be possible for some of the program to keep using v1 while other parts have upgraded to v2. But then the program’s build, and the program’s final binary, must include both v1 and v2 of the dependency. Giving them the same import path would lead to confusion, violating what we might call the import uniqueness rule: different packages must have different import paths. The only way to have partial code upgrades, import uniqueness, and semantic versioning is to adopt semantic import versioning as well.

It is of course possible to build systems that use semantic versioning without semantic import versioning, but only by giving up either partial code upgrades or import uniqueness. Cargo allows partial code upgrades by giving up import uniqueness: a given import path can have different meanings in different parts of a large build. Dep ensures import uniqueness by giving up partial code upgrades: all packages involved in a large build must find a single agreed-upon version of a given dependency, raising the possibility that large programs will be unbuildable. Cargo is right to insist on partial code upgrades, which are critical to large-scale software development. Dep is equally right to insist on import uniqueness. Complex uses of Go’s current vendoring support can violate import uniqueness. When they have, the resulting problems have been quite challenging for both developers and tools to understand. Deciding between partial code upgrades and import uniqueness requires predicting which will hurt more to give up. Semantic import versioning lets us avoid the choice and keep both instead.

I was also surprised to discover how much import compatibility simplifies version selection, which is the problem of deciding which package versions to use for a given build. The constraints of Cargo and Dep make version selection equivalent to solving Boolean satisfiability, meaning it can be very expensive to determine whether a valid version configuration even exists. And then there may be many valid configurations, with no clear criteria for choosing the “best” one. Relying on import compatibility can instead let Go use a trivial, linear-time algorithm to find the single best configuration, which always exists. This algorithm, which I call minimal version selection, in turn eliminates the need for separate lock and manifest files. It replaces them with a single, short configuration file, edited directly by both developers and tools, that still supports reproducible builds.

Our experience with Dep demonstrates the impact of compatibility. Following the lead of Cargo and earlier systems, we designed Dep to give up import compatibility as part of adopting semantic versioning. I don’t believe we decided this deliberately; we just followed those other systems. The first-hand experience of using Dep helped us better understand exactly how much complexity is created by permitting incompatible import paths. Reviving the import compatibility rule by introducing semantic import versioning eliminates that complexity, leading to a much simpler system.

Progress, a Prototype, and a Proposal

Dep was released in January 2017. Its basic model—code tagged with semantic versions, along with a configuration file that specified dependency requirements—was a clear step forward from most of the Go vendoring tools, and converging on Dep itself was also a clear step forward. I wholeheartedly encouraged its adoption, especially to help developers get used to thinking about Go package versions, both for their own code and their dependencies. While Dep was clearly moving us in the right direction, I had lingering concerns about the complexity devil in the details. I was particularly concerned about Dep lacking support for gradual code upgrades in large programs. Over the course of 2017, I talked to many people, including Sam Boyer and the rest of the package management working group, but none of us could see any clear way to reduce the complexity. (I did find many approaches that added to it.) Approaching the end of the year, it still seemed like SAT solvers and unsatisfiable builds might be the best we could do.

In mid-November, trying once again to work through how Dep could support gradual code upgrades, I realized that our old advice about import compatibility implied semantic import versioning. That seemed like a real breakthrough. I wrote a first draft of what became my semantic import versioning blog post, concluding it by suggesting that Dep adopt the convention. I sent the draft to the people I’d been talking to, and it elicited very strong responses: everyone loved it or hated it. I realized that I needed to work out more of the implications of semantic import versioning before circulating the idea further, and I set out to do that.

In mid-December, I discovered that import compatibility and semantic import versioning together allowed cutting version selection down to minimal version selection. I wrote a basic implementation to be sure I understood it, I spent a while learning the theory behind why it was so simple, and I wrote a draft of the post describing it. Even so, I still wasn’t sure the approach would be practical in a real tool like Dep. It was clear that a prototype was needed.

In January, I started work on a simple go command wrapper that implemented semantic import versioning and minimal version selection. Trivial tests worked well. Approaching the end of the month, my simple wrapper could build Dep, a real program that made use of many versioned packages. The wrapper still had no command-line interface—the fact that it was building Dep was hard-coded in a few string constants—but the approach was clearly viable.

I spent the first three weeks of February turning the wrapper into a full versioned go command, vgo; writing drafts of a blog post series introducing vgo; and discussing them with Sam Boyer, the package management working group, and the Go team. And then I spent the last week of February finally sharing vgo and the ideas behind it with the whole Go community.

In addition to the core ideas of import compatibility, semantic import versioning, and minimal version selection, the vgo prototype introduces a number of smaller but significant changes motivated by eight years of experience with goinstall and go get: the new concept of a Go module, which is a collection of packages versioned as a unit; verifiable and verified builds; and version-awareness throughout the go command, enabling work outside $GOPATH and the elimination of (most) vendor directories.

The result of all of this is the official Go proposal, which I filed last week. Even though it might look like a complete implementation, it’s still just a prototype, one that we will all need to work together to complete. You can download and try the vgo prototype from golang.org/x/vgo, and you can read the Tour of Versioned Go to get a sense of what using vgo is like.

The Path Forward

The proposal I filed last week is exactly that: an initial proposal. I know there are problems with it that the Go team and I can’t see, because Go developers use Go in many clever ways that we don’t know about. The goal of the proposal feedback process is for us all to work together to identify and address the problems in the current proposal, to make sure that the final implementation that ships in a future Go release works well for as many developers as possible. Please point out problems on the proposal discussion issue. I will keep the discussion summary and FAQ updated as feedback arrives.

For this proposal to succeed, the Go ecosystem as a whole—and in particular today’s major Go projects—will need to adopt the import compatibility rule and semantic import versioning. To make sure that can happen smoothly, we will also be conducting user feedback sessions by video conference with projects that have questions about how to incorporate the new versioning proposal into their code bases or have feedback about their experiences. If you are interested in participating in such a session, please email Steve Francia at [email protected]

We’re looking forward to (finally!) providing the Go community with a single, official answer to the question of how to incorporate package versioning into go get. Thanks to everyone who helped us get this far, and to everyone who will help us going forward. We hope that, with your help, we can ship something that Go developers will love.