Go Modules have a v2+ Problem
- Comments:
- 9
Go has a problem. Go modules place a strange naming requirement on modules version 2 or greater. Module names on modules v2+ must end in the major version ala …/v2, and communication of this rule has been weak. It's non-obvious, and the community at large does not understand it.
I have seen many very large projects including Google owned projects get it wrong.
I brought the issue up at my local Go meetup, and no one had ever heard about the rule. They were very skeptical the rule existed at all.
A little history
For a long time Go did not include a built in method for versioning dependencies. It did include the sometimes controversial go get method of fetching packages. The go get tool until recently simply cloned the HEAD of a packages primary branch based on URL and placed it in your $GOPATH.
In what has been oft linked to Go's "extremist position" on backwards compatibility, the expectation was largely that if you published a library, you were expected to maintain compatibility when at all possible.
A plethora of tools popped up over the years to fill this void, gopkg.in, glide, and even the at one time "official experiment" dep.
dep was the heir apparent to dependency management with backing from Google.
Seemingly suddenly however Russ "rsc" Cox sprung vgo (aka Go Modules) onto the world as the official Go solution. There was seemingly no input from the community in its creation, to the chagrin of many.
Whereas dep was a largely standard dependency manager in the vein of npm and the like, Go modules is a decidedly opinionated "Go" solution. It's handled as part of the language as a build step. When you go build if you are missing the versioned dependency, the build tool will fetch it.
The dependencies requirements are listed in go.mod and the exact versions expected are in go.sum. Both files are in their own sort of domain specific language. The prior has a syntax reminiscent of Go itself. The latter is a space separated list of dependency versions largely not intended for human consumption.
The build step will add to these files as necessary, and running go mod tidy will clean them up.
The v2+ problem in detail
Go modules place a very strange requirement on package developers. When a module hits major version 2 or higher, the module name must end in the major version. The advantage to this is it creates a separate package. The result is that a project can now depend on multiple major versions of the same library.
There are two tactics to achieving /v2 modules:
The first method is to change the module name in your go.mod
An example may seen
in the go.mod of github.com/google/go-github seems simple enough, but requires you to also change any cross references within your package. While not the end of the world, it is error prone and even Google missed some.
The biggest problem with this method is that it breaks subpackages in non-module-aware versions of Go. As time goes by this becomes less important, but in older versions of Go the package name had to exactly match it's location. https://github.com/pkg/foo could not be github.com/pkg/foo/v2 when there is no v2 directory. The Go Team acknowledging this trouble and patched rudimentary module support into point releases 1.9.7 and 1.10.3 of the Go language to help ease the transition.
The second method and the method the Go Team themselves promote is leaving your v1 project as is in the root of your repo, and actually creating a v2/ subfolder containing your v2 library.
This feels weird, but the advantage is it works cleanly with older versions of Go that do not understand modules. Off the top of my head I only knew of a single project that went this route, and they appear to have gone back on it.
For what it's worth, the official Go Blog tried to clear the situation up some, and posted a write-up about it.
Even as a seasoned developer, I found this write-up somewhat impenetrable. For how important and unusual of a requirement this is, the write-up is not accessible enough.
Many projects including Buildkite's terminal-to-html simply didn't know about the requirement until they forget to handle it, and end up publishing broken releases. A v2+ Go library with a go.mod file, but not respecting the module v2+ rules is a compile time error for dependants. This is worse for users than not being a Go module at all.
Some projects like GORM sidestep the issue entirely by tagging their 2.0 releases as far flung 1.x releases. That something of a solution, but smells terrible.
What's to be done?
First, I think the Go Team needs to do a better job of shining a light on this rule. They need to scream it from the rooftops. They need at the very least a section of the documentation that lays it out clearly and in layman's terms.
Beyond this, there's room for warnings from the Go tooling. Subpackages importing older versions of the root package should throw a warning. It's way too easy an issue to miss.
There's room for similar notes on godoc.org / pkg.go.dev, although the usefulness is questionable.
The best and least likely option would be for Go to drop the requirement and make it optional. I believe this is a workable solution. Packages who want to allow users to include multiple versions can follow the v2 guidelines, and others don't need to.
An option I would personally promote, but is likely never going to happen, is to require /v0 and /v1 as well, such that you'd hit the issue right away rather than years into a project. Teach the convention early.
Discussion at: news.ycombinator.com/item?id=24429045
Nice work on shinning the light on an often overlooked issue
Comment by: Milos on
Agreed. I was unaware of this until I saw some "+incompatible" verisions in my go.mod. After little investigation I found out about this rule.There are some packages which were tagged as v2+ even before go modules intruduced and if they haven't implemented somo kind of fix you mentioned, those packages are all "+incompatible".
It would be OK that we had that rule from the beginning in Go.
Comment by: Tristan Hyams on
I have adopted the v1 (not v0) approach for this.Also the entire point is to prevent people from upgrading automatically to a breaking API change... but nothing stops a developer from actually breaking the API change within v1.
It's poorly conceived idea and it really illustrates that too many smart people are working on this. One moron, like myself in the room, would have pointed out it doesn't stop the thing they think it is.
Just makes it a nightmare as I discovered when doing a rewrite and bumping up the v2.x.x like a moron thinking it would just work.
Comment by: Jose Luis Vázquez González on
Naming is hard, figuring out proper APIs right the first time is hard.The Go team seems to have clearly screwed up this one, as it is very well explained here. But the underlying issues that plague dependency systems remains.
Dependencies are a balance between convenience and liability.
Dependency systems like npm, much maximise ease of dependency handling at the cost of huge maintenance and security headaches later on are not the way.
Maybe the Go approach for dependencies is not the way either, but at least go developers have grown a bit more weary of taking deps is if they were for free.
A good rule to follow when you publish something is NEVER to break compatibility. If you ever do, for a small thing, it has to be for VERY good reasons (security comes to mind) and should come with automation tooling to help with the transition.
And if you need a revamp or you think you figured out the right API this time, and can just leave the old stuff around in the same project... fork it with a different name, because you know, it is different!
Also never change semantics under a given public call. Changing semantics will break consumers, so a new name is needed.
A way to trip yourself the least is to make the API as small as possible, exposing as few and small public interfaces as possible. For languages that don't have proper access rules (eg. Python) feel free to break consumers that do NOT follow the access conventions (eg. - prefixes in Python).
For consumers:
- NEVER ever take binary dependencies that you can't rebuild or patch if the producer drops support.
- Measure your dep graph.
- Keep the deps up to date with automation. This will make you aware of their liability while saving you from being forced to upgrade later on when it is most inconvenient for you.
- Sometimes, just copy some code in, it would be best.
It would be GREAT to have dep analytic tooling that tells you stats and ratings on your deps like:
- How many deps you are polling.
- How much code on your deps as a whole or per dep you are actually using.
- A dep score rating in security or maintenance hassle based upon number of CVEs that dep suffered due to itself or its deps, or the number of times it broke builds for consumers.
Comment by: Jose Luis Vázquez González on
Sorry for the many typos in my post, hope the text is still understandable.
Comment by: Damien on
It predates go module when we only had GOPATH and packages couldn't have breaking changes without changing the import path.
Huh, I didn't realize this. This is great information, thank you. I will need to ingest some more information on this and update the post.
It seems that you've outlined the problem perfectly. I've attempted to outline the solution (how to bump a package to v2 modules, in layman's terms):