Go's New Iterators Smell (A Little) Funny, but It's Probably OK
- Comments:
- 2
Go's new Iterators were released on August 13th with Go 1.23.
They've had just a minute to marinate. In some parts of the community they're unpopular. I don't particularly mind them. The old alternatives seem a little more "Go" to me, but the new iterators are largely fine.
I am mildly disappointed there isn't just a generic iterable type similar to the generic comparable — the imaginary iterable type being just a generic type that can accept maps, slices, arrays, and iterators. As I understand it, iterators combined with slices.All and maps.All are the blessed alternative to that. I'm not a huge fan, but it works well enough for those purposes.
All that's neither here nor there, however, and not what I'm here to talk about.
The part that smells funny to me is that range now accepts any already existing function that happens to fulfill either of the following interfaces.
func(yield func(any) bool)
func(yield func(any,any) bool)It would be less weird if that wasn't also somewhat troublesome.
Example time!
The first interface, known as Seq takes a yield method as an argument that itself accepts any single parameter. The second interface, known as Seq2 similarly takes a yield method that accepts any two parameters as essentially a key/value tuple.
Note that yield is not a magic keyword as in many languages, but a passed-in callable. This is very akin to Go allowing you to name receivers rather than magically defining this.
At first blush to the uninitiated, this iterator looks fine. It seemingly runs fine as well.
package main
import "fmt"
func mightBeAnIterator(f func(string) bool) {
f("foo")
f("bar")
f("baz")
}
func main() {
for x := range mightBeAnIterator {
fmt.Println(x)
}
}
foo
bar
bazWhat's the problem then?
The problem is iterators MUST handle early exit of the loop manually. With the current interface fulfilling "iterator", if we just add what seems a harmless break to our loop as follows, suddenly the code that was working moments ago panics.
for x := range mightBeAnIterator {
fmt.Println(x)
break
}foo
panic: runtime error: range function continued iteration after function for loop body returned falseSomewhat surprisingly, as Go exits the loop, it allows the iterator to continue executing. In doing so, however, it signals the iterator through the result of the yielding method that it should return. This is to allow for teardown. For example, imagine an iterator that opens files or a socket; continuing execution allows it to close them before returning.
However, Go will panic if the signal to return is not honored and the yield function is then invoked again by the iterator.
To prevent our example iterator from panic-ing, we simply have to listen for the signal to exit, as shown below.
func mightBeAnIterator(f func(string) bool) {
if !f("foo") {
return
}
if !f("bar") {
return
}
if !f("baz") {
return
}
}And when run, now we receive simply
fooI would imagine the reason for the panic is to prevent iterators from accidentally running wild.
To my personal taste, I would have strongly preferred honoring the exit signal to be optional, and subsequent superfluous yield calls being ignored and returning false.
I can certainly imagine cases where an iterator would want to finish regardless of early exits by its consumer. While still possible, it's awkward to implement.
We are certainly in a place where, should they decide to go back on the panic behavior, it would not break any existing code. That's not a terrible place to be.
All this is to say it smells a little funny that such a simple and likely somewhat common interface has this magic range logic available to it.
Expecting the implementers to implement it correctly and panic-ing if they don't smells kind of like a Worse is Better solution to me. While you could complicate Go's iterator interface to enforce stricter guarantees around safe iteration, doing so would add unnecessary complexity and detract from Go's philosophy of simplicity.
It's one of the rare occasions where I wish interfaces had to be intentionally met. It's strange to me that anything that smells like an iterator can be passed to range, even if it doesn't implement the intended handshake.
It smells funny, but that's probably OK. It almost certainly falls into the case of "things that will never be a real problem in the real world". I look forward to the oncoming brave new world of libraries expecting iterators.
Comment by: jimbvolker on
Create a reusable wrapper to make exit optional. go.dev/play/p/nLoktlbWDfG
func optionalExit[T any](yield func(T) bool) func(T) bool {
ok := true
return func(v T) bool {
if ok {
ok = yield(v)
}
return ok
}
}
Thanks, Jim. That is a pretty elegant solution!