[swift-evolution] [Proposal] Change guarantee for GeneratorType.next() to always return nil past end

[Kevin Ballard]

On Sat, Mar 5, 2016, at 12:47 PM, Brent Royal-Gordon wrote:
> > Yes. My proposed .takeWhile() and .dropWhile() sequence adaptors (https://github.com/apple/swift-evolution/pull/95) would hit this case. Both of those adaptors would need to keep around extra state and in order to keep returning nil.
> 
> 
> How much extra state? If it's merely "an extra Bool" or "wrap the inner generator in an Optional and nil it when you hit the end", it might be worth it.
> 
> (Actually, can we see the implementations you're thinking of using? Because the ones I can imagine for `dropWhile`, at least, would keep returning `nil` as long as the underlying generator did.)

Oh you're right, it's just takeWhile() that would actually have usable post-nil behavior. dropWhile() would end up with the exact same post-nil behavior as its underlying generator.

> > All that said, I would be strongly in favor of dropping the language about triggering a precondition failure. I'd prefer to leave it as implementation-defined behavior, which an encouragement to keep returning nil if it's easy to do so.
> 
> I think "try to return `nil` but it's okay if you can't" is the worst of all worlds from a reliability standpoint. If most, but not all, generators continue to return `nil`, then people will write code assuming that all generators continue to return `nil`.

Most people don't write code that relies on the post-nil behavior of generators at all. Heck, most people don't even deal with generators directly, they deal with sequences, and sequences already have implementation-defined behavior of calling .generate() twice.

In fact, I don't think the post-nil behavior of generators is even the biggest potential gotcha. A much bigger issue is how generators behave when you make copies of them. If you ever copy a generator you're supposed to never touch the original again, but it's easy to violate that accidentally, and most generators actually behave sanely when copied anyway.

> One thing that might help is to make sure we have examples of the full range of generator behaviors—multi-pass and single-pass, nil forever and not, etc.—in the standard library and list an example of each in the SequenceType documentation along with an encouragement to test with all of them. But a lot of people will miss that.

I agree that having more examples is a good thing.

> (Another reason to lean towards always returning `nil`: it's testable. In fact, I was thinking it might be nice to have something in XCTest where you could hand it a Sequence, or perhaps a closure that creates one, and an array containing the values you expect it to have, and it would assert the basic properties we expect from a Sequence, like that the elements come out in the right order, it returns nil at the end, `underestimateCount()` is not incorrect, etc. The `nil` semantic, unlike the other possible ones, is something you could include in that test.)

Defining the post-nil behavior as implementation-defined means that, by definition, you don't need to test how arbitrary generators behave there. For specific concrete GeneratorTypes that define their post-nil semantics (and I think we'd want to try and define this for all of the stdlib GeneratorTypes) you can test the documented semantics. For any generator that defines it as returning nil forever (which I'd expect e.g. IndexingGenerator to do) you can easily test that semantic. And for generators like TakeWhileGenerator that effectively reset their behavior when returning nil, you can test that behavior too (e.g. `(1..<10).takeWhile({ $0 % 4 != 0 })` would return the sequences `[1,2,3]`, `[5,6,7]`, `[9]`, and then start returning nil forever).

Besides, you can't actually test that a given GeneratorType returns nil forever, you can only test that it returns nil for some finite number of calls to next(). It would be entirely possible to have a GeneratorType whose behavior depends on some external signal and will return nil until such time as that external signal has data again. For example, you could have a GeneratorType that reads lines from stdin in a non-blocking fashion, and so it would return nil until the user has typed another line of text. Or you could have a GeneratorType that represents an atomic FIFO queue and returns nil if the queue is empty.

-Kevin Ballard