[swift-evolution] [swift-evolution-announce] [Review] SE-0089: Replace protocol<P1, P2> syntax with Any<P1, P2>

Matthew Johnson matthew at anandabits.com
Thu Jun 9 12:26:36 CDT 2016

> On Jun 9, 2016, at 11:42 AM, Dave Abrahams <dabrahams at apple.com> wrote:
> on Thu Jun 09 2016, Matthew Johnson <matthew-AT-anandabits.com <http://matthew-at-anandabits.com/>> wrote:
>>> On Jun 9, 2016, at 9:55 AM, Dave Abrahams <dabrahams at apple.com <mailto:dabrahams at apple.com>> wrote:
>>> on Wed Jun 08 2016, Matthew Johnson <matthew-AT-anandabits.com <http://matthew-at-anandabits.com/> <http://matthew-at-anandabits.com/ <http://matthew-at-anandabits.com/>>> wrote:
>>>>> On Jun 8, 2016, at 1:33 PM, Dave Abrahams <dabrahams at apple.com <mailto:dabrahams at apple.com>> wrote:
>>>>> on Tue Jun 07 2016, Matthew Johnson <matthew-AT-anandabits.com <http://matthew-at-anandabits.com/>> wrote:
>>>>>>> On Jun 7, 2016, at 9:15 PM, Dave Abrahams <dabrahams at apple.com <mailto:dabrahams at apple.com>> wrote:
>>>>>>> on Tue Jun 07 2016, Matthew Johnson <matthew-AT-anandabits.com <http://matthew-at-anandabits.com/> <http://matthew-at-anandabits.com/ <http://matthew-at-anandabits.com/>>> wrote:
>>>>>>>>> On Jun 7, 2016, at 4:13 PM, Dave Abrahams via swift-evolution <swift-evolution at swift.org <mailto:swift-evolution at swift.org>> wrote:
>>>>>>>>> on Tue Jun 07 2016, Matthew Johnson <swift-evolution at swift.org <mailto:swift-evolution at swift.org>> wrote:
>>>>>>>>>>> , but haven't realized
>>>>>>>>>>> that if you step around the type relationships encoded in Self
>>>>>>>>>>> requirements and associated types you end up with types that appear to
>>>>>>>>>>> interoperate but in fact trap at runtime unless used in exactly the
>>>>>>>>>>> right way.
>>>>>>>>>> Trap at runtime?  How so?  Generalized existentials should still be
>>>>>>>>>> type-safe.  
>>>>>>>>> There are two choices when you erase static type relationships:
>>>>>>>>> 1. Acheive type-safety by trapping at runtime
>>>>>>>>> FloatingPoint(3.0 as Float) + FloatingPoint(3.0 as Double) // trap
>>>>>>>>> 2. Don't expose protocol requirements that involve these relationships,
>>>>>>>>> which would prevent the code above from compiling and prevent
>>>>>>>>> FloatingPoint from conforming to itself.
>>>>>>>>>> Or are you talking about the hypothetical types / behaviors people
>>>>>>>>>> think they want when they don’t fully understand what is happening...
>>>>>>>>> I don't know what you mean here.  I think generalized existentials will
>>>>>>>>> be nice to have, but I think most people will want them to do something
>>>>>>>>> they can't possibly do.
>>>>>>>> Exactly.  What I meant is that people think they want that expression
>>>>>>>> to compile because they don’t understand that the only thing it can do
>>>>>>>> is trap.  I said “hypothetical” because producing a compile time error
>>>>>>>> rather than a runtime trap is the only sane thing to do.  Your comment
>>>>>>>> surprised me because I can’t imagine we would move forward in Swift
>>>>>>>> with the approach of trapping.
>>>>>>> I would very much like to be able to create instances of “Collection
>>>>>>> where Element == Int” so we can throw away the wrappers in the stdlib.
>>>>>>> That will require some type mismatches to be caught at runtime via
>>>>>>> trapping.
>>>>>> For invalid index because the existential accepts a type erased index?
>>>>> Exactly.
>>>>>> How do you decide where to draw the line here?  It feels like a very
>>>>>> slippery slope for a language where safety is a stated priority to
>>>>>> start adopting a strategy of runtime trapping for something as
>>>>>> fundamental as how you expose members on an existential.
>>>>> If you don't do this, the alternative is that “Collection where Element
>>>>> == Int” does not conform to Collection.  
>>>> This isn’t directly related to having self or associated type
>>>> requirements.  It is true of all existentials.  
>>> That is just an implementation limitation today, IIUC.  What I'm talking
>>> about here would make it impossible for some to do that.
>> If it is just an implementation limitation I am happy to hear that.
>>>> If that changes for simple existentials and generalized existentials
>>>> expose all members (as in the latest draft of the proposal) maybe it
>>>> will be possible for all existentials to conform to their protocol.
>>> Not without introducing runtime traps.  See my “subscript function”
>>> example.
>>>>> That's weird and not very
>>>>> useful.  You could expose all the methods that were on protocol
>>>>> extensions of Collection on this existential, unless they used
>>>>> associated types other than the element type.  But you couldn't pass the
>>>>> existential to a generic function like
>>>>> func scrambled<C: Collection>(_ c: C) -> [C.Element]
>>>>>> IMO you should *have* to introduce unsafe behavior like that manually.
>>>>> Collection where Element == Int & Index == *
>>>>> ?
>>>> I didn’t mean directly through the type of the existential.
>>> My question is, why not?  That is still explicit.
>> It’s not explicit in the sense that nobody wrote `fatalError` or
>> similar in their code.  It’s too easy to write something like that
>> without realizing that it introduces the possibility of a crash.  If
>> we adopt syntax like that to introduce an existential that introduces
>> traps we should at least require members that can be trap to be
>> invoked using a `!` suffix or something like that to make it clear to
>> users that a trap will happen if they are not extremely careful when
>> using that member.
>> More generally though, I don’t want the rules of the language to be
>> written in a way that causes the compiler to synthesize traps in such
>> a general way.
>> The existential should not introduce a precondition that isn’t already
>> present in the semantics of the protocol itself.  If the semantics of
>> the protocol do not place preconditions on arguments beyond their type
>> (such as “must be a valid index into this specific instance”) the
>> compiler should not allow the existential to conform if a trap is
>> required in some circumstances.  That is a new precondition and
>> therefore the existential does not actually fulfill the requirements
>> of the protocol.
>> I could *maybe* live with a solution where protocol requirements are
>> marked as trapping, etc depending on the specific argument received at
>> runtime.  This is a total straw man syntax, but maybe `IndexableBase`
>> would declare the subscript `@trapping` (probably something different
>> but I hope this communicates the idea).  This alerts users to the fact
>> that they need to be extra careful - not any value of `Self.Index` is
>> valid and you can get a crash if you’re not careful.
>> Having this semantic explicit in the definition of the protocol opens
>> the door to maybe considering an existential synthesized by the
>> compiler that traps because it doesn’t introduce a new precondition
>> that wasn’t already present in the protocol.
>> I would want to give consideration to specific details of a proposal
>> along these lines before deciding how I feel about it, but I have a
>> more open mind to this approach than introducing traps not present in
>> the preconditions of the protocol.
>> /// You can subscript a collection with any valid index other than the
>> /// collection's end index. The end index refers to the position one past
>> /// the last element of a collection, so it doesn't correspond with an
>> /// element.
>> ///
>> /// - Parameter position: The position of the element to access. `position`
>> ///   must be a valid index of the collection that is not equal to the
>> ///   `endIndex` property.
>> @trapping public subscript(position: Self.Index) -> Self._Element { get }
>>>> One obvious mechanism for introducing unsafe behavior is to write
>>>> manual type erasure wrappers like we do today.
>>>> Another possibility would be to allow extending the existential type
>>>> (not the protocol).  This would allow you to write overloads on the
>>>> Collection existential that takes some kind of type erased index if
>>>> that is what you want and either trap if you receive an invalid index
>>>> or better (IMO) return an `Element?`.  I’m not sure how extensions on
>>>> existentials might be implemented, but this is an example of the kind
>>>> of operation you might want available on it that you wouldn’t want
>>>> available on all Collection types.
>>>>>> Collection indices are already something that isn’t fully statically
>>>>>> safe so I understand why you might want to allow this.  
>>>>> By the same measure, so are Ints :-)
>>>>> The fact that a type's methods have preconditions does *not* make it
>>>>> “statically unsafe.”
>>>> That depends on what you mean by safe.  Sure, those methods aren’t
>>>> going corrupt memory, but they *are* going to explicitly and
>>>> intentionally crash for some inputs.  That doesn’t qualify as “fully
>>>> safe” IMO.
>>> Please pick a term other than “unsafe” here; it's not unsafe in the
>>> sense we mean the word in Swift.  It's safe in exactly the same way that
>>> array indexes and integers are.  When you violate a precondition, it
>>> traps.
>> I am happy to use any word you like here.
>> Can you clarify what you mean by the word safe in Swift?  It doesn’t
>> appear to be limited to memory safety in the public about page
>> https://swift.org/about/ <https://swift.org/about/> <https://swift.org/about/ <https://swift.org/about/>>:
> I mean memory- and type-safe.
>> Safe. The most obvious way to write code should also behave in a safe
>> manner. Undefined behavior is the enemy of safety, and developer
>> mistakes should be caught before software is in production. Opting for
>> safety sometimes means Swift will feel strict, but we believe that
>> clarity saves time in the long run.
>> Safety
>> Swift was designed from the outset to be safer than C-based languages,
>> and eliminates entire classes of unsafe code. Variables are always
>> initialized before use, arrays and integers are checked for overflow,
>> and memory is managed automatically. Syntax is tuned to make it easy
>> to define your intent — for example, simple three-character keywords
>> define a variable (var) or constant (let).
>> Another safety feature is that by default Swift objects can never be
>> nil, and trying to make or use a nil object will results in a
>> compile-time error. This makes writing code much cleaner and safer,
>> and prevents a common cause of runtime crashes. However, there are
>> cases where nil is appropriate, and for these situations Swift has an
>> innovative feature known as optionals. An optional may contain nil,
>> but Swift syntax forces you to safely deal with it using ? to indicate
>> to the compiler you understand the behavior and will handle it safely.
>> This positioning statement makes it appear as if preventing common
>> causes of crashes falls within the meaning of safe that Swift is
>> using.  Having existentials introduce new preconditions and traps when
>> they are not met does not seem aligned with that goal IMO.
> Static typing “increases safety,” in the casual sense.  That doesn't
> mean that an operation that traps on a failed precondition check is
> “unsafe.”
>>> The user doesn't do anything “manual” to introduce that trapping
>>> behavior for integers.  Preconditions are a natural part of most types.
>> The user doesn’t, but isn’t the overflow trap implemented in the
>> standard library?  
> Whether it is or is not is an implementation detail.
>> Regardless, this is a specific case that has been given explicit
>> design attention by humans.  The precondition is designed, not
>> introduced by compiler rules that haven’t considered the specific case
>> in question.
>>>>>> But I don’t think having the language's existentials do this
>>>>>> automatically is the right approach.  Maybe there is another
>>>>>> approach that could be used in targeted use cases where the less
>>>>>> safe behavior makes sense and is carefully designed.
>>>>> Whether it makes sense or not really depends on the use-cases.  There's
>>>>> little point in generalizing existentials if the result isn't very useful.
>>>> Usefulness depends on your perspective.  
>>> Of course.  As I've said, let's look at the use cases.
>> Agree.  We can consider those in depth when the time comes to ramp up
>> discussion of Austin’s proposal.
>>>> I have run into several scenarios where they would be very useful
>>>> without needing to be prone to crashes when used incorrectly.  One
>>>> obvious basic use case is storing things in a heterogenous collection
>>>> where you bind .
>>> bind what?
>> Sorry, I must have gotten distracted and not finished that paragraph.
>> I meant to say bind the associated types that are necessary for your
>> use case.  Sometimes you bind *all* of the associated types to
>> concrete types and the protocol has no `Self` requirements.  In that
>> case there is no trouble at all in conforming the type-erased
>> “existential" to the protocol itself.  Austin’s proposal would
>> eliminate the need to manually write these “existentials” manually.
>>>>> The way to find out is to take a look at the examples we currently have
>>>>> of protocols with associated types or Self requirements and consider
>>>>> what you'd be able to do with their existentials if type relationships
>>>>> couldn't be erased.  
>>>>> We have known use-cases, currently emulated in the standard library, for
>>>>> existentials with erased type relationships.  *If* these represent the
>>>>> predominant use cases for something like generalized existentials, it
>>>>> seems to me that the language feature should support that.  Note: I have
>>>>> not seen anyone build an emulation of the other kind of generalized
>>>>> existential.  My theory: there's a good reason for that :-).
>>>> AFAIK (and I could be wrong) the only rules in the language that
>>>> require the compiler to synthesize a trap except using a nil IUO, `!`
>>>> on a nil Optional, and an invalid `as` cast .  These are all
>>>> syntactically explicit unsafe / dangerous operations.  All other traps
>>>> are in the standard library (array index, overflow, etc).  Most
>>>> important about all of these cases is that they have received direct
>>>> human consideration.
>>> There is no distinction in the user model between what might be
>>> synthesized by the language and what appears on standard library types.
>> Maybe I shouldn’t have made that distinction.  
>> The point I am trying to emphasize is that each of these are special
>> cases that have received direct human consideration.  The potential
>> for a trap is not introduced by language rules that apply to
>> user-defined constructs in without consideration of the specific
>> details of that construct.
>>>> Introducing a language (not library) mechanism that exposes members on
>>>> generalized existentials in a way that relies on runtime traps for
>>>> type safety feels to me like a pretty dramatic turn agains the stated
>>>> priority of safety.  It will mean you must understand exactly what is
>>>> going on and be extremely careful to use generalized existentials
>>>> without causing crashes.  This will either make Swift code much more
>>>> crashy or will scare people away from using generalized existentials
>>>> (and maybe both).  
>>> I don't accept either of those statements without seeing some analysis
>>> of the use-cases.  For example, I don't believe that AnyCollection et al
>>> are particularly crash-prone.  The likelihood that you'll use the wrong
>>> index type with a collection is very, very low.  I'm less certain of
>>> what happens with Self requirements in real cases.
>> But again, I believe this is an exceptional case as the precondition
>> is explicitly stated in the semantics of the protocol.
> IIUC, it has been cited by Doug as the exemplar of the
> predominantly-requested case by a 10:1 ratio!

In terms of forming the existential, storing it in variables, accepting arguments of that type, etc yes.  I don’t know how many of those requests expect it to conform to the protocol and expect to be able to use it in generic code constrained to the protocol.

>> IMO the burden of proof should be on the side that proposes a
>> mechanism to introduce traps, not the side that proposes avoiding
>> them.
> If you really want to make this about sides and burdens, the burden of
> proof always rests with the side proposing to extend the language.  We
> shouldn't be making changes without understanding how they will play out
> in real use-cases.

I agree with this.  But if we are discussing two different options for extending the language I think the option that doesn’t introduce crashes should be preferred without pretty compelling reasons to choose the option that can introduce crashes.

>>>> Neither of those outcomes is good.
>>>> Collection indices are a somewhat special case as there is already a
>>>> strong precondition that people are familiar with because it would be
>>>> too costly to performance and arguably too annoying to deal with an
>>>> Optional result in every array lookup.  IMO that is why the library is
>>>> able to get away with it in the current type erased AnyCollection.
>>>> But this is not a good model for exposing any members on an
>>>> existential that do not already have a strong precondition that causes
>>>> a trap when violated.
>>>> I think a big reason why you maybe haven’t seen a lot of examples of
>>>> people writing type erased “existentials" is because it is a huge pain
>>>> in the neck to write this stuff manually today.  People may be
>>>> designing around the need for them.  I haven’t seen a huge sampling of
>>>> type erased “existentials" other people are writing but I haven’t
>>>> written any that introduce a trap like this.  The only traps are in
>>>> the “abstract" base class whose methods will never be called (and
>>>> wouldn’t even be implemented if they could be marked abstract).
>>>> What specific things do you think we need to be able to do that rely
>>>> on the compiler synthesizing a trap in the way it exposes the members
>>>> of the existential?
>>> I don't know.  I'm saying, I don't think we understand the use-cases
>>> well enough to make a determination.
>> That’s fair.  I agree that use cases should be carefully considered.  
>>>> Here are a few examples from Austin’s proposal that safely use
>>>> existential collections.  I don’t understand why you think this
>>>> approach is insufficient.  Maybe you could supply a concrete example
>>>> of a use case that can’t be written with the mechanism in Austin’s
>>>> proposal.
>>>> https://github.com/austinzheng/swift-evolution/blob/az-existentials/proposals/XXXX-enhanced-existentials.md#associated-types-and-member-exposure <https://github.com/austinzheng/swift-evolution/blob/az-existentials/proposals/XXXX-enhanced-existentials.md#associated-types-and-member-exposure> <https://github.com/austinzheng/swift-evolution/blob/az-existentials/proposals/XXXX-enhanced-existentials.md#associated-types-and-member-exposure <https://github.com/austinzheng/swift-evolution/blob/az-existentials/proposals/XXXX-enhanced-existentials.md#associated-types-and-member-exposure>> <https://github.com/austinzheng/swift-evolution/blob/az-existentials/proposals/XXXX-enhanced-existentials.md#associated-types-and-member-exposure <https://github.com/austinzheng/swift-evolution/blob/az-existentials/proposals/XXXX-enhanced-existentials.md#associated-types-and-member-exposure><https://github.com/austinzheng/swift-evolution/blob/az-existentials/proposals/XXXX-enhanced-existentials.md#associated-types-and-member-exposure <https://github.com/austinzheng/swift-evolution/blob/az-existentials/proposals/XXXX-enhanced-existentials.md#associated-types-and-member-exposure>>>
>>>> let a : Any<Collection>
>>>> // A variable whose type is the Index associated type of the underlying
>>>> // concrete type of 'a'.
>>>> let theIndex : a.Index = ...
>>>> // A variable whose type is the Element associated type of the underlying
>>>> // concrete type of 'a'.
>>>> let theElement : a.Element = ...
>>>> // Given a mutable collection, swap its first and last items.
>>>> // Not a generic function. 
>>>> func swapFirstAndLast(inout collection: Any<BidirectionalMutableCollection>) {
>>>>   // firstIndex and lastIndex both have type "collection.Index"
>>>>   guard let firstIndex = collection.startIndex,
>>>>       lastIndex = collection.endIndex?.predecessor(collection) where lastIndex != firstIndex else {
>>>>           print("Nothing to do")
>>>>           return
>>>>   }
>>>>   // oldFirstItem has type "collection.Element"
>>>>   let oldFirstItem = collection[firstIndex]
>>>>   collection[firstIndex] = collection[lastIndex]
>>>>   collection[lastIndex] = oldFirstItem
>>>> }
>>>> var a : Any<BidirectionalMutableCollection where .Element == String> = ...
>>>> let input = "West Meoley"
>>>> // Not actually necessary, since the compiler knows "a.Element" is String.
>>>> // A fully constrained anonymous associated type is synonymous with the concrete
>>>> // type it's forced to take on, and the two are interchangeable.
>>>> // However, 'as' casting is still available if desired.
>>>> let anonymousInput = input as a.Element
>>>> a[a.startIndex] = anonymousInput
>>>> // as mentioned, this also works:
>>>> a[a.startIndex] = input
>>>> // If the collection allows it, set the first element in the collection to a given string.
>>>> func setFirstElementIn(inout collection: Any<Collection> toString string: String) {
>>>>   if let element = string as? collection.Element {
>>>>       // At this point, 'element' is of type "collection.Element"
>>>>       collection[collection.startIndex] = element
>>>>   }
>>>> }
>>> Neither of these look like they actually make *use* of the fact that
>>> there's type erasure involved (and therefore should probably be written
>>> as generics?).  The interesting cases with Any<Collection...>, for the
>>> purposes of this discussion, arise when you have multiple instances of
>>> the same existential type that wrap different concrete types.
>> One use case I have found is to work around the lack of higher-kinder
>> types.  
> Really, now: a use-case for feature A that is a workaround for the lack
> of feature B hardly justifies adding feature A!  We do want to add
> higher-kinded types eventually.
Good to know.  I thought higher-kinder types were on the “maybe if someone shows a compelling enough use case” list.  AFAIK this is the first time a member of the core team has stated the intent to add them.  If that is the case I agree that this use case isn’t relevant.  The workaround isn’t great because it loses type information that is critical to the optimizer (but it’s all we have available today).

>> If you have a protocol where specific implementations will return
>> different types, but all conform to a second protocol you can define
>> the protocol in terms of a generic type-erased wrapper which conforms
>> to the second protocol and accepts type arguments that match the
>> associated types (thus binding the associated types to concrete
>> types).  I have found this to be a useful technique (granted it is a
>> workaround and I’m not sure how useful it would continue to be if
>> Swift eventually gets higher-kinder types).
>>> Another problem I see: in this new world, what is the model for choosing
>>> whether to write a function as a protocol extension/generic, or as a
>>> regular function taking existential parameters?  Given that either of
>>> the above could have been written either way, we need to be able to
>>> answer that question.  When existentials don't conform to their
>>> protocols, it seems to me that the most general thing to do is use
>>> existentials whenever you can, and only resort to using generics when
>>> forced by the type system.  This does not seem like a particularly good
>>> programming model to me, but I might be convinced otherwise.
>> That doesn’t seem like a particularly good programming model to me either. 
>> The rule of thumb I am operating with for protocols with Self or
>> associated type requirements is to prefer generics and use type
>> erasure / existentials when that isn’t possible.  For example, when
>> heterogeneity is required or when you can’t form the necessary type in
>> a protocol requirement (as in the preceding example).
>> This heuristic has been working out pretty well for me thus far.  
> I do worry a bit that people will choose the opposite heuristic.
> It would be somewhat reassuring to me if we could prove to ourselves
> that, using your heuristic, one is never forced to copy/paste a generic
> function implementation into a corresponding function that uses
> existentials.
>> The primary impact of introducing a language mechanism for generalized
>> existentials in my code would be to eliminate a lot of manual type
>> erasing boilerplate.
> If your code has many manual type erasing wrappers corresponding to
> protocols with associated types and/or Self requirements that also never
> have to trap type mismatches, that would certainly be instructive
> empirical data.  Would you care to share the protocols and wrappers you
> are talking about?

I put together a sample implementation of a Cocoa-like responder chain in Swift a while ago when the “Swift dynamism” debate was raging.  

It isn't intended to be a Swifty design.  It is intended to be similar to Cocoa and show techniques that can be used to do things similar to Cocoa’s responder chain and targer-action in Swift.  It uses a type erased wrapper for actions that binds `Sender` while hiding the concrete `Action` type and also the `Handler` associated type.  It cannot and should not conform to the protocol it is derived from and could be replaced with the generalized existentials in Austin’s proposal.

https://gist.github.com/anandabits/ec26f67f682093cf18b170c21bcf433e <https://gist.github.com/anandabits/ec26f67f682093cf18b170c21bcf433e>

This is a good example to start with because it is related to a topic that has been hotly debated and is clearly something a lot of people want to be able to do.

>>> Anyway, my overall point is that this all seems like something we *can*
>>> do and that nicely fills gaps in the type system, but not necessarily
>>> something we *should* do until we better understand what it's actually
>>> *for* and how it affects the programming model.
>> That’s a very fair position to take.  :)
>>> -- 
>>> Dave
> -- 
> Dave

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-evolution/attachments/20160609/ea04db9c/attachment-0001.html>

More information about the swift-evolution mailing list