[swift-evolution] Proposal: Introduce User-defined "Dynamic Member Lookup" Types

Sat Dec 2 14:44:18 CST 2017

On Dec 1, 2017, at 5:50 PM, Matthew Johnson <matthew at anandabits.com> wrote:
>>> This design introduces a significant hole in the type system which is contrary to the spirit of Swift.
>> 
>> There is no “hole” in the type system.  The proposal is fully type safe. 
>> 
>> Further, the addition is absolutely in the spirit of Swift, because it is directly analogous to an existing feature: AnyObject lookup.  The difference between it and AnyObject lookup is that AnyObject lookup is:
>> 
>> a) specific to Objective-C interop
>> b) type unsafe
>> c) massively invasive on the rest of the compiler.
>> 
>> We’ve made many attempts to narrow the scope of AnyObject lookup, but it is difficult to do so given backwards compatibility constraints and the limited nature of Objective-C metadata.
> 
> I may have spoken incorrectly but I do still believe it is contrary to what I (and apparently many others) feel is the spirit of Swift.  
> 
> As I understand it, AnyObject lookup was a necessity during the historical development of Swift.  The attempts to narrow its scope have merit regardless of how far they can go.  They indicate that perhaps it is a feature we would remove if that were possible.  I would love to see that happen someday (although I know it is unlikely), or at least see it be made explicit at the call site.  I suspect if this feature did not exist and it was proposed today it would be met with at least as much resistance as your proposals have.

This isn’t the way I see it.  You’re right that we want to reduce AnyObject dispatch wherever possible: types are good after all.  Swift on Linux doesn’t even have AnyObject dispatch, so you can tell that it is a means to an end, not a feature that we think is great for Swift on its own merits.

What is that end?  It is interoperability with truly dynamic values, which do occur in Objective-C, where pervasive casting would harm code clarity and usability.  The desire to fix AnyObject you are seeing are related to two different issues: 1) The design of AnyObject dispatch itself is problematic in some ways, and 2) many values in ObjC APIs were typed as id simply because there was no way to describe a more specific type given Objective-C’s original type system (which has been improved).  However, given the presence of actually polymorphic values being used by some APIs, I don’t imagine it going away entirely.

In the case of other dynamic languages, the situation is even more severe.  Some dynamic languages have progressive type systems available, but not all do.  Those which do have to deal with the fact that the base languages were not designed for typing systems, which is far more severe a problem than Objective-C’s situation.  

If you haven’t already, I encourage you to read the mypy docs (http://mypy.readthedocs.io/en/stable/index.html <http://mypy.readthedocs.io/en/stable/index.html>) to understand how the type system they’re trying to build on top of Python works.  They have to contend with things like:

- the single array subscript operator taking both scalars and ranges, and returning different type results depending on their input (Python has no overloading)
- pervasive use of duck typing, e.g. int is duck type compatible with float and complex (duck typing is for many non-primitive types as well)
- pervasive use of variance
- APIs creep to “just work” when passed all sorts of weird values, meaning that their API contract is extremely loose.
- Python supports *dynamically computed base classes* and lots of other crazy things.
- and more..

Mypy also only supports a subset of Python - it doesn’t support properties with setters as one example, and its generic and class system is only partially implemented.  The only way to use it in practice is to use their ability to silence warnings, it is not an acceptable basis for a sound type system that we could depend on in Swift.

For all those reasons, we really do need something like AnyObject dispatch if we care about working with dynamically typed languages.  The design I’m suggesting carefully cordons this off into its own struct type, so it doesn’t infect the rest of the type system, and is non-invasive in the compiler.

>>> It doesn’t just make dynamic language interop easy, it also changes the semantics of any type which conforms to DynamicMemberLookupProtocol.  That is not something that is locally visible when looking at a piece of code.  Worse, it does so in a way that trivial mistakes such as a typo can turn into runtime errors.  Worst of all, as Doug points out it is possible to use retroactive conformance to change the semantics of vast quantities of widely used types in one fell swoop.
>> 
>> This is a feature, like many others, which can be misused.  This has not been the metric we have used to judge what should go into Swift.  I can elaborate if my response to Doug wasn’t clear.
> 
> The primary problem I have is that misuse is too easy and too subtle.  It is at least as easy to misuse as other features in Swift which have been carefully designed to be obvious at usage sites.  
> 
> I was happy to see that you are willing to consider Xiaodi’s suggestion to require conformance to be stated as part of the original type declaration.  That would be a significant improvement to the proposal IMO.

I’ve revised the proposal to require this.

> I strongly urge you to reconsider the decision of that dynamic members must be made available with no indication at usage sites.  An indication of dynamic lookup at usage sites aligns very well (IMO) with the rest of Swift (AnyObject lookup aside) by calling attention to code that requires extra care to get right.

I don’t understand this.  The proposal is fully type safe, and this approach is completely precedented by AnyObject.  Swift’s type system supports many ways to express fallibility, and keeping those decisions orthogonal to this proposal is the right thing to do, because it allows the author of the type to decide what model makes sense for them.

In any case, I hear loud and clear that there is concern about possible abuse of this proposal, so I added a section in the alternatives section to discuss further ways to reduce possibility for abuse:
https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438#reducing-potential-abuse <https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438#reducing-potential-abuse>

More suggestions are welcome.

>>> One additional concern that I don’t believe has been mentioned is that of evolution.  When a library is imported from a dynamic language and the library makes breaking changes Swift code written using this feature will break without notice until a runtime error occurs.  Is that acceptable?  That may be what users of dynamic languages are accustomed to but can we do better in Swift?  If we can make the process of upgrading dependencies less a less harrowing experience perhaps that could be a major selling point in moving them to Swift.  But we won’t accomplish that with this design.
>> 
>> How would you this to work with *any* interoperability approach?  Dynamic languages are in fact dynamic, and need the ability to do dynamic lookups somehow.  Those dynamic lookups will all have the problem you observe.
> 
> I’ll mostly let Doug continue the discussion of the approach he is suggesting.  It looks to me like that is an approach that might help in this area.  For example, if a symbol is removed it may be possible to produce a compiler error in at least some cases.  I don’t think Doug’s suggestion precludes full dynamism but would offer a much better experience in at least some cases.  Perhaps it would even be enough to call into question whether full dynamism is really necessary or not (as Doug seems to think).  We won’t know for sure without exploring this further.
> 
> I understand that you believe the direction he has advocated is not realistic.  Perhaps, but I do appreciate that the conversation is happening before a final decision is made.  I support further exploration of this at least as long as Doug believes it is a good idea to do so.

Yes, I agree, it is important to get aligned on this.

>>> Further, a consistent theme has been the desire to avoid things like compiler flags that could fragment the community.  I fear this feature has the potential to do that, especially if it is really successful at attracting people from the dynamic language community.  
>> 
>> I really fail to see how this concern relates here.  This has nothing to do with compiler flags.  This has the potential to expand the community, not fragment it.
> 
> One of the stated aims of this proposal is to bring many new programmers into the Swift community.  That is a wonderful goal!  
> 
> Of course that may include programmers with a different set of technical values than are currently predominant in the Swift community.  If we make it really easy to use Swift as a highly dynamic language we may soon see a large sub-community form that uses Swift in this way.  Regardless of how you would feel about this I think it’s an important possible outcome to consider.
…
> You mentioned that many of them may prefer proper Swift APIs for the libraries they use and are hoping that the issues with dynamism will incentive them to create these in time.  Perhaps requiring some kind of call-site annotation would increase this incentive without being significant enough to disincentive using these libraries in Swift.

I think you over-estimate how awesome Python (and other language APIs) are going to be, even with these two proposals.  Keep in mind that Python has the GIL for example, the naming conventions and design patterns are completely different, etc.  Even with both of these proposals, there will be high incentive for building pure Swift APIs to replace things over time.  The situation is similar to C interop with Swift: it is incredibly important that it exist, but people aren’t exactly aiming to use UnsafePointer for everything :-).  It is possible someone would do so, but people do all sorts of bad things locally in their codebase.  We cannot prevent that.

In fact, I would argue that making Python interoperability *too good* is the bigger risk we have.  If there is no reason for people to move off of using those APIs, they will continue to do so forever.  That isn’t a transition path, that’s a merger of communities.

> In particular, if you continue to insist on standard dot syntax for member lookup I would appreciate hearing more about why that is a sticking point for you, especially in light of your recent comment about people looking to move away from Python.

Thanks, I addressed this in the alternative section.

-Chris

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-evolution/attachments/20171202/8e7084f2/attachment.html>