[swift-evolution] Proposal: Introduce User-defined "Dynamic Member Lookup" Types

[Matthew Johnson]

> On Dec 2, 2017, at 2:44 PM, Chris Lattner <clattner at nondot.org> wrote:
> 
> On Dec 1, 2017, at 5:50 PM, Matthew Johnson <matthew at anandabits.com <mailto:matthew at anandabits.com>> wrote:
>>>> This design introduces a significant hole in the type system which is contrary to the spirit of Swift.
>>> 
>>> There is no “hole” in the type system.  The proposal is fully type safe. 
>>> 
>>> Further, the addition is absolutely in the spirit of Swift, because it is directly analogous to an existing feature: AnyObject lookup.  The difference between it and AnyObject lookup is that AnyObject lookup is:
>>> 
>>> a) specific to Objective-C interop
>>> b) type unsafe
>>> c) massively invasive on the rest of the compiler.
>>> 
>>> We’ve made many attempts to narrow the scope of AnyObject lookup, but it is difficult to do so given backwards compatibility constraints and the limited nature of Objective-C metadata.
>> 
>> I may have spoken incorrectly but I do still believe it is contrary to what I (and apparently many others) feel is the spirit of Swift.  
>> 
>> As I understand it, AnyObject lookup was a necessity during the historical development of Swift.  The attempts to narrow its scope have merit regardless of how far they can go.  They indicate that perhaps it is a feature we would remove if that were possible.  I would love to see that happen someday (although I know it is unlikely), or at least see it be made explicit at the call site.  I suspect if this feature did not exist and it was proposed today it would be met with at least as much resistance as your proposals have.
> 
> This isn’t the way I see it.  You’re right that we want to reduce AnyObject dispatch wherever possible: types are good after all.  Swift on Linux doesn’t even have AnyObject dispatch, so you can tell that it is a means to an end, not a feature that we think is great for Swift on its own merits.

It’s a means to an end and the need for that end is growing smaller over time.

> 
> What is that end?  It is interoperability with truly dynamic values, which do occur in Objective-C, where pervasive casting would harm code clarity and usability.  The desire to fix AnyObject you are seeing are related to two different issues: 1) The design of AnyObject dispatch itself is problematic in some ways, and 2) many values in ObjC APIs were typed as id simply because there was no way to describe a more specific type given Objective-C’s original type system (which has been improved).  However, given the presence of actually polymorphic values being used by some APIs, I don’t imagine it going away entirely.

Which is a reason to consider making this mechanism more visible in the syntax.

> 
> 
> In the case of other dynamic languages, the situation is even more severe.  Some dynamic languages have progressive type systems available, but not all do.  Those which do have to deal with the fact that the base languages were not designed for typing systems, which is far more severe a problem than Objective-C’s situation.  
> 
> If you haven’t already, I encourage you to read the mypy docs (http://mypy.readthedocs.io/en/stable/index.html <http://mypy.readthedocs.io/en/stable/index.html>) to understand how the type system they’re trying to build on top of Python works.  They have to contend with things like:
> 
> - the single array subscript operator taking both scalars and ranges, and returning different type results depending on their input (Python has no overloading)
> - pervasive use of duck typing, e.g. int is duck type compatible with float and complex (duck typing is for many non-primitive types as well)
> - pervasive use of variance
> - APIs creep to “just work” when passed all sorts of weird values, meaning that their API contract is extremely loose.
> - Python supports *dynamically computed base classes* and lots of other crazy things.
> - and more..
> 
> Mypy also only supports a subset of Python - it doesn’t support properties with setters as one example, and its generic and class system is only partially implemented.  The only way to use it in practice is to use their ability to silence warnings, it is not an acceptable basis for a sound type system that we could depend on in Swift.
> 
> 
> For all those reasons, we really do need something like AnyObject dispatch if we care about working with dynamically typed languages.  The design I’m suggesting carefully cordons this off into its own struct type, so it doesn’t infect the rest of the type system, and is non-invasive in the compiler.

I am quite familiar with dynamic languages and agree that this is necessary if we are going to fully open up access to these languages from Swift.

> 
> 
>>>> It doesn’t just make dynamic language interop easy, it also changes the semantics of any type which conforms to DynamicMemberLookupProtocol.  That is not something that is locally visible when looking at a piece of code.  Worse, it does so in a way that trivial mistakes such as a typo can turn into runtime errors.  Worst of all, as Doug points out it is possible to use retroactive conformance to change the semantics of vast quantities of widely used types in one fell swoop.
>>> 
>>> This is a feature, like many others, which can be misused.  This has not been the metric we have used to judge what should go into Swift.  I can elaborate if my response to Doug wasn’t clear.
>> 
>> The primary problem I have is that misuse is too easy and too subtle.  It is at least as easy to misuse as other features in Swift which have been carefully designed to be obvious at usage sites.  
>> 
>> I was happy to see that you are willing to consider Xiaodi’s suggestion to require conformance to be stated as part of the original type declaration.  That would be a significant improvement to the proposal IMO.
> 
> I’ve revised the proposal to require this.

Thank you!

> 
>> I strongly urge you to reconsider the decision of that dynamic members must be made available with no indication at usage sites.  An indication of dynamic lookup at usage sites aligns very well (IMO) with the rest of Swift (AnyObject lookup aside) by calling attention to code that requires extra care to get right.
> 
> I don’t understand this.  The proposal is fully type safe, and this approach is completely precedented by AnyObject.  Swift’s type system supports many ways to express fallibility, and keeping those decisions orthogonal to this proposal is the right thing to do, because it allows the author of the type to decide what model makes sense for them.

Allowing the author of the type to choose whether the mechanism is hidden or visible is exactly what I don’t want to allow.  I think you have the right design regarding types and semantics - the author chooses.  But I don’t want these calls to look like ordinary member lookup when I’m reading code.  

They inherently have a much greater chance of failure than ordinary member lookup.  Further, authors are likely to choose immediate traps or nil IUO as failure modes as forcing users to deal with Optional on every call is likely to be untenable.  I believe this behavior should be represented by some kind of syntax at the usage site.  I don’t believe it is an undue burden.  It would make the dynamic lookup semantic clear to all readers and would help to discourage abuse.

> 
> In any case, I hear loud and clear that there is concern about possible abuse of this proposal, so I added a section in the alternatives section to discuss further ways to reduce possibility for abuse:
> https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438#reducing-potential-abuse <https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438#reducing-potential-abuse>
> 
> More suggestions are welcome.

Thank you for adding this section.  It strengthens the proposal to have this perspective represented in some fashion.

> 
> 
>>>> One additional concern that I don’t believe has been mentioned is that of evolution.  When a library is imported from a dynamic language and the library makes breaking changes Swift code written using this feature will break without notice until a runtime error occurs.  Is that acceptable?  That may be what users of dynamic languages are accustomed to but can we do better in Swift?  If we can make the process of upgrading dependencies less a less harrowing experience perhaps that could be a major selling point in moving them to Swift.  But we won’t accomplish that with this design.
>>> 
>>> How would you this to work with *any* interoperability approach?  Dynamic languages are in fact dynamic, and need the ability to do dynamic lookups somehow.  Those dynamic lookups will all have the problem you observe.
>> 
>> I’ll mostly let Doug continue the discussion of the approach he is suggesting.  It looks to me like that is an approach that might help in this area.  For example, if a symbol is removed it may be possible to produce a compiler error in at least some cases.  I don’t think Doug’s suggestion precludes full dynamism but would offer a much better experience in at least some cases.  Perhaps it would even be enough to call into question whether full dynamism is really necessary or not (as Doug seems to think).  We won’t know for sure without exploring this further.
>> 
>> I understand that you believe the direction he has advocated is not realistic.  Perhaps, but I do appreciate that the conversation is happening before a final decision is made.  I support further exploration of this at least as long as Doug believes it is a good idea to do so.
> 
> Yes, I agree, it is important to get aligned on this.
> 
>>>> Further, a consistent theme has been the desire to avoid things like compiler flags that could fragment the community.  I fear this feature has the potential to do that, especially if it is really successful at attracting people from the dynamic language community.  
>>> 
>>> I really fail to see how this concern relates here.  This has nothing to do with compiler flags.  This has the potential to expand the community, not fragment it.
>> 
>> One of the stated aims of this proposal is to bring many new programmers into the Swift community.  That is a wonderful goal!  
>> 
>> Of course that may include programmers with a different set of technical values than are currently predominant in the Swift community.  If we make it really easy to use Swift as a highly dynamic language we may soon see a large sub-community form that uses Swift in this way.  Regardless of how you would feel about this I think it’s an important possible outcome to consider.
> …
>> You mentioned that many of them may prefer proper Swift APIs for the libraries they use and are hoping that the issues with dynamism will incentive them to create these in time.  Perhaps requiring some kind of call-site annotation would increase this incentive without being significant enough to disincentive using these libraries in Swift.
> 
> I think you over-estimate how awesome Python (and other language APIs) are going to be, even with these two proposals.  Keep in mind that Python has the GIL for example, the naming conventions and design patterns are completely different, etc.  Even with both of these proposals, there will be high incentive for building pure Swift APIs to replace things over time.  The situation is similar to C interop with Swift: it is incredibly important that it exist, but people aren’t exactly aiming to use UnsafePointer for everything :-).  It is possible someone would do so, but people do all sorts of bad things locally in their codebase.  We cannot prevent that.
> 
> In fact, I would argue that making Python interoperability *too good* is the bigger risk we have.  If there is no reason for people to move off of using those APIs, they will continue to do so forever.  That isn’t a transition path, that’s a merger of communities.

I’m not sure why you think I’m over-estimating this.  The suggestion above is in fact partly that requiring usage site annotation would help ensure that Python interoperability is too good.  This will be a subtle but regular reminder that it might be good to at least wrap up usage in an overlay to isolate and reduce the prevalence of dynamic lookup.  

I’m also suggesting that it’s possible that some of the programmers interested in using Swift in this way would also appreciate making this clear at the usage site.  I think it’s at least worth finding out.

> 
>> In particular, if you continue to insist on standard dot syntax for member lookup I would appreciate hearing more about why that is a sticking point for you, especially in light of your recent comment about people looking to move away from Python.
> 
> Thanks, I addressed this in the alternative section.

I appreciate this, thank you.

> Swift's type system already includes features (optionals, IUOs, runtime failure) for handling failability. Keeping that orthogonal to this proposal is good because that allows API authors to make the right decision for expressing the needs of their use-case.
As noted above, I think the semantics of the lookup and result of invoking the member itself are orthogonal to the visibility of the dynamic lookup mechanism at the usage site.  I’m not asking you to impose any semantics on library authors, only to make the mechanism of lookup visible.

> Swift already has a dynamic member lookup feature, "AnyObject dispatch" which does not use additional punctuation, so this would break precedent.
I would prefer if dynamic lookup were visible with AnyObject as well.  For that reason I don’t believe it makes a good precedent to follow.  In fact, I would prefer to see us go the other direction and perhaps even consider revising dynamic lookup syntax for AnyObject in the future.

> The point of this proposal is to make use of dynamic language APIs more elegant than what is already possible: making use of them ugly (this punctuation character would be pervasive through use of the APIs and just add visual noise, not clarity) undermines the entire purpose of this proposal.
It will make them significantly more elegant than what is already possible with or without one additional character.  I don’t think it undermines the proposal at all.  I think it strikes the best balance between dynamic lookup semantics and clarity at the usage site.  This is a semantic that I as a reader really want to see clearly when it is used.  Aesthetics are important until they interfere with clarity.  

This can be a difficult balance to strike and I am arguing for a different balance than you seem to prefer.  If this proposal is to be accepted the community and the core team will have to decide what balance to strike.  I think it’s a good idea to consider it carefully.

> 
> -Chris
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-evolution/attachments/20171202/3fb945b9/attachment.html>