[swift-evolution] Proposal: Introduce User-defined "Dynamic Member Lookup" Types

[Matthew Johnson]

> On Dec 1, 2017, at 12:23 AM, Chris Lattner <clattner at nondot.org> wrote:
> 
>> On Nov 30, 2017, at 6:15 AM, Matthew Johnson <matthew at anandabits.com> wrote:
>>> 
>>> I think better interoperability with Python (and other OO languages in widespread use) is a good goal, and I agree that the implementation of the feature described is straight-forward and not terribly invasive in the compiler.
>>> 
>>> However, I do not think this proposal is going in the right direction for Swift. I have objections on several different grounds.
>> 
>> +1 to everything Doug says here.  He articulates the concerns I tried to voice in an earlier thread more clearly and more thoroughly.  
> 
> Doug’s email was pretty confusing, so I’ll try to clear up some of the misconceptions here.  I also updated the proposal with a new section at the end, please check it out.

Thank you for your latest reply and the latest reply to Doug.  I can only speak for myself, but it helped me to better understand the motivation you have for the proposal.  I really do appreciate your view and the motivation behind it.

> 
>> This design introduces a significant hole in the type system which is contrary to the spirit of Swift.
> 
> There is no “hole” in the type system.  The proposal is fully type safe. 
> 
> Further, the addition is absolutely in the spirit of Swift, because it is directly analogous to an existing feature: AnyObject lookup.  The difference between it and AnyObject lookup is that AnyObject lookup is:
> 
> a) specific to Objective-C interop
> b) type unsafe
> c) massively invasive on the rest of the compiler.
> 
> We’ve made many attempts to narrow the scope of AnyObject lookup, but it is difficult to do so given backwards compatibility constraints and the limited nature of Objective-C metadata.

I may have spoken incorrectly but I do still believe it is contrary to what I (and apparently many others) feel is the spirit of Swift.  

As I understand it, AnyObject lookup was a necessity during the historical development of Swift.  The attempts to narrow its scope have merit regardless of how far they can go.  They indicate that perhaps it is a feature we would remove if that were possible.  I would love to see that happen someday (although I know it is unlikely), or at least see it be made explicit at the call site.  I suspect if this feature did not exist and it was proposed today it would be met with at least as much resistance as your proposals have.

> 
>> It doesn’t just make dynamic language interop easy, it also changes the semantics of any type which conforms to DynamicMemberLookupProtocol.  That is not something that is locally visible when looking at a piece of code.  Worse, it does so in a way that trivial mistakes such as a typo can turn into runtime errors.  Worst of all, as Doug points out it is possible to use retroactive conformance to change the semantics of vast quantities of widely used types in one fell swoop.
> 
> This is a feature, like many others, which can be misused.  This has not been the metric we have used to judge what should go into Swift.  I can elaborate if my response to Doug wasn’t clear.

The primary problem I have is that misuse is too easy and too subtle.  It is at least as easy to misuse as other features in Swift which have been carefully designed to be obvious at usage sites.  

I was happy to see that you are willing to consider Xiaodi’s suggestion to require conformance to be stated as part of the original type declaration.  That would be a significant improvement to the proposal IMO.  I strongly urge you to reconsider the decision of that dynamic members must be made available with no indication at usage sites.  An indication of dynamic lookup at usage sites aligns very well (IMO) with the rest of Swift (AnyObject lookup aside) by calling attention to code that requires extra care to get right.

> 
>> One additional concern that I don’t believe has been mentioned is that of evolution.  When a library is imported from a dynamic language and the library makes breaking changes Swift code written using this feature will break without notice until a runtime error occurs.  Is that acceptable?  That may be what users of dynamic languages are accustomed to but can we do better in Swift?  If we can make the process of upgrading dependencies less a less harrowing experience perhaps that could be a major selling point in moving them to Swift.  But we won’t accomplish that with this design.
> 
> How would you this to work with *any* interoperability approach?  Dynamic languages are in fact dynamic, and need the ability to do dynamic lookups somehow.  Those dynamic lookups will all have the problem you observe.

I’ll mostly let Doug continue the discussion of the approach he is suggesting.  It looks to me like that is an approach that might help in this area.  For example, if a symbol is removed it may be possible to produce a compiler error in at least some cases.  I don’t think Doug’s suggestion precludes full dynamism but would offer a much better experience in at least some cases.  Perhaps it would even be enough to call into question whether full dynamism is really necessary or not (as Doug seems to think).  We won’t know for sure without exploring this further.

I understand that you believe the direction he has advocated is not realistic.  Perhaps, but I do appreciate that the conversation is happening before a final decision is made.  I support further exploration of this at least as long as Doug believes it is a good idea to do so.

> 
>> Interop with dynamic languages is a worthy goal, but we should do it in a way that has no potential impact on code that does not need the feature.  A semantic variant of “don’t pay for what you don’t use” should apply here.  If we add this feature everyone will pay for it every time they read unfamiliar Swift code.  They will have to contend with the potential for conformances to this protocol, even on standard library types.  That feels like a pretty high cost to me, especially for programmers who aren’t using the dynamic language interop features themselves.
> 
> You seem to fear that people will pervasively adopt this and use it for weird things.  While that is possible, that would be simply one way to misuse Swift.  What makes it more likely for someone to do this than to use unsafe features incorrectly, providing an apparently type safe API that actually isn’t?

It’s not the potential that I fear.  As mentioned above, it is that the presence or absence of this potential in a given piece of code is too hard to detect, especially for a reader who is unfamiliar with the code in question.  It is pretty clear that I am not the only Swift developer with significant concerns about this.

> 
>> Further, a consistent theme has been the desire to avoid things like compiler flags that could fragment the community.  I fear this feature has the potential to do that, especially if it is really successful at attracting people from the dynamic language community.  
> 
> I really fail to see how this concern relates here.  This has nothing to do with compiler flags.  This has the potential to expand the community, not fragment it.

One of the stated aims of this proposal is to bring many new programmers into the Swift community.  That is a wonderful goal!  

Of course that may include programmers with a different set of technical values than are currently predominant in the Swift community.  If we make it really easy to use Swift as a highly dynamic language we may soon see a large sub-community form that uses Swift in this way.  Regardless of how you would feel about this I think it’s an important possible outcome to consider.

Your followup to Doug helped to clarify that many programmers in the data science and ML spaces you are hoping to bring into the Swift community are actually frustrated with having to use Python.  That is really interesting and should influence the design.  Perhaps these people would appreciate having dynamic member lookup be apparent at usage sites.  Have you asked them?  

You mentioned that many of them may prefer proper Swift APIs for the libraries they use and are hoping that the issues with dynamism will incentive them to create these in time.  Perhaps requiring some kind of call-site annotation would increase this incentive without being significant enough to disincentive using these libraries in Swift.

Changing the proposal to restrict conformance to the initial type declaration and requiring usage site annotation of some kind for dynamic member lookup would address my most significant concerns about your proposal.  I hope you will reconsider these points of design.  I don’t think we need something heavy-handed, but something similar in weight to optional chaining or force unwrapping.  If you modify this proposal to include such syntax perhaps it would eventually be appropriate to consider requiring the same syntax for AnyObject lookup as well (I would really like that).

In particular, if you continue to insist on standard dot syntax for member lookup I would appreciate hearing more about why that is a sticking point for you, especially in light of your recent comment about people looking to move away from Python.

I have conflicted feelings about this proposal regardless of whether the above mentioned changes are made or not.  The value of interoperability with established libraries in other languages is significant in some domains.  On the other hand, I question whether now is the time to introduce more dynamic features into Swift and whether this is the right first step.  It is a feature that may have a significant impact on how some people choose to use the language.  As others have pointed out, Swift is still lacking features that could solve many of the use cases for which dynamic features like this could be used (or abused).  

Both perspectives have merit and I really don’t know which path would lead to a better outcome.  In any case, I would like to see this proposal strengthened so that should it be introduced the benefits are maximized and the costs are minimized.  With all due respect, I don’t think there has been enough consideration given to the costs that may be incurred should the proposal be accepted as written.

Matthew

> 
> -Chris
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-evolution/attachments/20171201/1db5a2d8/attachment.html>