[swift-evolution] Proposal: Introduce User-defined "Dynamic Member Lookup" Types

Joe DeCapo snoogansbc at gmail.com
Sun Dec 3 10:52:29 CST 2017


One common criticism people keep bringing up is that people will misuse this and end up exposing dynamic api's into general use (and without notation at the call site that draws attention to this, the user may accidentally invoke something dynamic they didn't intend to). I feel like there is enough prior art in current Swift libraries to disabuse us of this notion. Basically every library I've seen goes out of its way to provide a safe api and leverage Swift's strong type system in their public api's. We don't see libraries using force unwraps willy nilly to hide the fact that a non-optional return type is actually coming from an optional value, but this is something that has been possible since day one and is trivially easy to do.

There are essentially 2 "bad" use cases for consumers of these dynamic features: 1) as the consumer of a library that exposes these dynamic features as public api; and 2) as the creator of a library/executable that uses them irresponsibly internally. For (1) it's a very easy decision to simply not use a Swift library that exposes dynamic features like this externally if you object to it, and instead use an alternative. I highly doubt this would become idiomatic in the Swift community, because the enthusiasm for Swift is mostly based on the strong typing it provides. For (2) it is within the creator's complete control to refactor the code to work the preferred way. The moment there's an unintended invocation of a dynamic call, it can simply be wrapped in a more strongly typed Swift wrapper.

As for the autocompletion and jump-to-definition capabilities of the state of the art in Python, I've been working on a Python project for a few years and until recently only used Atom, which provided code completion with a plug-in, but I could have gotten by without it. I recently started using PyCharm, and although it has jump-to-definition capabilities, the dynamic nature of the language means that there's almost always multiple different possible matches, and most often the top choices are the wrong ones. I don't see how Swift could improve on this situation, and for the case of providing quick help, it may actively provide the wrong information. This is not to say that autocompletion and the like aren't worthwhile goals, but I don't think they're really an important barrier to initial adoption of a way to call into dynamic languages.

I get the sense that this is a minor syntactic sugar feature that would probably mostly be used by implementers of library wrappers to provide Swifty api's that call into dynamic language api's. So the concerns about autocompletion and pervasive exposure of these dynamic aspects more generally really feel misplaced. When writing a wrapper around a REST api, I get zero autocompletion when specifying the JSON models, but that in no way hampers my ability to wrap these api's. Sure, there may be more runtime bugs that I have to sort out when initially implementing things, but that's inherent in the nature of dealing with a dynamic programming model. I imagine it will be the same for Swift libraries wrapping dynamic language libraries.


> On Dec 3, 2017, at 10:41 AM, David Hart via swift-evolution <swift-evolution at swift.org> wrote:
> 
> 
> 
>> On 3 Dec 2017, at 04:11, Matthew Johnson via swift-evolution <swift-evolution at swift.org <mailto:swift-evolution at swift.org>> wrote:
>> 
>> 
>> 
>> Sent from my iPad
>> 
>> On Dec 2, 2017, at 7:40 PM, Chris Lattner <clattner at nondot.org <mailto:clattner at nondot.org>> wrote:
>> 
>>> On Dec 2, 2017, at 2:13 PM, Matthew Johnson <matthew at anandabits.com <mailto:matthew at anandabits.com>> wrote:
>>>>> For all those reasons, we really do need something like AnyObject dispatch if we care about working with dynamically typed languages.  The design I’m suggesting carefully cordons this off into its own struct type, so it doesn’t infect the rest of the type system, and is non-invasive in the compiler.
>>>> 
>>>> I am quite familiar with dynamic languages and agree that this is necessary if we are going to fully open up access to these languages from Swift.
>>> 
>>> Ok, then it appears you agree that something like anyobject dispatch is necessary for effective dynamic language interop.
>>> 
>>>>>> I strongly urge you to reconsider the decision of that dynamic members must be made available with no indication at usage sites.  An indication of dynamic lookup at usage sites aligns very well (IMO) with the rest of Swift (AnyObject lookup aside) by calling attention to code that requires extra care to get right.
>>>>> 
>>>>> I don’t understand this.  The proposal is fully type safe, and this approach is completely precedented by AnyObject.  Swift’s type system supports many ways to express fallibility, and keeping those decisions orthogonal to this proposal is the right thing to do, because it allows the author of the type to decide what model makes sense for them.
>>>> 
>>>> Allowing the author of the type to choose whether the mechanism is hidden or visible is exactly what I don’t want to allow.  I think you have the right design regarding types and semantics - the author chooses.  But I don’t want these calls to look like ordinary member lookup when I’m reading code.  
>>>> 
>>>> They inherently have a much greater chance of failure than ordinary member lookup.  Further, authors are likely to choose immediate traps or nil IUO as failure modes as forcing users to deal with Optional on every call is likely to be untenable.  I believe this behavior should be represented by some kind of syntax at the usage site.  I don’t believe it is an undue burden.  It would make the dynamic lookup semantic clear to all readers and would help to discourage abuse.
>>> 
>>> I believe that adding explicit syntax would be counterproductive to your goals, and would not make dynamic lookup syntax more clear.  I assume that you would also want the same thing for DynamicCallable too, and operator overloads, subscripts, and every other operation you perform on these values, since they all have the exact same behavior.
>>> 
>>> If we required some syntax even as minimal as “foo.^bar” and "baz^(42)”, that change would turn this (which uses runtime failing or IUO return values like AnyObject):
>>> 
>>> 	let np = Python.import("numpy")
>>> 	let x = np.array([6, 7, 8])
>>> 	let y =  np.arange(24).reshape(2, 3, 4)
>>> 	
>>> 	let a = np.ones(3, dtype: np.int32)
>>> 	let b = np.linspace(0, pi, 3)
>>> 	let c = a+b
>>> 	let d = np.exp(c)
>>> 	print(d)
>>> 
>>> into:
>>> 
>>> 	let np = Python.import("numpy")
>>> 	let b = np^.array^([6, 7, 8])
>>> 	let y =  np^.arange^(24)^.reshape^(2, 3, 4)
>>> 	
>>> 	let a = np^.ones^(3, dtype: np^.int32)
>>> 	let b = np^.linspace^(0, pi, 3)
>>> 	let c = a+^b
>>> 	let d = np^.exp^(c)
>>> 
>>> This does not improve clarity of code, it merely serves to obfuscate logic.  It is immediately apparent from the APIs being used, the API style, and the static types (in Xcode or through static declarations) that this is all Python stuff.  
>> 
>> It may be immediately apparent when the types involved are obviously dynamic, such as in this example where Python.import is explicitly used.  However, my concern is less about the intended use case of dynamic language interop than I am that this feature will be generally available to all types in Swift.  
>> 
>> This is big change from AnyObject dispatch.  It opens up the dynamism to types and contexts that are not necessarily obviously using dynamic lookup, callable, etc.  Maybe this won’t turn out to be a problem in practice but I still think it’s a legitimate concern.
> 
> If dynamism if restricted to subclasses of a DynamicObject type, like Xiaodi suggested earlier, then we can protect ourselves from this dynamic dispatch being generally available to all types in Swift.
> 
>>> When you start mixing in use of native Swift types like dictionaries (something we want to encourage because they are typed!) you end up with an inconsistent mismash where people would just try adding syntax or applying fixits continuously until the code builds.
>>> 
>>> Beyond that, it is counterproductive to your goals, because it means that people are far less likely to use to use optional returns.  Doing so (which produces a safer result) would cause a double tax in syntax, and would be a confusing jumble.  I can’t bring myself to do the whole example above, one line - just converting member lookup syntax but not callable syntax - would end up:
>>> 
>>> 	let y =  np^.arange?^(24)^.reshape^?(2, 3, 4)
>>> 
>>> If you made DynamicCallable also return optional it would be:
>>> 
>>> 	let y =  np^.arange?^(24)?^.reshape^?(2, 3, 4)!
>>> 
>>> or something.  This is such madness that no one would do that.
>> 
>> Yes, I agree.  The interaction with optional chaining makes it unworkable.  I hadn’t thought that all the way through.  Thank you for indulging in the discussion about this idea.  
>> 
>> I’m uncertain what the right answer is.  I’m still not really comfortable with opening up dynamic lookup to any user-defined type without some way to indicate to readers that dynamic lookup is happening in a piece of code.  Maybe there is a less localized annotation that would indicate dynamic lookup is in effect for a larger chunk of code.
> 
> I think that making dynamic calls syntactically different to readers is going too far in the direction of safety. Plus, it makes the language inconsistent as we already have AnyObject dispatch with exactly the same syntax. But I understand why you would want it if *any* type could end up being conformed to a dynamic lookupable/callable protocol. Like said above, I think that a DynamicObject type is enough protection to not bother making the syntax heavier at the point of use.
> 
>> One idea that hasn’t been explored yet is introducing a dynamic lookup effect.  That would provide clean syntax at the expression level while still making it clear that dynamic lookup is happening.  This probably isn’t the right approach as it would be viral and wouldn’t even indicate that the code in a specific function body even contains a dynamic lookup.  I mention it mostly to broaden the conversation and thought space about what kind of approach might address my concerns without cluttering up expressions.
>> 
>> Another, potentially more viable approach would be to use an approach similar to try and the proposed async of a statement modifier (possibly also allowed at the expression level as with try and async).  This could be used in conjunction with an effect as mentioned above but could also be used independently even though there isn’t a current precedent for that.  Your example written with this approach might be:
>> 
>> 	let np = Python.import("numpy")
>> 	let b = dynamic np.array([6, 7, 8])
>> 	let y =  dynamic np.arange(24).reshape(2, 3, 4)
>> 	
>> 	let a = dynamic np.ones(3, dtype: np.int32)
>> 	let b = dynamic np.linspace(0, pi, 3)
>> 	let c = dynamic a + b
>> 	let d = dynamic np.exp(c)
>> 
>> Note: `dynamic` is just a straw man here.  This is obviously a lot of boilerplate repetition of the same modifier.  I would also be perfectly happy allowing this annotation to apply to a block of code or even a whole function body.  Then it might be:
>> 
>> dynamic {
>> 	let np = Python.import("numpy")
>> 	let b = np.array([6, 7, 8])
>> 	let y = np.arange(24).reshape(2, 3, 4)
>> 	
>> 	let a = np.ones(3, dtype: np.int32)
>> 	let b = np.linspace(0, pi, 3)
>> 	let c = a + b
>> 	let d = np.exp(c)
>> }
>> 
>> The most obvious objection to this is that it introduces nesting and does so in a way that only very indirectly influences control flow (through the potential for dynamic failure).
>> 
>> My objective certainly isn’t to make code ugly and obfuscate logic.  It is simply to make the usage of dynamic features that are prone to failure at runtime immediately clear to a reader.  This should be as lightweight as possible while still providing valuable information to the reader.
>> 
>>> 
>>> 
>>>>> Swift already has a dynamic member lookup feature, "AnyObject dispatch" which does not use additional punctuation, so this would break precedent.
>>>> I would prefer if dynamic lookup were visible with AnyObject as well.  For that reason I don’t believe it makes a good precedent to follow.  In fact, I would prefer to see us go the other direction and perhaps even consider revising dynamic lookup syntax for AnyObject in the future.
>>> 
>>> This is definitely not going to happen.  The change Doug mentioned is to have AnyObject lookup return optional instead of IUO, which forces ? on the clients.  Adding other syntax (like you’re suggesting) is certainly not going to happen.
>>> 
>>> The entire point of AnyObject dispatch is to improve syntactic elegance and clarity of code using it.  There is no other reason to exist.  Making code that uses it syntactically onerous completely defeats the point of having it in the first place, as I’ve mentioned before.
>> 
>> I agree.  The interaction with optional chaining is significantly more onerous than I had considered.  I should have worked through an example on my own.  I do hope you will consider a less localized approach to usage-site annotation though.
>> 
>>> 
>>> 
>>> Furthermore, your premise that Swift does not have invisibly failable operations is plainly wrong.  Array subscript and even integer addition can fail.  
>> 
>> I am well aware of these behaviors.  The difference IMO is that programmers tend to be well aware of these preconditions even if they also often choose to ignore them.  Dynamic lookup will not be so clear.  This is especially true if people use it with types that also have an API available through static lookup.
>> 
>>> Even the behavior of AnyObject was carefully designed and considered, and were really really good reasons for it returning IUO.
>> 
>> I am not trying to call into question the choices made in the past.  Swift wouldn’t be the great language with a bright future that it is today without an incredibly successful migration of a large user base from Objective-C to Swift.  This is a huge accomplishment and couldn’t have happened without making really good decisions about some really hard tradeoffs.
>> 
>>> 
>>> -Chris
>>> 
>> _______________________________________________
>> swift-evolution mailing list
>> swift-evolution at swift.org <mailto:swift-evolution at swift.org>
>> https://lists.swift.org/mailman/listinfo/swift-evolution <https://lists.swift.org/mailman/listinfo/swift-evolution>
> _______________________________________________
> swift-evolution mailing list
> swift-evolution at swift.org <mailto:swift-evolution at swift.org>
> https://lists.swift.org/mailman/listinfo/swift-evolution <https://lists.swift.org/mailman/listinfo/swift-evolution>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-evolution/attachments/20171203/40d3bef3/attachment.html>


More information about the swift-evolution mailing list