[swift-evolution] Proposal: Introduce User-defined "Dynamic Member Lookup" Types

[Douglas Gregor]

> On Nov 30, 2017, at 10:05 PM, Chris Lattner <clattner at nondot.org> wrote:
> 
> Hi Doug,
> 
> Thank you for the detailed email.  I have been traveling today, so I haven’t had a chance to respond until now.  I haven’t read the down-thread emails, so I apologize if any of this was already discussed:
> 
>> I think better interoperability with Python (and other OO languages in widespread use) is a good goal, and I agree that the implementation of the feature described is straight-forward and not terribly invasive in the compiler.
> 
> Fantastic, I’m really pleased to hear that!  I only care about solving the problem, so if we can find a good technical solution to the problems than I’ll be happy.
> 
> A funny thing about swift-evolution is that it is populated with lots of people who already use Swift and want to see incremental improvements, but the people who *aren’t* using Swift yet (but should be) aren’t represented here.  As you know, I’m perhaps the biggest proponent of Swift spreading into new domains and earning the love of new users.

While I, too, am interested in attracting new users to Swift, it should not come at the cost of making the language less coherent.

>> However, I do not think this proposal is going in the right direction for Swift. I have objections on several different grounds.
> 
> Thanks for being up front about this.  It turns out that a majority of the points you raise were brought up early in the pitch phases of the discussion, but I screwed up by not capturing that discussion into the alternatives section of the proposal.
> 
> Better late than never I guess: I just significantly revised the proposal, adding a new “Alternative Python Interoperability Approaches” section.  I’d appreciate it if you could read it and comment if you find any part disagreeable:
> https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438#alternative-python-interoperability-approaches <https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438#alternative-python-interoperability-approaches>
> 
> Many of the points you make are discussed there, but I’ll respond point-by-point below as well:
> 
>> Philosophy
>> Swift is, unabashedly, a strong statically-typed language.
> 
> That’s factually incorrect.

You’re going to have to explain that statement without reference to AnyObject (we’ll discuss that case below).

>> We don’t allow implicit down casting, we require “as?” so you have to cope with the possibility of failure (or use “as!” and think hard about the “!”). Even the gaping hole that is AnyObject dispatch still requires the existence of an @objc declaration and produces an optional lookup result, so the user must contend with the potential for dynamic failure.
> 
> AnyObject dispatch is one of the gaping holes for sure.  You don’t mention it, but its lookup results are represented as *ImplicitlyUnwrappedOptional* results, which do not force the user to contend with dynamic failure.  Further, unlike my proposal (which is fully type safe), AnyObject lookup is not type safe at all.

As noted, AnyObject is a gaping hole, and the fact that it still produces an ImplicitlyUnwrappedOptional does make my statement weaker. It means you can dynamically end up with “unrecognized selector” without having acknowledged the possibility with a ‘!’.

>  It is also far more invasively intertwined throughout the compiler.

Not so much any more; there’s the weird lookup rule and some expression kinds that get mapped directly through to objc_msgSend.

> 
> More problematically for your argument: your preferred approach requires the introduction of (something like) DynamicMemberLookupProtocol or something like AnyObject-For-Python, so your proposal would be additive on top of solving the core problem I’m trying to solve.  It isn’t an alternative approach at all.

I wouldn’t say that’s my preferred approach. My preferred approach involves taking the method/property/etc. declarations that already exist in Python and mapping them into corresponding Swift declarations so we have something to find with name lookup. One could put all of these declarations on some PyVal struct or PythonObject and there would be no need for AnyObject-for-Python or DynamicMemberLookupProtocol.

>> Whenever we discuss adding more dynamic features to Swift, there’s a strong focus on maintaining that strong static type system.
> 
> Which this does, by providing full type safety - unlike AnyObject lookup.

You get dynamic safety because it goes into the Python interpreter; fair enough. You get no help from your tools to form a correct invocation of any method provided by Python.

> 
>> IMO, this proposal is a significant departure from the fundamental character of Swift, because it allows access to possibly-nonexistent members (as well as calls with incorrect arguments, in the related proposal) without any indication that the operation might fail.
> 
> The only way your claim is correct is if someone implements the protocol wrong.  What you describe is true of AnyObject lookup though, so I understand how you could be confused by that.

AnyObject lookup still requires you to find an actual declaration with a type signature. Yes, there are still failure cases. The dynamic type might be totally unrelated to the class in which you found the declaration you’re supposedly calling, which is a typical “unrecognized selector” failure and will always be an issue with dynamic typing. The actual type safety hole you’re presumably referring to is that the selector could be overloaded with a different type signature, and we don’t proactively check that the signature we type-checked against matches the signature found at runtime. It’s doable with the Objective-C method encodings, but has never been considered worthwhile.

> 
>> It’s easy to fall through these cracks for any type that supports DynamicMemberLookupProtocol—a single-character typo when using a DynamicMemberLookupProtocol-capable type means you’ve fallen out of the safety that Swift provides.
> 
> Since you seem to be latching on to this, I’ll say it again: the proposal is fully type safe :-)
> 
> That said, the “simple typo” problem is fundamental to the problem of working with a dynamically typed language: unless you can eradicate all “fundamental dynamism” from the language, you cannot prevent this.
> 
> That said, since this is a problem inherent with these languages, it is something people are already very familiar with, and something that everyone using those APIs has had to deal with.  This is also not our problem to solve, and people in the Python community have been discussing and working on it over a large part of its 25 year history.  I find your belief that we could solve this for Python better than the Python community itself has both idealistic and a bit naive.

I’m not pretending we can fully solve the problem. I’m pointing out that depending entirely on DynamicMemberLookupProtocol throws away information about method declarations that is present in Python and used by Python tooling to good effect.


>> Tooling
>> The Python interoperability enabled by this proposal *does* look fairly nice when you look at a small, correctly-written example. However, absolutely none of the tooling assistance we rely on when writing such code will work for Python interoperability. Examples:
>> 
>> * As noted earlier, if you typo’d a name of a Python entity or passed the wrong number of arguments to it, the compiler will not tell you: it’ll be a runtime failure in the Python interpreter. I guess that’s what you’d get if you were writing the code in Python, but Swift is supposed to be *better* than Python if we’re to convince a community to use Swift instead.
>> * Code completion won’t work, because Swift has no visibility into declarations written in Python
>> * Indexing/jump-to-definition/lookup documentation/generated interface won’t ever work. None of the IDE features supported by SourceKit will work, which will be a significant regression for users coming from a Python-capable IDE.
> 
> Yes, welcome to the realities of modern Python development!

Python plugins for IDEs (e.g., for Atom) provide code completion, goto definition, and other related features. None of the Swift tooling will work if Swift’s interoperability with Python is entirely based on DynamicMemberLookupProtocol.

> 
>> Statically-typed languages should be a boon for tooling, but if a user coming from Python to Swift *because* it’s supposed to be a better development experience actually sees a significantly worse development experience, we’re not going to win them over. It’ll just feel inconsistent.
> 
> By your argument we should ban AnyObject lookup as well, given its inconsistency with the rest of the language.

By my argument, we should at least replace the ImplicitlyUnwrappedOptional result with a true Optional (so one has to acknowledge that the method shouldn’t be there). We’ve seriously considered it, but it’s a source-breaking change, and it hasn’t seemed worth the engineering effort to pursue it.

I don’t think that removing AnyObject dispatch entirely is possible at this point in Swift’s lifetime. While AnyObject has become much less prominent than it was in the Swift 1.0 days ([AnyObject] and [NSObject : AnyObject], oh my!), there is still a significant amount of code using it in the wild.

> I’m glad you mention the idea of users coming from Python to Swift: by many estimates, the Python community is *two* orders of magnitude or more larger than the Swift community - and lets not mention other dynamic languages like Javascript.  It really is high value to make the path from Python to Swift nicer, and if we do so, I believe people will find lots of reasons to write new Swift code to get all the advantages that you mention (and more).
> 
>> Dynamic Typing Features
>> It’s possible that the right evolutionary path for Swift involves some notion of dynamic typing, which would have a lot of the properties sought by this proposal (and the DynamicCallableProtocol one). If that is true—and I’m not at all convinced that it is—we shouldn’t accidentally fall into a suboptimal design by taking small, easy, steps.
> 
> Given that you haven’t followed the discussion on the many threads we’ve had on this, and haven’t proposed a workable approach to this problem, I’m not sure upon what basis your fears and uncertainty and doubt are founded.

A few meta-comments here. First of all, following all previous threads on a discussion is not realistic. This is part of the reason why we have different stages in a proposal’s lifetime, and is the responsibility of the proposal’s authors to capture alternatives and rationale in the proposal to make it self-contained. This proposal went through rapid iteration in the pitch phase and has now been elevated to a pull request to ask for formal review—you should expect more people to come on board having not read those threads. I appreciate that you have now captured more alternatives and rationale in the proposal.

Second, it is absolutely reasonable to disagree with the technical direction of a proposal without providing a complete solution to the problem that the proposal is attempting to solve. Some problems aren’t worth solving at all, or fully.

>> How Should Python Interoperability Work?
>> Going back to the central motivator for this proposal, I think that providing something akin to the Clang Importer provides the best interoperability experience: it would turn Python declarations into *real* Swift declarations, so that we get the various tooling benefits of having a strong statically-typed language.
> 
> This could be an theoretically interesting refinement to this proposal but I’m personally very skeptical that this is every going to happen.  I’ve put the rationale into the alternatives section of the proposal.  I don’t explain it in the proposal in this way directly, but I believe it is far more likely for a Pythonista transplant into Swift to rewrite their code in Swift than it is to use Python type annotations.

I assume that this belief is based on type annotations lack of traction in the Python community thus far?

>> Sure, the argument types will all by PyObject or PyVal,
> 
> That’s the root of the problem.  Python has the “fully dynamic” equivalent of “id” in Objective-C, so we need to represent that somehow.  Even if we followed the implementation approach of the Clang importer, we would need some way to represent this dynamic case.  That type needs features like DynamicMemberLookup or AnyObject.  In my opinion, the DynamicMemberLookup approach is better in every way than AnyObject is.  

The AnyObject approach has the advantage of knowing the set of declared, reachable APIs:

* Code completion shows all of the APIs that are possible to use via dynamic dispatch, with their signatures so can fill in the right # of arguments, see the names of the parameters, see documentation, etc.
* Indexing/refactoring/goto definition all point you to the declarations that could be the targets of dynamic dispatch

The DynamicMemberLookup approach is better for cases where you don’t have a declaration of the member you want to access. I suspect that’s not the common case.

> Which approach do you think is the best way to handle the untyped “actually dynamic” case? 

AnyObject already exists in the language, and it fits the untyped “actually dynamic” case well. It does require having a declaration for the thing you want to reference, which I consider to be important: we can code-complete those declarations, goto-definition to see those declarations, index/refactor/look up documentation based on those declarations.

I’d be more inclined to push for the ImplicitlyUnwrappedOptional -> Optional change if we did something to make AnyObject more prominent in Swift.

>> but the names are there for code completion (and indexing, etc.) to work, and one could certainly imagine growing the importer to support Python’s typing annotations <https://docs.python.org/3/library/typing.html>.
> 
> You’re basing this on the flawed assumption that local variables will pervasively have types, which I can’t imagine being the case.  Even on "typable” API, I wouldn’t expect people to commonly get code completion results for reasons now explained in the proposal.

Remember that one *does* get code completion results for member access into an AnyObject… lots of them… but the list filters down pretty fast when you type a few characters, and then you get a member access that’ll fill in stubs for (say) the arguments to the method you were trying to call. But you can’t get those code completion results without having declarations to complete to.

> 
>> But the important part here is that it doesn’t change the language model at all—it’s a compiler feature, separate from the language. Yes, the Clang importer is a big gnarly beast—but if the goal is to support N such importers, we can refactor and share common infrastructure to make them similar, perhaps introducing some kind of type provider infrastructure to allow one to write new importers as Swift modules.
> 
> Sure, doing something like this could be an interesting direction to consider after getting basic fully dynamic support in.  
> 
> In earlier threads, we had extensive discussion about type providers, and discussed that they don’t solve the problem for the same reasons that Clang importer doesn’t solve the problem: you still need the way to “provide” the fully dynamic type.
> 
> If you have to do that, it is perfectly reasonable to get the basic dynamic case in, then weigh the cost/benefit advantage of getting more static typing in if available somehow.  While importing progressive typing annotations could be practical for Javascript interop (for example) I’m pretty skeptical it would be  widely used for Python (again, rationale laid out in the proposal).

I’m saying we can do this instead of fully dynamic support; see below.

> 
>> In truth, you don’t even need the compiler to be involved. The dynamic “subscript” operation could be implemented in a Swift library, and one could write a Python program to process a Python module and emit Swift wrappers that call into that subscript operation. You’ll get all of the tooling benefits with no compiler changes, and can tweak the wrapper generation however much you want, using typing annotations or other Python-specific information to create better wrappers over time.
> 
> I’d love for you to sketch out how any of this works with an acceptable user experience, because I don't see anything concrete here.  

We don’t need the basic dynamic case in the language to do this experiment. Take the PyVal struct from the proposal. Now, write a Python script that loads some module Foo and uses Python’s inspect <https://docs.python.org/3/library/inspect.html> module to go find the classes, methods, etc., and pretty-print Swift code that uses PyVal. So this:

	def add_trick(self, trick):

turns into

	extension PyVal {
	  func add_trick(_ trick: PyVal) -> PyVal {
	    /* do the magic to call into Python */
	  }
	}

Using the inspect module, you can extract parameter names, default arguments, docstrings, and more to reflect the existing Python API as Swift API, packed into a bridging module.

Note that we have a “flat” namespace of all Python methods on PyVal, which is basically what you get with AnyObject today. Swift tooling will provide code completion for member accesses into PyVal. Goto definition will jump to the pretty-printed declarations, which could have the docstrings formatted in comments and would show up in QuickHelp. The types are weak (everything is PyVal), but that’s what we expect from importing a dynamically-typed language.

> You seem to be thinking that there is some world where Python isn’t actually a dynamic language, or where the Python community switches to pervasive adoption of type hints (something that is unlikely to happen for a ton of reasons).  Python is less statically typable than Objective-C is, even ignoring the constraints on the problem, and yet Objective-C has AnyObject - why do you think that Python wouldn’t need it?  Since it needs it, then we need to design it: how do you think it should work?

Dynamic typing doesn’t have to mean “give up on all static checking”; it means that we don’t restrict the set of operations that the language permits you to apply to a given value based on the static type of that value. The PyVal extensions I describe above, like AnyObject, give you access to all declared APIs to which we can dynamically dispatch.

	- Doug

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-evolution/attachments/20171201/d7b0128b/attachment.html>