[swift-evolution] Proposal: Introduce User-defined "Dynamic Member Lookup" Types

[Jacob Williams]

I’ve been going back and forth on whether I’m for or against the current revision of the proposal and I’d have to say that I am in favor of it as is for the following reasons (many of which have already been stated):

It was stated early on in this proposal that Python is not going to be given the same first class support that C/Objective-C have in the compiler. It would be unreasonable and would set a precedent that all languages would get the same treatment when the community wants to add interoperability. Many arguments have been made that we NEED to have static typing for dynamic languages. I don’t see static typing coming without first class compiler support. 

Also, the very nature of these dynamic languages means static typing would violate basic principles of the dynamic language. I think it would be much worse and scare away python developers who are suddenly forced to statically type things that they may know to be dynamic types in their python APIs. Of course, the other side of the coin is that swift developers would be scared away by the lack of static typing when using Python interop. I see the latter as a non-issue though because anyone using python knows to expect things without static types. 

Forcing static types where there are no static types sounds like a really bad idea to me. So I’m ok with the proposed PyVal type for all python types. Extensions can be written to make casting a PyVal object to the various swift types easy (if/when desired).

There has also been much discussion on the potential for abuse. There was an example given of adding conformance to NSObject I believe. Chris and Xiaodi both proposed solutions to this that would minimize (and possibly eliminate) the potential for abuse. 

However, there will always be potential for a feature to be misused. The potential does not mean it is an absolute given that it will be misused (I’m an optimist if you can’t tell ;).

I don’t view this protocol as a beginner user feature. This is not a protocol that will be commonly used or even commonly desired. It has a very specific set of uses and I strongly believe that the community will continue to do a good job at suggesting the right tool for the right job. The advanced users that are aware of this protocol will most likely properly recommend this protocol when applicable. Your average Joe Shmoe isn’t going to even be aware of this. Those who are curious, will probably ask about it. There will be StackOverflow responses describing it and what it does and when/why to use it. (This is just my two cents based on what I’ve witnessed in the swift community and on StackOverflow in the past). I’m sure that if the documentation on this protocol says “Used to provide interoperability with dynamic languages” then most users are going to steer clear of it unless they are writing an interop layer with a dynamic language.

I don’t think the goal here is to write Python (or ruby or javascript or [dynamic language]) in swift. The aim of this proposal it to make it easy and possible to use python from swift. I don’t expect to write a full blown python project in swift. I expect to call the few python APIs from module xyz and be done with python. I’m trying to write a swift project not a python one. If I wanted to use python extensively then I would just stick with python (I work as a python developer and Python is definitely not my language of choice, but it does have its upsides). I don’t want to write my own swift implementation for something where a python module already exists so I import the python module and do the few things with it that I actually need. I don’t need or expect IDE integration with the language I’m using at that moment. I can read docs for my 2-3 python API calls while I’m developing it.

Even with this proposal, people could write their own swift libraries that are just statically typed wrappers around a python library. This proposal does not limit people in any way. It does however, open the door for interoperability with a large set of dynamic languages.

> On Dec 1, 2017, at 10:37 AM, Chris Lattner via swift-evolution <swift-evolution at swift.org> wrote:
> 
> On Dec 1, 2017, at 12:26 AM, Douglas Gregor <dgregor at apple.com <mailto:dgregor at apple.com>> wrote:
>>> On Nov 30, 2017, at 10:05 PM, Chris Lattner <clattner at nondot.org <mailto:clattner at nondot.org>> wrote:
>>> Hi Doug,
>>> 
>>> Thank you for the detailed email.  I have been traveling today, so I haven’t had a chance to respond until now.  I haven’t read the down-thread emails, so I apologize if any of this was already discussed:
>>> 
>>>> I think better interoperability with Python (and other OO languages in widespread use) is a good goal, and I agree that the implementation of the feature described is straight-forward and not terribly invasive in the compiler.
>>> 
>>> Fantastic, I’m really pleased to hear that!  I only care about solving the problem, so if we can find a good technical solution to the problems than I’ll be happy.
> 
> After thinking about your email a bit more, I think I might understand the disconnect we’re having.  I think we have different versions in mind of what “success” looks like:
> 
> I believe your view is that people start using Python APIs from their Swift code, but gradually add type annotations (either inline or in a sidecar database) to make those APIs progressively more Swifty.  New features will be added to Python (its type system, compilers, databases for common APIs, etc) to improve this interoperability, along the lines of what we’ve done for Objective-C over the years.  Fast forward several years, and large Python libraries would be nice to use from Swift - perhaps nicer than they are to use from Python itself.  This view aligns with what happened with Objective-C <-> Swift interoperability.
> 
> In contrast, I’m specifically interested in developers in certain large domains (e.g. data science and ML) which are “forced” to use Python because that is where all the libraries are.  I have spoken to many of these sorts of people, and a large number of them really *dislike* using Python for all the obvious reasons (including the tooling issues you point out).  My view of success is that we allow them to write all of *their code* in Swift, which will lead to a massive quality of life benefit for these frustrated people.  You’re right that they will still chaff when using imported Python APIs (which don’t feel “swifty” for LOTS of reasons - e.g. method naming and design patterns), but my view is that this will provide incentive for these people to provide a proper Swift implementation for these libraries over time.
> 
> In short, your end game is a pervasively intertwined Swift/Python world (like ObjC and Swift are).  My view is that Swift holds Python at arm's length, and wins over the hearts and minds of developers, leading to new Swift APIs designed for Swift.
> 
> 
> I’m not sure if you agree with that portrayal of your position, and if not, I’m sorry and will try to understand another way.  However, if I’m close, then I have several concerns about that vision and don’t believe the end-game is achievable or better than what I’m proposing.  Consider:
> 
> - I don’t think there will be a lot of success getting people who *actually love* Python to use Swift, unless there is already an extrinsic reason for them to use it.
> - Type hints are not widely used in Python, and there are believable reasons that they won’t ever be.  Because they are specifically poorly suited for libraries, their use seems to be in “user’s own code” - but my proposal solves this already!  See below for examples.
> - Even if type hints were widely adopted, it would require massive extensions to them and to Python to make them be "good enough” to provide value for the things you’re envisioning.  Analogs to instancetype, objc generics, lots of the C macros, and many of the other things we’ve added to ObjC would have to be added.
> - The Python community has no reason to do this work for the Swift community, or accept changes to Python that are required to make this actually great.
> - Beyond the core type system, the Python and Objective-C languages work extremely differently in other ways (e.g. lack of umbrella headers making AnyObject-style dispatch questionable).
> - Even with those annotations and all the work, the APIs we’d end up with are not going to be good Swift APIs.  I don’t think that “renamification” and "IUO audits" would ever actually happen in practice, for example.
> - I believe the engineering effort required to implement this vision is so massive (including the changes to Swift, Python, and Python libraries) that it simply will never actually happen.
> 
> 
> More concerning to me is that your design of using the existing AnyObject type presents a really concerning technical problem, scalability: It is not simply a Swift/ObjC/Python world, Javascript is also super important.  There are also a number of other less-widely used dynamic languages that are interesting.  Your design leads to them all being mashed together into a common runtime system.  I cannot imagine how this would end up being a good thing for system complexity, and would lead to each of them being jeopardized in different ways.  As I have mentioned before numerous times, it also opens the door for enormous Swift compiler complexity, as each of the object models need to be supported in the compiler (so you can subclass each languages’ types), and many other complexities.
> 
> 
> If you are serious about improving the tooling situation for people using Python APIs in Swift, the most natural way to do so is to follow the approach that the MyPy community (they are the ones who have thought about this the most) is using: provide progressive typing extensions, use data flow analysis to propagate around types, and enhance the tooling to have support this.  I’m skeptical that this will ever be worthwhile, but a sketch could look like this:
> 
> 
> Consider the first example from: http://mypy-lang.org/examples.html <http://mypy-lang.org/examples.html>
> 
> With my proposals you could write the dictionary part very similar to the Python part (other pieces of the example, e.g. list comprehensions are uninteresting to me):
> 
> let d: PyVal = {:}
> …
> d[word] = d.get(word, 0) + 1
> 
> We can teach the compiler simple data flow analysis, to know that ‘d’ is a Python dictionary.  We could then allow the user to go further, with some new syntax along the lines of:
> 
> let d: @type(Dict<String, Int>) PyVal = {:}        // Lots of other syntactic choices possible.
> …
> d[word] = d.get(word, 0) + 1
> 
> 
> 
> However, the really important thing to recognize about these examples it that the type annotations are really poor at representing “generic” code like common Python libraries.  They are designed for “user code” - like that on the mypy web page - which use concrete types.
> 
> As it turns out, my proposal provides a solution for this part of the problem, because we’re allowing user code to be written in Swift!  A Swift programmer working with Python APIs would actually write this as:
> 
> 
> let d: Dictionary<String,Int> = {:}
> …
> d[word, default: 0] += 1
> 
> 
> Similarly, there is no reason to do anything to make the second and third examples work nicely in Swift: you’d just define a Swift class, and a Swift function.
> 
> MyPy:
> 
> class BankAccount:
>     def __init__(self, initial_balance: int = 0) -> None:
>         self.balance = initial_balance
>     def deposit(self, amount: int) -> None:
>         self.balance += amount
>     def withdraw(self, amount: int) -> None:
>         self.balance -= amount
>     def overdrawn(self) -> bool:
>         return self.balance < 0
> 
> my_account = BankAccount(15)
> my_account.withdraw(5)
> print(my_account.balance)
> Swift:
> 
> class BankAccount { 
>    var balance : Int
>    init(initialBalance: Int) { … }
>   func deposit(….
> }
> 
> Even the proposals that I’m making - simple and isolated though they are - are enough to provide a major quality of life improvement for people writing large amounts of code against Python APIs.  Beyond that, they are small extensions with low complexity, scale to supporting many different dynamic languages over time,  require a level of engineering effort that is plausible to be built, and do not require some sort of "executive buy in” from the Python community.
> 
> -Chris
> 
> 
> 
> 
> _______________________________________________
> swift-evolution mailing list
> swift-evolution at swift.org
> https://lists.swift.org/mailman/listinfo/swift-evolution

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-evolution/attachments/20171201/08cfbd04/attachment.html>