[swift-evolution] Proposal: Introduce User-defined "Dynamic Member Lookup" Types

[Douglas Gregor]

> On Dec 1, 2017, at 9:37 AM, Chris Lattner <clattner at nondot.org> wrote:
> 
> On Dec 1, 2017, at 12:26 AM, Douglas Gregor <dgregor at apple.com <mailto:dgregor at apple.com>> wrote:
>>> On Nov 30, 2017, at 10:05 PM, Chris Lattner <clattner at nondot.org <mailto:clattner at nondot.org>> wrote:
>>> Hi Doug,
>>> 
>>> Thank you for the detailed email.  I have been traveling today, so I haven’t had a chance to respond until now.  I haven’t read the down-thread emails, so I apologize if any of this was already discussed:
>>> 
>>>> I think better interoperability with Python (and other OO languages in widespread use) is a good goal, and I agree that the implementation of the feature described is straight-forward and not terribly invasive in the compiler.
>>> 
>>> Fantastic, I’m really pleased to hear that!  I only care about solving the problem, so if we can find a good technical solution to the problems than I’ll be happy.
> 
> After thinking about your email a bit more, I think I might understand the disconnect we’re having.  I think we have different versions in mind of what “success” looks like:
> 
> I believe your view is that people start using Python APIs from their Swift code, but gradually add type annotations (either inline or in a sidecar database) to make those APIs progressively more Swifty.  New features will be added to Python (its type system, compilers, databases for common APIs, etc) to improve this interoperability, along the lines of what we’ve done for Objective-C over the years.  Fast forward several years, and large Python libraries would be nice to use from Swift - perhaps nicer than they are to use from Python itself.  This view aligns with what happened with Objective-C <-> Swift interoperability.
> 
> In contrast, I’m specifically interested in developers in certain large domains (e.g. data science and ML) which are “forced” to use Python because that is where all the libraries are.  I have spoken to many of these sorts of people, and a large number of them really *dislike* using Python for all the obvious reasons (including the tooling issues you point out).  My view of success is that we allow them to write all of *their code* in Swift, which will lead to a massive quality of life benefit for these frustrated people.  You’re right that they will still chaff when using imported Python APIs (which don’t feel “swifty” for LOTS of reasons - e.g. method naming and design patterns), but my view is that this will provide incentive for these people to provide a proper Swift implementation for these libraries over time.
> 
> In short, your end game is a pervasively intertwined Swift/Python world (like ObjC and Swift are).  My view is that Swift holds Python at arm's length, and wins over the hearts and minds of developers, leading to new Swift APIs designed for Swift.

That’s not the end game I’m looking at. I’d rather make it feasible to generate Swift wrappers for Python libraries that fit well into Swift’s existing machinery. The complexity goes outside the compiler, with small extralinguistic tie-ins to make it work well.

> I’m not sure if you agree with that portrayal of your position, and if not, I’m sorry and will try to understand another way.  However, if I’m close, then I have several concerns about that vision and don’t believe the end-game is achievable or better than what I’m proposing.  Consider:
> 
> - I don’t think there will be a lot of success getting people who *actually love* Python to use Swift, unless there is already an extrinsic reason for them to use it.
> - Type hints are not widely used in Python, and there are believable reasons that they won’t ever be.  Because they are specifically poorly suited for libraries, their use seems to be in “user’s own code” - but my proposal solves this already!  See below for examples.
> - Even if type hints were widely adopted, it would require massive extensions to them and to Python to make them be "good enough” to provide value for the things you’re envisioning.  Analogs to instancetype, objc generics, lots of the C macros, and many of the other things we’ve added to ObjC would have to be added.
> - The Python community has no reason to do this work for the Swift community, or accept changes to Python that are required to make this actually great.
> - Beyond the core type system, the Python and Objective-C languages work extremely differently in other ways (e.g. lack of umbrella headers making AnyObject-style dispatch questionable).
> - Even with those annotations and all the work, the APIs we’d end up with are not going to be good Swift APIs.  I don’t think that “renamification” and "IUO audits" would ever actually happen in practice, for example.
> - I believe the engineering effort required to implement this vision is so massive (including the changes to Swift, Python, and Python libraries) that it simply will never actually happen.

They don’t have to be good, strictly-typed APIs to be useful. They should have method/property names, have documentation, and be associated with named classes to work well with existing tooling.

> More concerning to me is that your design of using the existing AnyObject type presents a really concerning technical problem, scalability: It is not simply a Swift/ObjC/Python world, Javascript is also super important.  There are also a number of other less-widely used dynamic languages that are interesting.  Your design leads to them all being mashed together into a common runtime system.  I cannot imagine how this would end up being a good thing for system complexity, and would lead to each of them being jeopardized in different ways.  As I have mentioned before numerous times, it also opens the door for enormous Swift compiler complexity, as each of the object models need to be supported in the compiler (so you can subclass each languages’ types), and many other complexities.

The AnyObject model of dispatching to any method within a given dynamic space (whether it’s Objective-C or Python or something else) fits well into Swift. You’re absolutely right that we can’t make Python objects work with AnyObject.

> If you are serious about improving the tooling situation for people using Python APIs in Swift, the most natural way to do so is to follow the approach that the MyPy community (they are the ones who have thought about this the most) is using: provide progressive typing extensions, use data flow analysis to propagate around types, and enhance the tooling to have support this.  I’m skeptical that this will ever be worthwhile, but a sketch could look like this:

I doubt this would be worthwhile. If we wanted to do something along these lines, we’d want to seriously investigate something akin to gradual typing.
> 
> Similarly, there is no reason to do anything to make the second and third examples work nicely in Swift: you’d just define a Swift class, and a Swift function.
> 
> MyPy:
> 
> class BankAccount:
>     def __init__(self, initial_balance: int = 0) -> None:
>         self.balance = initial_balance
>     def deposit(self, amount: int) -> None:
>         self.balance += amount
>     def withdraw(self, amount: int) -> None:
>         self.balance -= amount
>     def overdrawn(self) -> bool:
>         return self.balance < 0
> 
> my_account = BankAccount(15)
> my_account.withdraw(5)
> print(my_account.balance)
> Swift:
> 
> class BankAccount { 
>    var balance : Int
>    init(initialBalance: Int) { … }
>   func deposit(….
> }
> 
> Even the proposals that I’m making - simple and isolated though they are - are enough to provide a major quality of life improvement for people writing large amounts of code against Python APIs.

They’re an improvement to the resulting code (i.e., the Swift code looks comparable to the Python code), but it’s not a quality-of-life improvement for the process of writing the code (or understanding it once it’s written), because you don’t get any of the tooling benefits you’re used to having as a Swift programmer: no code completion, no “”quick help”, no go-to-definition.

>  Beyond that, they are small extensions with low complexity, scale to supporting many different dynamic languages over time,  require a level of engineering effort that is plausible to be built, and do not require some sort of "executive buy in” from the Python community.

Here’s an alternative proposal that provides a better development experience: write a wrapper generator, in Python, that maps a Python module’s interfaces to Swift, and provide small tie-ins to the compiler to make it fit well. Take the example from the DynamicMemberLookup proposal:

class Dog:
    """Man's best friend"""
    def __init__(self, name):
        """Summon a new dog"""
        self.name = name
        self.tricks = []    # creates a new empty list for each dog

    def add_trick(self, trick):
        """Teach the old dog a new trick"""
        self.tricks.append(trick)


The output of the wrapper generator is:

/// Man's best friend
@_foreign(python)
class Dog : PythonObject {
  /// Summon a new dog
  @_foreign(python, “__init__”)
  init(_ name: PythonArgument) { /* code to dynamically call __init__  on the Dog class object*/ }

  /// Teach the old dog a new trick
  @_foreign(python, “add_trick”)
  func add_trick(_ trick: PythonArgument) -> PythonObject? { /* code to dynamically call add_trick */ }
}

Several pieces above require explanation, but first the code to use this from Swift:

import DogModule
let dog = Dog("Brianna")
dog.add_trick("Roll over")
log do2 = Dog("Kaylee").add_trick("snore")

Now, because we have declarations in the Swift module, the tools helped us write this code: I got code completion for the “Dog” type, its initializers, its “add_trick” method. The Python docstrings were carried through into documentation comments that show up in my IDE. I can inspect the Swift interface created by the wrapper generator to see what it looks like; my class browser can show me the Python class hierarchy.

Let’s break down the pieces we need to get here:

PythonObject: this would be a root class, defined by the Python interoperability library:

@_foreign(python)
class PythonObject {
  static func retain(_ object: UnsafePointer<Void>) { Py_INCREF(object) }
  static func release(_ object: UnsafePointer<Void>) {Py_DECREF(object) }
}

PythonObject is a foreign class for the Python language (hence the @_foreign). So while you get subtyping as normal (Dog is a subtype of PythonObject), they aren’t a subtype of AnyObject because it’s not a native object model. Nor could you mix a foreign class marked as @_foreign(python) with one marked as @_foreign(ruby), but they’re different object models.

The retain/release operations would be called by the compiler in the obvious places. It shouldn’t be too difficult to build an appropriate value witness table to make this work.

PythonArgument: This is a protocol to which any type can conform to map it into Python, and IIRC you already have this in your design in some form:

protocol PythonArgument {
  func toPythonObject() -> PythonObject
}

extension PythonObject: PythonArgument {
  func toPythonObject() -> PythonObject { return self }
}

extension Int: PythonArgument {
  func toPythonObject() -> PythonObject { … }
}

@_foreign on methods/properties/etc.: This is there to enable AnyObject-style dispatch. Consider the add_trick example:

  @_foreign(python, “add_trick”)
  func add_trick(_ trick: PythonArgument) -> PythonObject { /* code to dynamically call add_trick */ }

The two arguments to @_foreign are the foreign-class-model indicator (python): one gets AnyObject lookup from objects whose type is in that same foreign class space (e.g., PythonObject is also in the “python” foreign class model), and the second string (here, “add_trick”) is the uniquing string that indicates what name would be passed through to the underling runtime. This becomes important when you have overlap at different parts in the class hierarchy, e.g.,

@_foreign(python)
class MagicShow: PythonObject {
  @_foreign(python, “add_trick”)
  func add_trick(_ trick: PythonArgument) -> PythonObject? { … }
}

And then we have something like:

func foo(po: PythonObject) {
  po.add_trick?(“something”)    // finds both Dog.add_trick(_:) and MagicShow.add_trick(_:), but since they’re both uniqued by “add_trick” and have the same signature, we can pick one arbitrarily
}

This is a generalization of how AnyObject dispatch already works in the type checker. For Objective-C, AnyObject is the root for the “objc” class model and the Objective-C selector (with + or - prepended) is used as the uniquing key; you can see the code at https://github.com/apple/swift/blob/master/lib/Sema/ConstraintSystem.cpp#L265-L273.

Factory initializers: unfortunately, to actually make initializers work cleanly, we’ll need something like factory initializers in the language. We should have those anyway, but until we get them one can fake it with top-level functions:

@_foreign(python, “__init__”)
func Dog(_ name: PythonArgument) -> Dog { /* … */ }

“Truly” dynamic calls: Python code does dynamically create new properties, methods, classes, etc. Those won’t get reflected into the generated Swift interface (because the wrapper generator won’t see them when it just imports the module to inspect it). So, there needs to be some string-based interface for such dynamic calls, which would be provided by the interoperability library in whatever syntax makes it clear what’s happening:

extension PythonObject {
  subscript(property name: String) -> PythonObject? { get { /* … */ } set { /* … */ }
  func call(_ methodName: String, _ args: PythonArgument…, keywordArgs: [String: PythonArgument] = [:]) -> PythonObject? { … }
}

This enables the fully-dynamic behavior, but makes it clear when we’ve conjured up the name as a string.

On-demand wrapper generation: you’ve mentioned earlier in this thread that it’s important to support the playgrounds use case, where one can import whatever Python modules are available to play with them. I *personally* don’t think running a one-off wrapper script once per module is all that onerous, but if it is, we could extend the Swift compiler with a very light plugin mechanism where the plugin mechanism has two operations:

1) Enumerate the names of modules that it can translate (e.g., find all of the Python modules in the Python path and current working directory), and
2) Given a module name, produce the Swift wrapper code for the Swift compiler to then compile (and cache)

Overall, it’s probably more work than DynamicMemberLookup, but it provides a more friendly development experience because the entities in the Python module show throw in Swift—they’re documented, inspectable, code-completable, etc. We get the class hierarchy reflected into Swift so we get proper subtyping behavior. And writing these wrapper generators is something that is easily hackable by anyone who knows the languages involved, it’s easy to test and improve over time.

	- Doug


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-evolution/attachments/20171219/48514157/attachment.html>