<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Dec 1, 2017, at 9:37 AM, Chris Lattner <<a href="mailto:clattner@nondot.org" class="">clattner@nondot.org</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><meta http-equiv="Content-Type" content="text/html charset=utf-8" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">On Dec 1, 2017, at 12:26 AM, Douglas Gregor <<a href="mailto:dgregor@apple.com" class="">dgregor@apple.com</a>> wrote:<div class=""><blockquote type="cite" class=""><div class=""><meta http-equiv="Content-Type" content="text/html; charset=utf-8" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class=""><blockquote type="cite" class=""><div class="">On Nov 30, 2017, at 10:05 PM, Chris Lattner <<a href="mailto:clattner@nondot.org" class="">clattner@nondot.org</a>> wrote:</div><div class=""><meta http-equiv="Content-Type" content="text/html charset=utf-8" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">Hi Doug,</div><div class=""><br class=""></div><div class="">Thank you for the detailed email. I have been traveling today, so I haven’t had a chance to respond until now. I haven’t read the down-thread emails, so I apologize if any of this was already discussed:</div><div class=""><br class=""></div><div class=""><blockquote type="cite" class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class="">I think better interoperability with Python (and other OO languages in widespread use) is a good goal, and I agree that the implementation of the feature described is straight-forward and not terribly invasive in the compiler.</div></div></div></blockquote><div class=""><br class=""></div><div class="">Fantastic, I’m really pleased to hear that! I only care about solving the problem, so if we can find a good technical solution to the problems than I’ll be happy.</div><div class=""></div></div></div></div></blockquote></div></div></div></blockquote><div class=""><br class=""></div><div class="">After thinking about your email a bit more, I think I might understand the disconnect we’re having. I think we have different versions in mind of what “success” looks like:</div><div class=""><br class=""></div><div class="">I believe your view is that people start using Python APIs from their Swift code, but gradually add type annotations (either inline or in a sidecar database) to make those APIs progressively more Swifty. New features will be added to Python (its type system, compilers, databases for common APIs, etc) to improve this interoperability, along the lines of what we’ve done for Objective-C over the years. Fast forward several years, and large Python libraries would be nice to use from Swift - perhaps nicer than they are to use from Python itself. This view aligns with what happened with Objective-C <-> Swift interoperability.</div><div class=""><br class=""></div><div class="">In contrast, I’m specifically interested in developers in certain large domains (e.g. data science and ML) which are “forced” to use Python because that is where all the libraries are. I have spoken to many of these sorts of people, and a large number of them really *dislike* using Python for all the obvious reasons (including the tooling issues you point out). My view of success is that we allow them to write all of *their code* in Swift, which will lead to a massive quality of life benefit for these frustrated people. You’re right that they will still chaff when using imported Python APIs (which don’t feel “swifty” for LOTS of reasons - e.g. method naming and design patterns), but my view is that this will provide incentive for these people to provide a proper Swift implementation for these libraries over time.</div><div class=""><br class=""></div><div class="">In short, your end game is a pervasively intertwined Swift/Python world (like ObjC and Swift are). My view is that Swift holds Python at arm's length, and wins over the hearts and minds of developers, leading to new Swift APIs designed for Swift.</div></div></div></div></blockquote><div><br class=""></div>That’s not the end game I’m looking at. I’d rather make it feasible to generate Swift wrappers for Python libraries that fit well into Swift’s existing machinery. The complexity goes outside the compiler, with small extralinguistic tie-ins to make it work well.</div><div><br class=""><blockquote type="cite" class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class=""><div class="">I’m not sure if you agree with that portrayal of your position, and if not, I’m sorry and will try to understand another way. However, if I’m close, then I have several concerns about that vision and don’t believe the end-game is achievable or better than what I’m proposing. Consider:</div><div class=""><br class=""></div><div class="">- I don’t think there will be a lot of success getting people who *actually love* Python to use Swift, unless there is already an extrinsic reason for them to use it.</div><div class="">- Type hints are not widely used in Python, and there are believable reasons that they won’t ever be. Because they are specifically poorly suited for libraries, their use seems to be in “user’s own code” - but my proposal solves this already! See below for examples.</div><div class="">- Even if type hints were widely adopted, it would require massive extensions to them and to Python to make them be "good enough” to provide value for the things you’re envisioning. Analogs to instancetype, objc generics, lots of the C macros, and many of the other things we’ve added to ObjC would have to be added.</div><div class="">- The Python community has no reason to do this work for the Swift community, or accept changes to Python that are required to make this actually great.</div><div class="">- Beyond the core type system, the Python and Objective-C languages work extremely differently in other ways (e.g. lack of umbrella headers making AnyObject-style dispatch questionable).</div><div class=""><div class="">- Even with those annotations and all the work, the APIs we’d end up with are not going to be good Swift APIs. I don’t think that “renamification” and "IUO audits" would ever actually happen in practice, for example.</div><div class="">- I believe the engineering effort required to implement this vision is so massive (including the changes to Swift, Python, and Python libraries) that it simply will never actually happen.</div></div></div></div></div></blockquote><div><br class=""></div><div>They don’t have to be good, strictly-typed APIs to be useful. They should have method/property names, have documentation, and be associated with named classes to work well with existing tooling.</div><br class=""><blockquote type="cite" class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class=""><div class="">More concerning to me is that your design of using the existing AnyObject type presents a really concerning technical problem, scalability: It is not simply a Swift/ObjC/Python world, Javascript is also super important. There are also a number of other less-widely used dynamic languages that are interesting. Your design leads to them all being mashed together into a common runtime system. I cannot imagine how this would end up being a good thing for system complexity, and would lead to each of them being jeopardized in different ways. As I have mentioned before numerous times, it also opens the door for enormous Swift compiler complexity, as each of the object models need to be supported in the compiler (so you can subclass each languages’ types), and many other complexities.</div></div></div></div></blockquote><div><br class=""></div><div>The AnyObject model of dispatching to any method within a given dynamic space (whether it’s Objective-C or Python or something else) fits well into Swift. You’re absolutely right that we can’t make Python objects work with AnyObject.</div><br class=""><blockquote type="cite" class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class=""><div class="">If you are serious about improving the tooling situation for people using Python APIs in Swift, the most natural way to do so is to follow the approach that the MyPy community (they are the ones who have thought about this the most) is using: provide progressive typing extensions, use data flow analysis to propagate around types, and enhance the tooling to have support this. I’m skeptical that this will ever be worthwhile, but a sketch could look like this:</div></div></div></div></blockquote><div><br class=""></div><div>I doubt this would be worthwhile. If we wanted to do something along these lines, we’d want to seriously investigate something akin to gradual typing.</div></div><div><blockquote type="cite" class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class=""><div class=""><br class=""></div><div class="">Similarly, there is no reason to do anything to make the second and third examples work nicely in Swift: you’d just define a Swift class, and a Swift function.</div><div class=""><br class=""></div><div class="">MyPy:</div><div class=""><br class=""></div><div class=""><pre class="ex" style="margin-top: 1.3em; margin-bottom: 1.3em; color: rgb(60, 60, 60); padding: 0.9em; background-color: rgb(238, 238, 238); border: 1px solid rgb(221, 221, 221); font-size: 13.0321px; font-variant-ligatures: normal; orphans: 2; widows: 2; font-family: 'Bitstream Vera Sans Mono', 'DejaVu Sans Mono', Monaco, Courier, monospace !important;"><span class="kw" style="color: rgb(92, 92, 160);">class</span> BankAccount:
<span class="kw" style="color: rgb(92, 92, 160);">def</span> __init__<span class="hili" style="background-color: rgb(223, 223, 223);">(self, initial_balance: <span class="pr" style="color: rgb(92, 92, 160);">int</span> = 0) -> None</span>:
self.balance = initial_balance
<span class="kw" style="color: rgb(92, 92, 160);">def</span> deposit<span class="hili" style="background-color: rgb(223, 223, 223);">(self, amount: <span class="pr" style="color: rgb(92, 92, 160);">int</span>) -> None</span>:
self.balance += amount
<span class="kw" style="color: rgb(92, 92, 160);">def</span> withdraw<span class="hili" style="background-color: rgb(223, 223, 223);">(self, amount: <span class="pr" style="color: rgb(92, 92, 160);">int</span>) -> None</span>:
self.balance -= amount
<span class="kw" style="color: rgb(92, 92, 160);">def</span> overdrawn<span class="hili" style="background-color: rgb(223, 223, 223);">(self) -> <span class="pr" style="color: rgb(92, 92, 160);">bool</span></span>:
<span class="kw" style="color: rgb(92, 92, 160);">return</span> self.balance < 0
my_account = BankAccount(15)
my_account.withdraw(5)
<span class="pr" style="color: rgb(92, 92, 160);">print</span>(my_account.balance)</pre><div class="">Swift:</div><div class=""><br class=""></div><div class="">class BankAccount { </div><div class=""> var balance : Int</div><div class=""> init(initialBalance: Int) { … }</div></div><div class=""> func deposit(….</div><div class="">}</div><div class=""><br class=""></div><div class="">Even the proposals that I’m making - simple and isolated though they are - are enough to provide a major quality of life improvement for people writing large amounts of code against Python APIs. </div></div></div></div></blockquote><div><br class=""></div><div>They’re an improvement to the resulting code (i.e., the Swift code looks comparable to the Python code), but it’s not a quality-of-life improvement for the process of writing the code (or understanding it once it’s written), because you don’t get any of the tooling benefits you’re used to having as a Swift programmer: no code completion, no “”quick help”, no go-to-definition.</div><br class=""><blockquote type="cite" class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class=""><div class=""> Beyond that, they are small extensions with low complexity, scale to supporting many different dynamic languages over time, require a level of engineering effort that is plausible to be built, and do not require some sort of "executive buy in” from the Python community.</div></div></div></div></blockquote><br class=""></div><div>Here’s an alternative proposal that provides a better development experience: write a wrapper generator, in Python, that maps a Python module’s interfaces to Swift, and provide small tie-ins to the compiler to make it fit well. Take the example from the DynamicMemberLookup proposal:</div><div><br class=""></div><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div><div><font face="Menlo" class="">class Dog:</font></div></div><div><div><font face="Menlo" class=""> """Man's best friend"""</font></div></div><div><div><font face="Menlo" class=""> def __init__(self, name):</font></div></div><div><div><font face="Menlo" class=""> """Summon a new dog"""</font></div></div><div><div><font face="Menlo" class=""> self.name = name</font></div></div><div><div><font face="Menlo" class=""> self.tricks = [] # creates a new empty list for each dog</font></div></div><div><div><font face="Menlo" class=""><br class=""></font></div></div><div><div><font face="Menlo" class=""> def add_trick(self, trick):</font></div></div><div><div><font face="Menlo" class=""> """Teach the old dog a new trick"""</font></div></div><div><div><font face="Menlo" class=""> self.tricks.append(trick)</font></div></div></blockquote><div><div class=""><br class=""></div></div><div><br class=""></div><div>The output of the wrapper generator is:</div><div><br class=""></div><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div><font face="Menlo" class="">/// </font><span style="font-family: Menlo;" class="">Man's best friend</span></div><div><font face="Menlo" class="">@_foreign(python)</font></div><div><font face="Menlo" class="">class Dog : PythonObject {</font></div><div><font face="Menlo" class=""> /// </font><span style="font-family: Menlo;" class="">Summon a new dog</span></div><div><font face="Menlo" class=""> @_foreign(python, “__init__”)</font></div><div><font face="Menlo" class=""> init(_ name: PythonArgument) { /* code to dynamically call __init__ on the Dog class object*/ }</font></div><div><font face="Menlo" class=""><br class=""></font></div><div><font face="Menlo" class=""> /// </font><span style="font-family: Menlo;" class="">Teach the old dog a new trick</span></div><div><font face="Menlo" class=""> @_foreign(python, “add_trick”)</font></div><div><font face="Menlo" class=""> func add_trick(_ trick: PythonArgument) -> PythonObject? { /* code to dynamically call add_trick */ }</font></div><div><font face="Menlo" class="">}</font></div></blockquote><div><br class=""></div><div>Several pieces above require explanation, but first the code to use this from Swift:</div><div><br class=""></div><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div><div><font face="Menlo" class="">import DogModule</font></div></div><div><div><font face="Menlo" class="">let dog = Dog("Brianna")</font></div></div><div><div><font face="Menlo" class="">dog.add_trick("Roll over")</font></div></div><div><div><font face="Menlo" class="">log do2 = Dog("Kaylee").add_trick("snore")</font></div></div></blockquote><div><div class=""><br class=""></div></div><div>Now, because we have declarations in the Swift module, the tools helped us write this code: I got code completion for the “Dog” type, its initializers, its “add_trick” method. The Python docstrings were carried through into documentation comments that show up in my IDE. I can inspect the Swift interface created by the wrapper generator to see what it looks like; my class browser can show me the Python class hierarchy.</div><div><br class=""></div><div>Let’s break down the pieces we need to get here:</div><div><br class=""></div><div>PythonObject: this would be a root class, defined by the Python interoperability library:</div><div><br class=""></div><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div><font face="Menlo" class="">@_foreign(python)</font></div><div><font face="Menlo" class="">class PythonObject {</font></div><div><font face="Menlo" class=""> static func retain(_ object: UnsafePointer<Void>) { Py_INCREF(object) }</font></div><div><font face="Menlo" class=""> static func release(_ object: UnsafePointer<Void>) {Py_DECREF(object) }</font></div><div><span style="font-family: Menlo;" class="">}</span></div></blockquote><div><br class=""></div><div>PythonObject is a foreign class for the Python language (hence the @_foreign). So while you get subtyping as normal (Dog is a subtype of PythonObject), they aren’t a subtype of AnyObject because it’s not a native object model. Nor could you mix a foreign class marked as @_foreign(python) with one marked as @_foreign(ruby), but they’re different object models.</div><div><br class=""></div><div>The retain/release operations would be called by the compiler in the obvious places. It shouldn’t be too difficult to build an appropriate value witness table to make this work.</div><div><br class=""></div><div>PythonArgument: This is a protocol to which any type can conform to map it into Python, and IIRC you already have this in your design in some form:</div><div><br class=""></div><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div><font face="Menlo" class="">protocol PythonArgument {</font></div><div><font face="Menlo" class=""> func toPythonObject() -> PythonObject</font></div><div><font face="Menlo" class="">}</font></div></blockquote><div><font face="Menlo" class=""><br class=""></font></div><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div><font face="Menlo" class="">extension PythonObject: PythonArgument {</font></div><div><font face="Menlo" class=""> func toPythonObject() -> PythonObject { return self }</font></div><div><font face="Menlo" class="">}</font></div><div><font face="Menlo" class=""><br class=""></font></div><div><font face="Menlo" class="">extension Int: PythonArgument {</font></div><div><font face="Menlo" class=""> func toPythonObject() -> PythonObject { … }</font></div><div><font face="Menlo" class="">}</font></div></blockquote><div><br class=""></div><div>@_foreign on methods/properties/etc.: This is there to enable AnyObject-style dispatch. Consider the add_trick example:</div><div><br class=""></div><div><div><font face="Menlo" class=""> @_foreign(python, “add_trick”)</font></div><div><font face="Menlo" class=""> func add_trick(_ trick: PythonArgument) -> PythonObject { /* code to dynamically call add_trick */ }</font></div><div class=""><font face="Menlo" class=""><br class=""></font></div></div><div>The two arguments to @_foreign are the foreign-class-model indicator (python): one gets AnyObject lookup from objects whose type is in that same foreign class space (e.g., PythonObject is also in the “python” foreign class model), and the second string (here, “add_trick”) is the uniquing string that indicates what name would be passed through to the underling runtime. This becomes important when you have overlap at different parts in the class hierarchy, e.g.,</div><div><br class=""></div><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div><font face="Menlo" class="">@_foreign(python)</font></div><div><font face="Menlo" class="">class MagicShow: PythonObject {</font></div><div><font face="Menlo" class=""> @_foreign(python, “add_trick”)</font></div><div><font face="Menlo" class=""> func add_trick(_ trick: PythonArgument) -> PythonObject? { … }</font></div><div><font face="Menlo" class="">}</font></div></blockquote><div><br class=""></div><div>And then we have something like:</div><div><br class=""></div><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div><font face="Menlo" class="">func foo(po: PythonObject) {</font></div><div><font face="Menlo" class=""> po.add_trick?(“something”) // finds both Dog.add_trick(_:) and MagicShow.add_trick(_:), but since they’re both uniqued by “add_trick” and have the same signature, we can pick one arbitrarily</font></div><div><font face="Menlo" class="">}</font></div></blockquote><div><br class=""></div><div>This is a generalization of how AnyObject dispatch already works in the type checker. For Objective-C, AnyObject is the root for the “objc” class model and the Objective-C selector (with + or - prepended) is used as the uniquing key; you can see the code at <a href="https://github.com/apple/swift/blob/master/lib/Sema/ConstraintSystem.cpp#L265-L273" class="">https://github.com/apple/swift/blob/master/lib/Sema/ConstraintSystem.cpp#L265-L273</a>.</div><div><br class=""></div><div>Factory initializers: unfortunately, to actually make initializers work cleanly, we’ll need something like factory initializers in the language. We should have those anyway, but until we get them one can fake it with top-level functions:</div><div><br class=""></div><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div><font face="Menlo" class="">@_foreign(python, “__init__”)</font></div><div><font face="Menlo" class="">func Dog(_ name: PythonArgument) -> Dog { /* … */ }</font></div></blockquote><div><br class=""></div><div>“Truly” dynamic calls: Python code does dynamically create new properties, methods, classes, etc. Those won’t get reflected into the generated Swift interface (because the wrapper generator won’t see them when it just imports the module to inspect it). So, there needs to be some string-based interface for such dynamic calls, which would be provided by the interoperability library in whatever syntax makes it clear what’s happening:</div><div><br class=""></div><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div><font face="Menlo" class="">extension PythonObject {</font></div><div><font face="Menlo" class=""> subscript(property name: String) -> PythonObject? { get { /* … */ } set { /* … */ }</font></div><div><font face="Menlo" class=""> func call(_ methodName: String, _ args: PythonArgument…, keywordArgs: [String: PythonArgument] = [:]) -> PythonObject? { … }</font></div><div><font face="Menlo" class="">}</font></div></blockquote><div><br class=""></div><div>This enables the fully-dynamic behavior, but makes it clear when we’ve conjured up the name as a string.</div><div><br class=""></div><div>On-demand wrapper generation: you’ve mentioned earlier in this thread that it’s important to support the playgrounds use case, where one can import whatever Python modules are available to play with them. I *personally* don’t think running a one-off wrapper script once per module is all that onerous, but if it is, we could extend the Swift compiler with a very light plugin mechanism where the plugin mechanism has two operations:</div><div><br class=""></div><div>1) Enumerate the names of modules that it can translate (e.g., find all of the Python modules in the Python path and current working directory), and</div><div>2) Given a module name, produce the Swift wrapper code for the Swift compiler to then compile (and cache)</div><div><br class=""></div><div>Overall, it’s probably more work than DynamicMemberLookup, but it provides a more friendly development experience because the entities in the Python module show throw in Swift—they’re documented, inspectable, code-completable, etc. We get the class hierarchy reflected into Swift so we get proper subtyping behavior. And writing these wrapper generators is something that is easily hackable by anyone who knows the languages involved, it’s easy to test and improve over time.</div><div><br class=""></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>- Doug</div><div><br class=""></div><br class=""></body></html>