<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">On Dec 1, 2017, at 12:26 AM, Douglas Gregor <<a href="mailto:dgregor@apple.com" class="">dgregor@apple.com</a>> wrote:<div><blockquote type="cite" class=""><div class=""><meta http-equiv="Content-Type" content="text/html; charset=utf-8" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class=""><blockquote type="cite" class=""><div class="">On Nov 30, 2017, at 10:05 PM, Chris Lattner <<a href="mailto:clattner@nondot.org" class="">clattner@nondot.org</a>> wrote:</div><div class=""><meta http-equiv="Content-Type" content="text/html charset=utf-8" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">Hi Doug,</div><div class=""><br class=""></div><div class="">Thank you for the detailed email. I have been traveling today, so I haven’t had a chance to respond until now. I haven’t read the down-thread emails, so I apologize if any of this was already discussed:</div><div class=""><br class=""></div><div class=""><blockquote type="cite" class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class="">I think better interoperability with Python (and other OO languages in widespread use) is a good goal, and I agree that the implementation of the feature described is straight-forward and not terribly invasive in the compiler.</div></div></div></blockquote><div class=""><br class=""></div><div class="">Fantastic, I’m really pleased to hear that! I only care about solving the problem, so if we can find a good technical solution to the problems than I’ll be happy.</div><div class=""></div></div></div></div></blockquote></div></div></div></blockquote><div><br class=""></div><div>After thinking about your email a bit more, I think I might understand the disconnect we’re having. I think we have different versions in mind of what “success” looks like:</div><div><br class=""></div><div>I believe your view is that people start using Python APIs from their Swift code, but gradually add type annotations (either inline or in a sidecar database) to make those APIs progressively more Swifty. New features will be added to Python (its type system, compilers, databases for common APIs, etc) to improve this interoperability, along the lines of what we’ve done for Objective-C over the years. Fast forward several years, and large Python libraries would be nice to use from Swift - perhaps nicer than they are to use from Python itself. This view aligns with what happened with Objective-C <-> Swift interoperability.</div><div><br class=""></div><div>In contrast, I’m specifically interested in developers in certain large domains (e.g. data science and ML) which are “forced” to use Python because that is where all the libraries are. I have spoken to many of these sorts of people, and a large number of them really *dislike* using Python for all the obvious reasons (including the tooling issues you point out). My view of success is that we allow them to write all of *their code* in Swift, which will lead to a massive quality of life benefit for these frustrated people. You’re right that they will still chaff when using imported Python APIs (which don’t feel “swifty” for LOTS of reasons - e.g. method naming and design patterns), but my view is that this will provide incentive for these people to provide a proper Swift implementation for these libraries over time.</div><div><br class=""></div><div>In short, your end game is a pervasively intertwined Swift/Python world (like ObjC and Swift are). My view is that Swift holds Python at arm's length, and wins over the hearts and minds of developers, leading to new Swift APIs designed for Swift.</div><div><br class=""></div><div><br class=""></div><div>I’m not sure if you agree with that portrayal of your position, and if not, I’m sorry and will try to understand another way. However, if I’m close, then I have several concerns about that vision and don’t believe the end-game is achievable or better than what I’m proposing. Consider:</div><div><br class=""></div><div>- I don’t think there will be a lot of success getting people who *actually love* Python to use Swift, unless there is already an extrinsic reason for them to use it.</div><div>- Type hints are not widely used in Python, and there are believable reasons that they won’t ever be. Because they are specifically poorly suited for libraries, their use seems to be in “user’s own code” - but my proposal solves this already! See below for examples.</div><div>- Even if type hints were widely adopted, it would require massive extensions to them and to Python to make them be "good enough” to provide value for the things you’re envisioning. Analogs to instancetype, objc generics, lots of the C macros, and many of the other things we’ve added to ObjC would have to be added.</div><div>- The Python community has no reason to do this work for the Swift community, or accept changes to Python that are required to make this actually great.</div><div>- Beyond the core type system, the Python and Objective-C languages work extremely differently in other ways (e.g. lack of umbrella headers making AnyObject-style dispatch questionable).</div><div><div>- Even with those annotations and all the work, the APIs we’d end up with are not going to be good Swift APIs. I don’t think that “renamification” and "IUO audits" would ever actually happen in practice, for example.</div><div>- I believe the engineering effort required to implement this vision is so massive (including the changes to Swift, Python, and Python libraries) that it simply will never actually happen.</div><div><br class=""></div><div class=""><br class=""></div></div><div>More concerning to me is that your design of using the existing AnyObject type presents a really concerning technical problem, scalability: It is not simply a Swift/ObjC/Python world, Javascript is also super important. There are also a number of other less-widely used dynamic languages that are interesting. Your design leads to them all being mashed together into a common runtime system. I cannot imagine how this would end up being a good thing for system complexity, and would lead to each of them being jeopardized in different ways. As I have mentioned before numerous times, it also opens the door for enormous Swift compiler complexity, as each of the object models need to be supported in the compiler (so you can subclass each languages’ types), and many other complexities.</div><div><br class=""></div><div><br class=""></div><div>If you are serious about improving the tooling situation for people using Python APIs in Swift, the most natural way to do so is to follow the approach that the MyPy community (they are the ones who have thought about this the most) is using: provide progressive typing extensions, use data flow analysis to propagate around types, and enhance the tooling to have support this. I’m skeptical that this will ever be worthwhile, but a sketch could look like this:</div><div><br class=""></div><div><br class=""></div><div>Consider the first example from: <a href="http://mypy-lang.org/examples.html" class="">http://mypy-lang.org/examples.html</a></div><div><br class=""></div><div>With my proposals you could write the dictionary part very similar to the Python part (other pieces of the example, e.g. list comprehensions are uninteresting to me):</div><div><br class=""></div><div></div></div><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div><div>let d: PyVal = {:}</div></div><div><div>…</div></div><div><div><div class="">d[word] = d.get(word, 0) + 1</div></div></div></blockquote><div><div><br class="Apple-interchange-newline"></div><div>We can teach the compiler simple data flow analysis, to know that ‘d’ is a Python dictionary. We could then allow the user to go further, with some new syntax along the lines of:</div><div><br class=""></div><div><blockquote style="margin: 0px 0px 0px 40px; border: none; padding: 0px;" class=""><div><div>let d: @type(Dict<String, Int>) PyVal = {:} // Lots of other syntactic choices possible.</div></div><div><div>…</div></div><div><div>d[word] = d.get(word, 0) + 1</div></div></blockquote><div></div></div><div><br class=""></div><div><br class=""></div><div><br class=""></div><div>However, the really important thing to recognize about these examples it that the type annotations are really poor at representing “generic” code like common Python libraries. They are designed for “user code” - like that on the mypy web page - which use concrete types.</div><div><br class=""></div><div>As it turns out, my proposal provides a solution for this part of the problem, because we’re allowing user code to be written in Swift! A Swift programmer working with Python APIs would actually write this as:</div><div><br class=""></div><div><br class=""></div><div><blockquote style="margin: 0px 0px 0px 40px; border: none; padding: 0px;" class=""><div><div>let d: Dictionary<String,Int> = {:}</div></div><div><div>…</div></div><div><div>d[word, default: 0] += 1</div></div></blockquote><div></div></div><div><br class=""></div><div><br class=""></div><div>Similarly, there is no reason to do anything to make the second and third examples work nicely in Swift: you’d just define a Swift class, and a Swift function.</div><div><br class=""></div><div>MyPy:</div><div><br class=""></div><div><pre class="ex" style="margin-top: 1.3em; margin-bottom: 1.3em; color: rgb(60, 60, 60); padding: 0.9em; background-color: rgb(238, 238, 238); border: 1px solid rgb(221, 221, 221); font-size: 13.0321px; font-variant-ligatures: normal; orphans: 2; widows: 2; font-family: 'Bitstream Vera Sans Mono', 'DejaVu Sans Mono', Monaco, Courier, monospace !important;"><span class="kw" style="color: rgb(92, 92, 160);">class</span> BankAccount:
<span class="kw" style="color: rgb(92, 92, 160);">def</span> __init__<span class="hili" style="background-color: rgb(223, 223, 223);">(self, initial_balance: <span class="pr" style="color: rgb(92, 92, 160);">int</span> = 0) -> None</span>:
self.balance = initial_balance
<span class="kw" style="color: rgb(92, 92, 160);">def</span> deposit<span class="hili" style="background-color: rgb(223, 223, 223);">(self, amount: <span class="pr" style="color: rgb(92, 92, 160);">int</span>) -> None</span>:
self.balance += amount
<span class="kw" style="color: rgb(92, 92, 160);">def</span> withdraw<span class="hili" style="background-color: rgb(223, 223, 223);">(self, amount: <span class="pr" style="color: rgb(92, 92, 160);">int</span>) -> None</span>:
self.balance -= amount
<span class="kw" style="color: rgb(92, 92, 160);">def</span> overdrawn<span class="hili" style="background-color: rgb(223, 223, 223);">(self) -> <span class="pr" style="color: rgb(92, 92, 160);">bool</span></span>:
<span class="kw" style="color: rgb(92, 92, 160);">return</span> self.balance < 0
my_account = BankAccount(15)
my_account.withdraw(5)
<span class="pr" style="color: rgb(92, 92, 160);">print</span>(my_account.balance)</pre><div class="">Swift:</div><div class=""><br class=""></div><div class="">class BankAccount { </div><div class=""> var balance : Int</div><div class=""> init(initialBalance: Int) { … }</div></div><div> func deposit(….</div><div>}</div><div><br class=""></div><div>Even the proposals that I’m making - simple and isolated though they are - are enough to provide a major quality of life improvement for people writing large amounts of code against Python APIs. Beyond that, they are small extensions with low complexity, scale to supporting many different dynamic languages over time, require a level of engineering effort that is plausible to be built, and do not require some sort of "executive buy in” from the Python community.</div><div><br class=""></div><div>-Chris</div><div><br class=""></div><div><br class=""></div><div><br class=""></div><div><br class=""></div></div></body></html>