<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Dec 1, 2017, at 1:30 AM, Chris Lattner <<a href="mailto:clattner@nondot.org" class="">clattner@nondot.org</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><meta http-equiv="Content-Type" content="text/html charset=utf-8" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class=""><blockquote type="cite" class=""><div class="">On Dec 1, 2017, at 12:26 AM, Douglas Gregor <<a href="mailto:dgregor@apple.com" class="">dgregor@apple.com</a>> wrote:</div><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class=""><blockquote type="cite" class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class=""><blockquote type="cite" class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class=""><b class="">Philosophy</b></div></div></div></blockquote></div></div></div></blockquote></div></div></div></blockquote><br class=""><blockquote type="cite" class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class=""><blockquote type="cite" class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class=""><div class="">More problematically for your argument: your preferred approach requires the introduction of (something like) DynamicMemberLookupProtocol or something like AnyObject-For-Python, so your proposal would be additive on top of solving the core problem I’m trying to solve. It isn’t an alternative approach at all.</div></div></div></div></blockquote><div class=""><br class=""></div><div class="">I wouldn’t say that’s my preferred approach. My preferred approach involves taking the method/property/etc. declarations that already exist in Python and mapping them into corresponding Swift declarations so we have something to find with name lookup. One could put all of these declarations on some PyVal struct or PythonObject and there would be no need for AnyObject-for-Python or DynamicMemberLookupProtocol.</div></div></div></div></blockquote><div class=""><br class=""></div><div class="">You’re suggesting that the transitive closure of all Python methods and properties be preprocessed into a single gigantic Swift PyVal type? I guess something like that could be done.</div></div></div></div></blockquote><div><br class=""></div><div>That’s effectively what Swift is already doing with AnyObject.</div><br class=""><blockquote type="cite" class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class=""><div class="">I would be concerned because there are many N^2 or worse algorithms in the Swift compiler would probably explode. </div></div></div></div></blockquote><div><br class=""></div><div>As noted above, we already do this for AnyObject with every @objc entity everywhere. The # of overloads for a given name is usually not so high that it’s a problem for the compiler.</div><br class=""><blockquote type="cite" class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class=""><div class=""> It also doesn’t provide the great tooling experience that you’re seeking, given that code completion would show everything in the Python universe, which is not helpful.</div></div></div></div></blockquote><div><br class=""></div><div>It’s better for code completion to provide too much than to provide nothing at all. If code completion provides too much, typing a small number of characters will reduce the completion set down to something manageable quite fast.</div><br class=""><blockquote type="cite" class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class=""><div class="">Further, it doesn’t provide a *better* experience than what I’m suggesting, it seems strictly worse. </div></div></div></div></blockquote><div><br class=""></div><div>From my prior message, having the declarations of all Python methods and properties available on PyVal makes Swift tooling work:</div><div><br class=""></div><div><div class="">For the same working Swift code</div></div><div><br class=""></div><div> dog.add_trick(rollOver)</div><div><br class=""></div><div>that calls into Python, the solution I’m proposing:</div><div><br class=""></div><div>* Gave you the add_trick(<#trick#>) code completion with the number of parameters and their names</div><div>* Supports goto definition to jump to the wrapper method on PyVal</div><div>* Supports “quick help” to show the declaration of that wrapper method on PyVal, showing the documentation (docstring) and in which Python class it was declared.</div><div>* Supports indexing functionality so we can find the likely uses of “add_trick"</div><div><br class=""></div><div>DynamicMemberLookup doesn’t gave any of those, because “add_trick” is just a string that looks like an identifier. Only the Python runtime can resolve.</div><br class=""><blockquote type="cite" class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class=""><div class="">A preprocessing step prevents users from playfully importing random Python modules into the Swift repl and playgrounds. </div></div></div></div></blockquote><div><br class=""></div><div>*At worst*, you invoke the tool from the command line to build the Swift module that corresponds to a given Python module.</div><div><br class=""></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>py2swift <pythonmodulename></div><div><br class=""></div><div>We could absolutely introduce tooling hooks to make the compiler initiate that step.</div><br class=""><blockquote type="cite" class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class=""><div class=""> It also seems worse for implementors (who will get a stream of new bugs about compiler scalability).</div></div></div></div></blockquote><div><br class=""></div><div>To my knowledge, AnyObject lookup has not been a serious source of performance problems for the compiler. The global lookup table it requires was annoying to implement, but it’s just a symbol table.</div><br class=""><blockquote type="cite" class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class=""><div class=""><br class=""></div><blockquote type="cite" class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class=""><blockquote type="cite" class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class=""><blockquote type="cite" class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class=""> Whenever we discuss adding more dynamic features to Swift, there’s a strong focus on maintaining that strong static type system.</div></div></div></blockquote><div class=""><br class=""></div><div class="">Which this does, by providing full type safety - unlike AnyObject lookup.</div></div></div></div></blockquote><div class=""><br class=""></div><div class="">You get dynamic safety because it goes into the Python interpreter; fair enough. You get no help from your tools to form a correct invocation of any method provided by Python.</div></div></div></div></blockquote><div class=""><br class=""></div>Sure, that’s status quo for Python APIs.</div></div></div></blockquote><div><br class=""></div>That’s not the status quo for Python IDEs. They will give you the basic shape of method calls.</div><div><br class=""><blockquote type="cite" class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class=""><br class=""><blockquote type="cite" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class=""><blockquote type="cite" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class=""><blockquote type="cite" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class=""><b class="">Tooling</b></div><div class="">The Python interoperability enabled by this proposal *does* look fairly nice when you look at a small, correctly-written example. However, absolutely none of the tooling assistance we rely on when writing such code will work for Python interoperability. Examples:</div><div class=""><br class=""></div><div class="">* As noted earlier, if you typo’d a name of a Python entity or passed the wrong number of arguments to it, the compiler will not tell you: it’ll be a runtime failure in the Python interpreter. I guess that’s what you’d get if you were writing the code in Python, but Swift is supposed to be *better* than Python if we’re to convince a community to use Swift instead.</div><div class="">* Code completion won’t work, because Swift has no visibility into declarations written in Python</div><div class="">* Indexing/jump-to-definition/lookup documentation/generated interface won’t ever work. None of the IDE features supported by SourceKit will work, which will be a significant regression for users coming from a Python-capable IDE.</div></div></blockquote><div class=""><br class=""></div><div class="">Yes, welcome to the realities of modern Python development!</div></div></div></blockquote><div class=""><br class=""></div><div class="">Python plugins for IDEs (e.g., for Atom) provide code completion, goto definition, and other related features. None of the Swift tooling will work if Swift’s interoperability with Python is entirely based on DynamicMemberLookupProtocol.</div></div></div></blockquote><div class=""><br class=""></div><div class="">I don’t understand your rationale here. I think you agree that we need to support the fully dynamic case (which is what I’m proposing). That said, this step does not preclude introducing importer magic (either through compiler hackery or a theoretical "type providers” style of feature). Doing so would provide the functionality you’re describing.</div></div></div></div></blockquote><div><br class=""></div><div>I don’t agree that we need language support for the fully-dynamic case. There has to be a way to handle the fully-dynamic case, but it can be string-based APIs vended by an Python interoperability library.</div><div><br class=""></div><div>I believe that the majority of property and method accesses in Python programs will be resolved to a definition in the current model or an imported module, i.e., a definition that is visible to the compiler at compile-time. There are exceptional cases involving programmatically creating properties from JSON, a database, etc., but while prominent I suspect they account for a relatively small fraction of use sites. Do you agree with this statement?</div><div><br class=""></div><blockquote type="cite" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class=""><blockquote type="cite" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class=""><blockquote type="cite" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class=""><blockquote type="cite" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class=""><b class="">How Should Python Interoperability Work?</b></div><div class="">Going back to the central motivator for this proposal, I think that providing something akin to the Clang Importer provides the best interoperability experience: it would turn Python declarations into *real* Swift declarations, so that we get the various tooling benefits of having a strong statically-typed language. </div></div></blockquote><div class=""><br class=""></div><div class="">This could be an theoretically interesting refinement to this proposal but I’m personally very skeptical that this is every going to happen. I’ve put the rationale into the alternatives section of the proposal. I don’t explain it in the proposal in this way directly, but I believe it is far more likely for a Pythonista transplant into Swift to rewrite their code in Swift than it is to use Python type annotations.</div></div></div></blockquote><div class=""><br class=""></div><div class="">I assume that this belief is based on type annotations lack of traction in the Python community thus far?</div></div></div></blockquote><div class=""><br class=""></div><div class="">There are many parts to this, which have to do with the ObjC<->Swift situation being very different than the Python<->Swift situation:</div><div class=""><br class=""></div><div class="">1) The annotations don’t have significant traction in the Python community. </div><div class="">2) The Python annotations are not as powerful as ObjC generics are, and thus lack important expressive capability.</div><div class="">3) Many Python APIs are wrappers for C APIs. “Swiftizing” a Python API in this case means writing a new Swift wrapper for the API, not adding type annotations.</div><div class=""><div class="">4) The Python community doesn’t care about Swift, and are not motivated to do things to make Swift succeed.</div><div class="">5) There is no “clang equivalent” for Python (that I’m aware of) which close enough to the way Clang does for us to directly use. The owners of the existing Python compiler/interpreter implementations are not going to be strongly motivated to change their stuff for us.</div></div><div class=""><br class=""></div><div class="">Finally, just MHO, but I don’t expect a lot of “mix and match" Python/Swift apps to exist (where the developer owns both the Python and the Swift code), which is one case where type annotations are super awesome in ObjC. IMO, the most important use-case is a Swift program that uses some Python APIs.</div></div></div></blockquote><div><br class=""></div><div>Let’s consider Python type annotations to be useless for our purposes. None of my argument hinges on it.</div><div><br class=""></div><br class=""><blockquote type="cite" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class=""><blockquote type="cite" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class=""><blockquote type="cite" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class=""><blockquote type="cite" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class="">Sure, the argument types will all by PyObject or PyVal, </div></div></blockquote><div class=""><br class=""></div><div class="">That’s the root of the problem. Python has the “fully dynamic” equivalent of “id” in Objective-C, so we need to represent that somehow. Even if we followed the implementation approach of the Clang importer, we would need some way to represent this dynamic case. That type needs features like DynamicMemberLookup or AnyObject. In my opinion, the DynamicMemberLookup approach is better in every way than AnyObject is. </div></div></div></blockquote><div class=""><br class=""></div><div class="">The AnyObject approach has the advantage of knowing the set of declared, reachable APIs:</div><div class=""><br class=""></div><div class="">* Code completion shows all of the APIs that are possible to use via dynamic dispatch, with their signatures so can fill in the right # of arguments, see the names of the parameters, see documentation, etc.</div><div class="">* Indexing/refactoring/goto definition all point you to the declarations that could be the targets of dynamic dispatch</div><div class=""><br class=""></div><div class="">The DynamicMemberLookup approach is better for cases where you don’t have a declaration of the member you want to access. I suspect that’s not the common case.</div></div></div></blockquote><div class=""><br class=""></div><div class="">Your points are valid, but the advantages for Objective-C don’t obviously translate to Swift. Note that ObjC (due to its heritage) has very long method names that are perhaps arguably designed to not conflict with each other often. Python doesn’t have this heritage, and it has much shorter names, which means that we’ll get a lot more conflicts and a lot less “safety" out of this.</div></div></div></blockquote><div><br class=""></div><div>AnyObject lookup resolves conflicts by collapsing similar overloads, and that’s okay: we don’t really care which class contained the declaration we found; we just care about the shape of the declaration. The lack of this magic will be a problem for the wrapper approach I’m describing, but that’s fixable.</div><br class=""><blockquote type="cite" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class=""><div class="">AnyObject lookup also depends on a strange set of scoping heuristics that was designed to be similar to Clang’s “header import” scope. It isn’t clear that this approach will work in Python, given that it doesn’t have an analogue of umbrella headers that import things that cross frameworks.</div></div></div></blockquote><div><br class=""></div><div>Not sure what you think the scoping heuristics are. AnyObject lookup is fairly simple to define: it finds all @objc members of every class and protocol in your current module and any modules you imported.</div><br class=""><blockquote type="cite" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class=""><blockquote type="cite" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class=""><blockquote type="cite" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class=""><div class="">Which approach do you think is the best way to handle the untyped “actually dynamic” case? </div></div></div></blockquote><div class=""><br class=""></div><div class="">AnyObject already exists in the language, and it fits the untyped “actually dynamic” case well. It does require having a declaration for the thing you want to reference, which I consider to be important: we can code-complete those declarations, goto-definition to see those declarations, index/refactor/look up documentation based on those declarations.</div><div class=""><br class=""></div><div class="">I’d be more inclined to push for the ImplicitlyUnwrappedOptional -> Optional change if we did something to make AnyObject more prominent in Swift.</div></div></div></blockquote><div class=""><br class=""></div>I didn’t realize that you were thinking we would literally use AnyObject itself. I haven’t thought fully through it, but I think this will provide several problems:</div><div class=""><br class=""></div><div class="">1) You’re mushing all of the ObjC and Python world’s together, making the ObjC interop worse just because you’re doing some Python stuff too.</div><div class="">2) You’re introducing ambiguity: does “ao = [1,2,3]” create an NSArray or a Python array? How do string literals work? (The answer is obvious, Python loses). Maybe there is some really complicated bridging solution to these problems, but that causes its own massive complexity spiral.</div><div class="">3) You can’t realistically overload the Python operator set on AnyObject, which means you get a worse python experience.</div><div class="">4) AnyObject magic is currently limited to Apple platforms. This would bring its problems to other platforms like Linux.</div><div class=""><br class=""></div><div class="">There are probably other issues, but I haven’t thought through it.</div></div></blockquote><div><br class=""></div><div>You are correct that literally using AnyObject has these issues. There appears to be a lot of confusion about what the AnyObject model *is*, so let’s sort through that. I think it is reasonable to take the AnyObject model and replicate it for PythonObject.</div><br class=""><blockquote type="cite" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class=""><blockquote type="cite" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class=""><blockquote type="cite" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class=""><blockquote type="cite" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class="">In truth, you don’t even need the compiler to be involved. The dynamic “subscript” operation could be implemented in a Swift library, and one could write a Python program to process a Python module and emit Swift wrappers that call into that subscript operation. You’ll get all of the tooling benefits with no compiler changes, and can tweak the wrapper generation however much you want, using typing annotations or other Python-specific information to create better wrappers over time.</div></div></blockquote><br class=""></div><div class="">I’d love for you to sketch out how any of this works with an acceptable user experience, because I don't see anything concrete here. </div></div></blockquote><div class=""><br class=""></div><div class="">We don’t need the basic dynamic case in the language to do this experiment. Take the PyVal struct from the proposal. Now, write a Python script that loads some module Foo and uses Python’s <a href="https://docs.python.org/3/library/inspect.html" class="">inspect</a> module to go find the classes, methods, etc., and pretty-print Swift code that uses PyVal. So this:</div><div class=""><br class=""></div><div class=""><span class="Apple-tab-span" style="white-space: pre;"> </span>def add_trick(self, trick):</div><div class=""><br class=""></div><div class=""><div class="">turns into</div><div class=""><br class=""></div><div class=""><span class="Apple-tab-span" style="white-space: pre;"> </span>extension PyVal {</div><div class=""><span class="Apple-tab-span" style="white-space: pre;"> </span> func add_trick(_ trick: PyVal) -> PyVal {</div><div class=""><span class="Apple-tab-span" style="white-space: pre;"> </span> /* do the magic to call into Python */</div><div class=""><span class="Apple-tab-span" style="white-space: pre;"> </span> }</div><div class=""><span class="Apple-tab-span" style="white-space: pre;"> </span>}</div><div class=""><br class=""></div><div class="">Using the inspect module, you can extract parameter names, default arguments, docstrings, and more to reflect the existing Python API as Swift API, packed into a bridging module.</div><div class=""><br class=""></div><div class="">Note that we have a “flat” namespace of all Python methods on PyVal, which is basically what you get with AnyObject today. Swift tooling will provide code completion for member accesses into PyVal. Goto definition will jump to the pretty-printed declarations, which could have the docstrings formatted in comments and would show up in QuickHelp. The types are weak (everything is PyVal), but that’s what we expect from importing a dynamically-typed language.</div></div></div></div></blockquote><div class=""><br class=""></div><div class="">As I mention above, I expect this to expose significant scalability problems in the Swift compiler and it also defeats REPL/Playgrounds. Being able to use the Swift REPL is really important for Python programmers.</div></div></div></blockquote></div><div class=""><br class=""></div>I’ve addressed both of these issues above. The approach I am proposing requires no language or compiler support, works with existing Swift tooling, fits with an existing notion in the language (AnyObject), and can be implemented via a Python script.<br class=""><div class=""><br class=""></div><div class=""><span class="Apple-tab-span" style="white-space:pre"> </span>- Doug</div><div class=""><br class=""></div></body></html>