[swift-evolution] Proposal: Introduce User-defined "Dynamic Member Lookup" Types

[Chris Lattner]

HI Doug,

Your approach can definitely work, but before responding with a tradeoff analysis, I’d like to make sure that I understand what you’re suggesting specifically.

> On Dec 1, 2017, at 10:39 AM, Douglas Gregor <dgregor at apple.com> wrote:
>>  It also doesn’t provide the great tooling experience that you’re seeking, given that code completion would show everything in the Python universe, which is not helpful.
> 
> It’s better for code completion to provide too much than to provide nothing at all. If code completion provides too much, typing a small number of characters will reduce the completion set down to something manageable quite fast.

Yes, I understand your goal.

> DynamicMemberLookup doesn’t gave any of those, because “add_trick” is just a string that looks like an identifier. Only the Python runtime can resolve.

Agreed.

>> A preprocessing step prevents users from playfully importing random Python modules into the Swift repl and playgrounds.
> 
> *At worst*, you invoke the tool from the command line to build the Swift module that corresponds to a given Python module.
> 
> 	py2swift <pythonmodulename>
> 
> We could absolutely introduce tooling hooks to make the compiler initiate that step.

This would have to be integrated into the swift compiler to provide a usable experience IMO, because otherwise it won’t work with Playgrounds.

>>  It also seems worse for implementors (who will get a stream of new bugs about compiler scalability).
> 
> To my knowledge, AnyObject lookup has not been a serious source of performance problems for the compiler. The global lookup table it requires was annoying to implement, but it’s just a symbol table.

Right, AnyObject lookup itself is not a problem, because significant engineering effort has been put in place to build a module cache that can be efficiently queried.  I thought you were suggesting that we do a preprocessing step to smoosh together all of the “interesting” python APIs into a single huge extension on PyVal which would then be literally parsed and that overload resolution would resolve against.  This approach would perform very differently than AnyObject lookup does for ObjC.

>>> Python plugins for IDEs (e.g., for Atom) provide code completion, goto definition, and other related features. None of the Swift tooling will work if Swift’s interoperability with Python is entirely based on DynamicMemberLookupProtocol.
>> 
>> I don’t understand your rationale here.  I think you agree that we need to support the fully dynamic case (which is what I’m proposing).  That said, this step does not preclude introducing importer magic (either through compiler hackery or a theoretical "type providers” style of feature).  Doing so would provide the functionality you’re describing.
> 
> I don’t agree that we need language support for the fully-dynamic case. There has to be a way to handle the fully-dynamic case, but it can be string-based APIs vended by an Python interoperability library.
> 
> I believe that the majority of property and method accesses in Python programs will be resolved to a definition in the current model or an imported module, i.e., a definition that is visible to the compiler at compile-time. There are exceptional cases involving programmatically creating properties from JSON, a database, etc., but while prominent I suspect they account for a relatively small fraction of use sites. Do you agree with this statement?

Without pervasive type annotations I’m not sure what you mean.  Can you elaborate more?  Even if foo is typed, “foo.bar.baz()” won’t be in general because Python does not have typed property declarations.

>> Finally, just MHO, but I don’t expect a lot of “mix and match" Python/Swift apps to exist (where the developer owns both the Python and the Swift code), which is one case where type annotations are super awesome in ObjC.  IMO, the most important use-case is a Swift program that uses some Python APIs.
> 
> Let’s consider Python type annotations to be useless for our purposes. None of my argument hinges on it.

Good.  I’ll ignore them for the moment.

>> 
>> I didn’t realize that you were thinking we would literally use AnyObject itself.  I haven’t thought fully through it, but I think this will provide several problems:
> 
> You are correct that literally using AnyObject has these issues. There appears to be a lot of confusion about what the AnyObject model *is*, so let’s sort through that. I think it is reasonable to take the AnyObject model and replicate it for PythonObject.

Ok, good, using literally AnyObject is super problematic in my opinion.

So your implementation approach suggestion is to:

1) Define a new builtin type PythonObject.  Is this a base class, a new nominal type, or something else?  I assume a base class.
2) It supports AnyObject-style lookup, so code completion works.
3) The database of methods is built from the Python modules.
4) PythonObject is callable, dynamic member lookupable, subscriptable, and has implicit conversions to make it work well.
5) Python derived classes are modeled as Swift classes that derive from PythonObject.

Am I missing something?  If this is the approach you’re suggesting, I’ll explain the pros and cons vs my approach.

>> As I mention above, I expect this to expose significant scalability problems in the Swift compiler and it also defeats REPL/Playgrounds.  Being able to use the Swift REPL is really important for Python programmers.
> 
> 
> I’ve addressed both of these issues above. The approach I am proposing requires no language or compiler support, works with existing Swift tooling, fits with an existing notion in the language (AnyObject), and can be implemented via a Python script.

I’m very confused, are you suggesting a preprocessor, or something actually integrated with the compiler?  The goal here is to provide something with acceptable usability that encourages exploration and playing around.

I’d also appreciate it if you could respond to my other email with your thoughts:
https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20171127/041868.html

-Chris