[swift-evolution] Proposal: Introduce User-defined "Dynamic Member Lookup" Types

[Benjamin G]

On Fri, Dec 1, 2017 at 11:03 AM, Benjamin G via swift-evolution <
swift-evolution at swift.org> wrote:

> I think the fear most of us , poor developers working with developers of
> various skills, is the potential for abuse. I've heard many times that it
> isn't a guiding principle for swift, so i'll just reformulate the concern
> in a more acceptable way :
> Could your proposal explain how the swift language will still keep
> encouraging developers to use static / compile-time type checking *when
> it's possible* ? (because i think we can all argue that a compile-time
> checks is better than a runtime one). Obviously, python interop isn't my
> concern here. I'm talking about the impact of your proposal on pure swift
> code.
>
> As example, calling some feature "unsafe", or "force", makes every
> developer pause and wonder if there isn't a better alternative. I'd like to
> see the same kind of things for dynamic calls.
>

A few examples of the kind of things that i would think about :
- use different file names for code using dynamic features.
- have every dynamic code included in a dynamic{} block
- have a different syntax for dynamic function calls, using ! or !!!
- Make the dynamic calls work only of subclasses of a base "PythonObject".
That would make the goal clear.
etc.
Now, i think those are all bad ideas (and that's why i don't like the
proposal in the first place). But i really think the concern should be
addressed.



>
>
> On Fri, Dec 1, 2017 at 10:30 AM, Chris Lattner via swift-evolution <
> swift-evolution at swift.org> wrote:
>
>> On Dec 1, 2017, at 12:26 AM, Douglas Gregor <dgregor at apple.com> wrote:
>>
>> *Philosophy*
>> Swift is, unabashedly, a strong statically-typed language.
>>
>>
>> That’s factually incorrect.
>>
>>
>> You’re going to have to explain that statement without reference to
>> AnyObject (we’ll discuss that case below).
>>
>>
>> Perhaps it depends on what you mean by “strong”: I interpreted that as
>> meaning that it provides type safety, with a level of strength akin to what
>> Java provides: a level that could support mobile code deployment.
>>
>> Swift certainly does not provide that strong of a static type system,
>> because it gives people explicit ways to opt out of that.
>> UnsafeMutableRawPointer, unsafe bitcast, and many other facilities support
>> this.  It also allows calling into non-type safe code, so it isn’t very
>> strong that way.  There are also race conditions and other holes in the
>> type system.
>>
>> All that said, I think it is correct that a subset of Swift exists that
>> does provide strong type safety, but particularly when bridging to C/ObjC
>> is involved, that quickly goes away.
>>
>>
>> More problematically for your argument: your preferred approach requires
>> the introduction of (something like) DynamicMemberLookupProtocol or
>> something like AnyObject-For-Python, so your proposal would be additive on
>> top of solving the core problem I’m trying to solve.  It isn’t an
>> alternative approach at all.
>>
>>
>> I wouldn’t say that’s my preferred approach. My preferred approach
>> involves taking the method/property/etc. declarations that already exist in
>> Python and mapping them into corresponding Swift declarations so we have
>> something to find with name lookup. One could put all of these declarations
>> on some PyVal struct or PythonObject and there would be no need for
>> AnyObject-for-Python or DynamicMemberLookupProtocol.
>>
>>
>> You’re suggesting that the transitive closure of all Python methods and
>> properties be preprocessed into a single gigantic Swift PyVal type?  I
>> guess something like that could be done.
>>
>> I would be concerned because there are many N^2 or worse algorithms in
>> the Swift compiler would probably explode.  It also doesn’t provide the
>> great tooling experience that you’re seeking, given that code completion
>> would show everything in the Python universe, which is not helpful.
>>
>> Further, it doesn’t provide a *better* experience than what I’m
>> suggesting, it seems strictly worse.  A preprocessing step prevents users
>> from playfully importing random Python modules into the Swift repl and
>> playgrounds.  It also seems worse for implementors (who will get a stream
>> of new bugs about compiler scalability).
>>
>> Whenever we discuss adding more dynamic features to Swift, there’s a
>> strong focus on maintaining that strong static type system.
>>
>>
>> Which this does, by providing full type safety - unlike AnyObject lookup.
>>
>>
>> You get dynamic safety because it goes into the Python interpreter; fair
>> enough. You get no help from your tools to form a correct invocation of any
>> method provided by Python.
>>
>>
>> Sure, that’s status quo for Python APIs.
>>
>> IMO, this proposal is a significant departure from the fundamental
>> character of Swift, because it allows access to possibly-nonexistent
>> members (as well as calls with incorrect arguments, in the related
>> proposal) without any indication that the operation might fail.
>>
>>
>> The only way your claim is correct is if someone implements the protocol
>> wrong.  What you describe is true of AnyObject lookup though, so I
>> understand how you could be confused by that.
>>
>>
>> AnyObject lookup still requires you to find an actual declaration with a
>> type signature. Yes, there are still failure cases. The dynamic type might
>> be totally unrelated to the class in which you found the declaration you’re
>> supposedly calling, which is a typical “unrecognized selector” failure and
>> will always be an issue with dynamic typing. The actual type safety hole
>> you’re presumably referring to is that the selector could be overloaded
>> with a different type signature, and we don’t proactively check that the
>> signature we type-checked against matches the signature found at runtime.
>> It’s doable with the Objective-C method encodings, but has never been
>> considered worthwhile.
>>
>>
>> To be clear, I’m not suggesting a change to AnyObject lookup.  I don’t
>> think that it is worth changing at this point in time.  My point was to
>> observe that the proposed DynamicMemberLookupProtocol proposal does not
>> suffer from these problems.  It really is type / memory safe, assuming a
>> sane implementation.
>>
>>
>> It’s easy to fall through these cracks for any type that supports
>> DynamicMemberLookupProtocol—a single-character typo when using a
>> DynamicMemberLookupProtocol-capable type means you’ve fallen out of the
>> safety that Swift provides.
>>
>>
>> Since you seem to be latching on to this, I’ll say it again: the proposal
>> is fully type safe :-)
>>
>> That said, the “simple typo” problem is fundamental to the problem of
>> working with a dynamically typed language: unless you can eradicate all
>> “fundamental dynamism” from the language, you cannot prevent this.
>>
>> That said, since this is a problem inherent with these languages, it is
>> something people are already very familiar with, and something that
>> everyone using those APIs has had to deal with.  This is also not our
>> problem to solve, and people in the Python community have been discussing
>> and working on it over a large part of its 25 year history.  I find your
>> belief that we could solve this for Python better than the Python community
>> itself has both idealistic and a bit naive.
>>
>>
>> I’m not pretending we can fully solve the problem. I’m pointing out that
>> depending entirely on DynamicMemberLookupProtocol throws away information
>> about method declarations that is present in Python and used by Python
>> tooling to good effect.
>>
>>
>> I’m not opposed to going further over time, but I’d like to get started
>> at some point :-).  I’m not in a super urgent hurry to get this in in the
>> next week or month or anything like that, but I also don’t want to wait
>> until Swift 10.
>>
>>
>> *Tooling*
>> The Python interoperability enabled by this proposal *does* look fairly
>> nice when you look at a small, correctly-written example. However,
>> absolutely none of the tooling assistance we rely on when writing such code
>> will work for Python interoperability. Examples:
>>
>> * As noted earlier, if you typo’d a name of a Python entity or passed the
>> wrong number of arguments to it, the compiler will not tell you: it’ll be a
>> runtime failure in the Python interpreter. I guess that’s what you’d get if
>> you were writing the code in Python, but Swift is supposed to be *better*
>> than Python if we’re to convince a community to use Swift instead.
>> * Code completion won’t work, because Swift has no visibility into
>> declarations written in Python
>> * Indexing/jump-to-definition/lookup documentation/generated interface
>> won’t ever work. None of the IDE features supported by SourceKit will work,
>> which will be a significant regression for users coming from a
>> Python-capable IDE.
>>
>>
>> Yes, welcome to the realities of modern Python development!
>>
>>
>> Python plugins for IDEs (e.g., for Atom) provide code completion, goto
>> definition, and other related features. None of the Swift tooling will work
>> if Swift’s interoperability with Python is entirely based on
>> DynamicMemberLookupProtocol.
>>
>>
>> I don’t understand your rationale here.  I think you agree that we need
>> to support the fully dynamic case (which is what I’m proposing).  That
>> said, this step does not preclude introducing importer magic (either
>> through compiler hackery or a theoretical "type providers” style of
>> feature).  Doing so would provide the functionality you’re describing.
>>
>>
>> Statically-typed languages should be a boon for tooling, but if a user
>> coming from Python to Swift *because* it’s supposed to be a better
>> development experience actually sees a significantly worse development
>> experience, we’re not going to win them over. It’ll just feel inconsistent.
>>
>>
>> By your argument we should ban AnyObject lookup as well, given its
>> inconsistency with the rest of the language.
>>
>>
>> By my argument, we should at least replace the
>> ImplicitlyUnwrappedOptional result with a true Optional (so one has to
>> acknowledge that the method shouldn’t be there). We’ve seriously considered
>> it, but it’s a source-breaking change, and it hasn’t seemed worth the
>> engineering effort to pursue it.
>>
>>
>> Ok, that would be a nice step, but doesn’t fix the type safety hole.
>>
>> In any case, my proposal allows the use of strong optional results as
>> well, so the fate of AnyObject isn’t really bound up with it.
>>
>> I don’t think that removing AnyObject dispatch entirely is possible at
>> this point in Swift’s lifetime. While AnyObject has become much less
>> prominent than it was in the Swift 1.0 days ([AnyObject] and [NSObject :
>> AnyObject], oh my!), there is still a significant amount of code using it
>> in the wild.
>>
>>
>> Completely agreed.  The major advantage I see of changing it now is if
>> there is some small mostly-user-invisible-change that allows a dramatic
>> simplification to the compiler implementation.
>>
>> *Dynamic Typing Features*
>> It’s possible that the right evolutionary path for Swift involves some
>> notion of dynamic typing, which would have a lot of the properties sought
>> by this proposal (and the DynamicCallableProtocol one). If that is true—and
>> I’m not at all convinced that it is—we shouldn’t accidentally fall into a
>> suboptimal design by taking small, easy, steps.
>>
>>
>> Given that you haven’t followed the discussion on the many threads we’ve
>> had on this, and haven’t proposed a workable approach to this problem, I’m
>> not sure upon what basis your fears and uncertainty and doubt are founded.
>>
>>
>> A few meta-comments here. First of all, following all previous threads on
>> a discussion is not realistic.
>>
>>
>> I understand that, but you’re also accusing the proposal of being a
>> suboptimal design made by looking at a series of small easy steps, instead
>> of the right design for the long term.  I’m pointing out that it is hard to
>> see the rationale for that sort of claim.
>>
>> This is part of the reason why we have different stages in a proposal’s
>> lifetime, and is the responsibility of the proposal’s authors to capture
>> alternatives and rationale in the proposal to make it self-contained.
>>
>>
>> Agreed.  As I mentioned in my previous email, I definitely screwed that
>> up by not capturing this discussion in the proposal.  Thank you again for
>> pulling this perspective to the front of the discussion so I could fix that
>> oversight.
>>
>> This proposal went through rapid iteration in the pitch phase and has now
>> been elevated to a pull request to ask for formal review—you should expect
>> more people to come on board having not read those threads. I appreciate
>> that you have now captured more alternatives and rationale in the proposal.
>>
>>
>> Agreed.  This is why I’ve been proactive about starting threads and
>> trying to keep visibility on the proposal each time there is a significant
>> change.  I really do value the discussion and feedback (both on the
>> proposed direction but also the writing itself).
>>
>> Second, it is absolutely reasonable to disagree with the technical
>> direction of a proposal without providing a complete solution to the
>> problem that the proposal is attempting to solve. Some problems aren’t
>> worth solving at all, or fully.
>>
>>
>> Ok, but at some point, if there is no alternative proposed, then a strong
>> opposition has the appearance of saying “we shouldn’t solve this problem”.
>> It was my understanding that thought that this was a worthwhile problem to
>> solve.
>>
>>
>> *How Should Python Interoperability Work?*
>> Going back to the central motivator for this proposal, I think that
>> providing something akin to the Clang Importer provides the best
>> interoperability experience: it would turn Python declarations into *real*
>> Swift declarations, so that we get the various tooling benefits of having a
>> strong statically-typed language.
>>
>>
>> This could be an theoretically interesting refinement to this proposal
>> but I’m personally very skeptical that this is every going to happen.  I’ve
>> put the rationale into the alternatives section of the proposal.  I don’t
>> explain it in the proposal in this way directly, but I believe it is far
>> more likely for a Pythonista transplant into Swift to rewrite their code in
>> Swift than it is to use Python type annotations.
>>
>>
>> I assume that this belief is based on type annotations lack of traction
>> in the Python community thus far?
>>
>>
>> There are many parts to this, which have to do with the ObjC<->Swift
>> situation being very different than the Python<->Swift situation:
>>
>> 1) The annotations don’t have significant traction in the Python
>> community.
>> 2) The Python annotations are not as powerful as ObjC generics are, and
>> thus lack important expressive capability.
>> 3) Many Python APIs are wrappers for C APIs.  “Swiftizing” a Python API
>> in this case means writing a new Swift wrapper for the API, not adding type
>> annotations.
>> 4) The Python community doesn’t care about Swift, and are not motivated
>> to do things to make Swift succeed.
>> 5) There is no “clang equivalent” for Python (that I’m aware of) which
>> close enough to the way Clang does for us to directly use.  The owners of
>> the existing Python compiler/interpreter implementations are not going to
>> be strongly motivated to change their stuff for us.
>>
>> Finally, just MHO, but I don’t expect a lot of “mix and match"
>> Python/Swift apps to exist (where the developer owns both the Python and
>> the Swift code), which is one case where type annotations are super awesome
>> in ObjC.  IMO, the most important use-case is a Swift program that uses
>> some Python APIs.
>>
>>
>> Sure, the argument types will all by PyObject or PyVal,
>>
>>
>> That’s the root of the problem.  Python has the “fully dynamic”
>> equivalent of “id” in Objective-C, so we need to represent that somehow.
>> Even if we followed the implementation approach of the Clang importer, we
>> would need some way to represent this dynamic case.  That type needs
>> features like DynamicMemberLookup or AnyObject.  In my opinion, the
>> DynamicMemberLookup approach is better in every way than AnyObject is.
>>
>>
>> The AnyObject approach has the advantage of knowing the set of declared,
>> reachable APIs:
>>
>> * Code completion shows all of the APIs that are possible to use via
>> dynamic dispatch, with their signatures so can fill in the right # of
>> arguments, see the names of the parameters, see documentation, etc.
>> * Indexing/refactoring/goto definition all point you to the declarations
>> that could be the targets of dynamic dispatch
>>
>> The DynamicMemberLookup approach is better for cases where you don’t have
>> a declaration of the member you want to access. I suspect that’s not the
>> common case.
>>
>>
>> Your points are valid, but the advantages for Objective-C don’t obviously
>> translate to Swift.  Note that ObjC (due to its heritage) has very long
>> method names that are perhaps arguably designed to not conflict with each
>> other often.  Python doesn’t have this heritage, and it has much shorter
>> names, which means that we’ll get a lot more conflicts and a lot less
>> “safety" out of this.
>>
>> AnyObject lookup also depends on a strange set of scoping heuristics that
>> was designed to be similar to Clang’s “header import” scope.  It isn’t
>> clear that this approach will work in Python, given that it doesn’t have an
>> analogue of umbrella headers that import things that cross frameworks.
>>
>>
>> Which approach do you think is the best way to handle the untyped
>> “actually dynamic” case?
>>
>>
>> AnyObject already exists in the language, and it fits the untyped
>> “actually dynamic” case well. It does require having a declaration for the
>> thing you want to reference, which I consider to be important: we can
>> code-complete those declarations, goto-definition to see those
>> declarations, index/refactor/look up documentation based on those
>> declarations.
>>
>> I’d be more inclined to push for the ImplicitlyUnwrappedOptional ->
>> Optional change if we did something to make AnyObject more prominent in
>> Swift.
>>
>>
>> I didn’t realize that you were thinking we would literally use AnyObject
>> itself.  I haven’t thought fully through it, but I think this will provide
>> several problems:
>>
>> 1) You’re mushing all of the ObjC and Python world’s together, making the
>> ObjC interop worse just because you’re doing some Python stuff too.
>> 2) You’re introducing ambiguity: does “ao = [1,2,3]” create an NSArray or
>> a Python array?  How do string literals work? (The answer is obvious,
>> Python loses). Maybe there is some really complicated bridging solution to
>> these problems, but that causes its own massive complexity spiral.
>> 3) You can’t realistically overload the Python operator set on AnyObject,
>> which means you get a worse python experience.
>> 4) AnyObject magic is currently limited to Apple platforms.  This would
>> bring its problems to other platforms like Linux.
>>
>> There are probably other issues, but I haven’t thought through it.
>>
>>
>> but the names are there for code completion (and indexing, etc.) to work,
>> and one could certainly imagine growing the importer to support Python’s typing
>> annotations <https://docs.python.org/3/library/typing.html>.
>>
>>
>> You’re basing this on the flawed assumption that local variables will
>> pervasively have types, which I can’t imagine being the case.  Even on
>> "typable” API, I wouldn’t expect people to commonly get code completion
>> results for reasons now explained in the proposal.
>>
>>
>> Remember that one *does* get code completion results for member access
>> into an AnyObject… lots of them… but the list filters down pretty fast when
>> you type a few characters, and then you get a member access that’ll fill in
>> stubs for (say) the arguments to the method you were trying to call. But
>> you can’t get those code completion results without having declarations to
>> complete to.
>>
>>
>> Fair point, it’s unclear to me how useful this would be with python’s
>> style of naming, but it could work.
>>
>> In truth, you don’t even need the compiler to be involved. The dynamic
>> “subscript” operation could be implemented in a Swift library, and one
>> could write a Python program to process a Python module and emit Swift
>> wrappers that call into that subscript operation. You’ll get all of the
>> tooling benefits with no compiler changes, and can tweak the wrapper
>> generation however much you want, using typing annotations or other
>> Python-specific information to create better wrappers over time.
>>
>>
>> I’d love for you to sketch out how any of this works with an acceptable
>> user experience, because I don't see anything concrete here.
>>
>>
>> We don’t need the basic dynamic case in the language to do this
>> experiment. Take the PyVal struct from the proposal. Now, write a Python
>> script that loads some module Foo and uses Python’s inspect
>> <https://docs.python.org/3/library/inspect.html> module to go find the
>> classes, methods, etc., and pretty-print Swift code that uses PyVal. So
>> this:
>>
>> def add_trick(self, trick):
>>
>> turns into
>>
>> extension PyVal {
>>   func add_trick(_ trick: PyVal) -> PyVal {
>>     /* do the magic to call into Python */
>>   }
>> }
>>
>> Using the inspect module, you can extract parameter names, default
>> arguments, docstrings, and more to reflect the existing Python API as Swift
>> API, packed into a bridging module.
>>
>> Note that we have a “flat” namespace of all Python methods on PyVal,
>> which is basically what you get with AnyObject today. Swift tooling will
>> provide code completion for member accesses into PyVal. Goto definition
>> will jump to the pretty-printed declarations, which could have the
>> docstrings formatted in comments and would show up in QuickHelp. The types
>> are weak (everything is PyVal), but that’s what we expect from importing a
>> dynamically-typed language.
>>
>>
>> As I mention above, I expect this to expose significant scalability
>> problems in the Swift compiler and it also defeats REPL/Playgrounds.  Being
>> able to use the Swift REPL is really important for Python programmers.
>>
>> -Chris
>>
>>
>> _______________________________________________
>> swift-evolution mailing list
>> swift-evolution at swift.org
>> https://lists.swift.org/mailman/listinfo/swift-evolution
>>
>>
>
> _______________________________________________
> swift-evolution mailing list
> swift-evolution at swift.org
> https://lists.swift.org/mailman/listinfo/swift-evolution
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-evolution/attachments/20171201/72fbfb62/attachment.html>