[swift-evolution] Proposal: Introduce User-defined "Dynamic Member Lookup" Types

[Benjamin G]

I think the fear most of us , poor developers working with developers of
various skills, is the potential for abuse. I've heard many times that it
isn't a guiding principle for swift, so i'll just reformulate the concern
in a more acceptable way :
Could your proposal explain how the swift language will still keep
encouraging developers to use static / compile-time type checking *when
it's possible* ? (because i think we can all argue that a compile-time
checks is better than a runtime one). Obviously, python interop isn't my
concern here. I'm talking about the impact of your proposal on pure swift
code.

As example, calling some feature "unsafe", or "force", makes every
developer pause and wonder if there isn't a better alternative. I'd like to
see the same kind of things for dynamic calls.


On Fri, Dec 1, 2017 at 10:30 AM, Chris Lattner via swift-evolution <
swift-evolution at swift.org> wrote:

> On Dec 1, 2017, at 12:26 AM, Douglas Gregor <dgregor at apple.com> wrote:
>
> *Philosophy*
> Swift is, unabashedly, a strong statically-typed language.
>
>
> That’s factually incorrect.
>
>
> You’re going to have to explain that statement without reference to
> AnyObject (we’ll discuss that case below).
>
>
> Perhaps it depends on what you mean by “strong”: I interpreted that as
> meaning that it provides type safety, with a level of strength akin to what
> Java provides: a level that could support mobile code deployment.
>
> Swift certainly does not provide that strong of a static type system,
> because it gives people explicit ways to opt out of that.
> UnsafeMutableRawPointer, unsafe bitcast, and many other facilities support
> this.  It also allows calling into non-type safe code, so it isn’t very
> strong that way.  There are also race conditions and other holes in the
> type system.
>
> All that said, I think it is correct that a subset of Swift exists that
> does provide strong type safety, but particularly when bridging to C/ObjC
> is involved, that quickly goes away.
>
>
> More problematically for your argument: your preferred approach requires
> the introduction of (something like) DynamicMemberLookupProtocol or
> something like AnyObject-For-Python, so your proposal would be additive on
> top of solving the core problem I’m trying to solve.  It isn’t an
> alternative approach at all.
>
>
> I wouldn’t say that’s my preferred approach. My preferred approach
> involves taking the method/property/etc. declarations that already exist in
> Python and mapping them into corresponding Swift declarations so we have
> something to find with name lookup. One could put all of these declarations
> on some PyVal struct or PythonObject and there would be no need for
> AnyObject-for-Python or DynamicMemberLookupProtocol.
>
>
> You’re suggesting that the transitive closure of all Python methods and
> properties be preprocessed into a single gigantic Swift PyVal type?  I
> guess something like that could be done.
>
> I would be concerned because there are many N^2 or worse algorithms in the
> Swift compiler would probably explode.  It also doesn’t provide the great
> tooling experience that you’re seeking, given that code completion would
> show everything in the Python universe, which is not helpful.
>
> Further, it doesn’t provide a *better* experience than what I’m
> suggesting, it seems strictly worse.  A preprocessing step prevents users
> from playfully importing random Python modules into the Swift repl and
> playgrounds.  It also seems worse for implementors (who will get a stream
> of new bugs about compiler scalability).
>
> Whenever we discuss adding more dynamic features to Swift, there’s a
> strong focus on maintaining that strong static type system.
>
>
> Which this does, by providing full type safety - unlike AnyObject lookup.
>
>
> You get dynamic safety because it goes into the Python interpreter; fair
> enough. You get no help from your tools to form a correct invocation of any
> method provided by Python.
>
>
> Sure, that’s status quo for Python APIs.
>
> IMO, this proposal is a significant departure from the fundamental
> character of Swift, because it allows access to possibly-nonexistent
> members (as well as calls with incorrect arguments, in the related
> proposal) without any indication that the operation might fail.
>
>
> The only way your claim is correct is if someone implements the protocol
> wrong.  What you describe is true of AnyObject lookup though, so I
> understand how you could be confused by that.
>
>
> AnyObject lookup still requires you to find an actual declaration with a
> type signature. Yes, there are still failure cases. The dynamic type might
> be totally unrelated to the class in which you found the declaration you’re
> supposedly calling, which is a typical “unrecognized selector” failure and
> will always be an issue with dynamic typing. The actual type safety hole
> you’re presumably referring to is that the selector could be overloaded
> with a different type signature, and we don’t proactively check that the
> signature we type-checked against matches the signature found at runtime.
> It’s doable with the Objective-C method encodings, but has never been
> considered worthwhile.
>
>
> To be clear, I’m not suggesting a change to AnyObject lookup.  I don’t
> think that it is worth changing at this point in time.  My point was to
> observe that the proposed DynamicMemberLookupProtocol proposal does not
> suffer from these problems.  It really is type / memory safe, assuming a
> sane implementation.
>
>
> It’s easy to fall through these cracks for any type that supports
> DynamicMemberLookupProtocol—a single-character typo when using a
> DynamicMemberLookupProtocol-capable type means you’ve fallen out of the
> safety that Swift provides.
>
>
> Since you seem to be latching on to this, I’ll say it again: the proposal
> is fully type safe :-)
>
> That said, the “simple typo” problem is fundamental to the problem of
> working with a dynamically typed language: unless you can eradicate all
> “fundamental dynamism” from the language, you cannot prevent this.
>
> That said, since this is a problem inherent with these languages, it is
> something people are already very familiar with, and something that
> everyone using those APIs has had to deal with.  This is also not our
> problem to solve, and people in the Python community have been discussing
> and working on it over a large part of its 25 year history.  I find your
> belief that we could solve this for Python better than the Python community
> itself has both idealistic and a bit naive.
>
>
> I’m not pretending we can fully solve the problem. I’m pointing out that
> depending entirely on DynamicMemberLookupProtocol throws away information
> about method declarations that is present in Python and used by Python
> tooling to good effect.
>
>
> I’m not opposed to going further over time, but I’d like to get started at
> some point :-).  I’m not in a super urgent hurry to get this in in the next
> week or month or anything like that, but I also don’t want to wait until
> Swift 10.
>
>
> *Tooling*
> The Python interoperability enabled by this proposal *does* look fairly
> nice when you look at a small, correctly-written example. However,
> absolutely none of the tooling assistance we rely on when writing such code
> will work for Python interoperability. Examples:
>
> * As noted earlier, if you typo’d a name of a Python entity or passed the
> wrong number of arguments to it, the compiler will not tell you: it’ll be a
> runtime failure in the Python interpreter. I guess that’s what you’d get if
> you were writing the code in Python, but Swift is supposed to be *better*
> than Python if we’re to convince a community to use Swift instead.
> * Code completion won’t work, because Swift has no visibility into
> declarations written in Python
> * Indexing/jump-to-definition/lookup documentation/generated interface
> won’t ever work. None of the IDE features supported by SourceKit will work,
> which will be a significant regression for users coming from a
> Python-capable IDE.
>
>
> Yes, welcome to the realities of modern Python development!
>
>
> Python plugins for IDEs (e.g., for Atom) provide code completion, goto
> definition, and other related features. None of the Swift tooling will work
> if Swift’s interoperability with Python is entirely based on
> DynamicMemberLookupProtocol.
>
>
> I don’t understand your rationale here.  I think you agree that we need to
> support the fully dynamic case (which is what I’m proposing).  That said,
> this step does not preclude introducing importer magic (either through
> compiler hackery or a theoretical "type providers” style of feature).
> Doing so would provide the functionality you’re describing.
>
>
> Statically-typed languages should be a boon for tooling, but if a user
> coming from Python to Swift *because* it’s supposed to be a better
> development experience actually sees a significantly worse development
> experience, we’re not going to win them over. It’ll just feel inconsistent.
>
>
> By your argument we should ban AnyObject lookup as well, given its
> inconsistency with the rest of the language.
>
>
> By my argument, we should at least replace the ImplicitlyUnwrappedOptional
> result with a true Optional (so one has to acknowledge that the method
> shouldn’t be there). We’ve seriously considered it, but it’s a
> source-breaking change, and it hasn’t seemed worth the engineering effort
> to pursue it.
>
>
> Ok, that would be a nice step, but doesn’t fix the type safety hole.
>
> In any case, my proposal allows the use of strong optional results as
> well, so the fate of AnyObject isn’t really bound up with it.
>
> I don’t think that removing AnyObject dispatch entirely is possible at
> this point in Swift’s lifetime. While AnyObject has become much less
> prominent than it was in the Swift 1.0 days ([AnyObject] and [NSObject :
> AnyObject], oh my!), there is still a significant amount of code using it
> in the wild.
>
>
> Completely agreed.  The major advantage I see of changing it now is if
> there is some small mostly-user-invisible-change that allows a dramatic
> simplification to the compiler implementation.
>
> *Dynamic Typing Features*
> It’s possible that the right evolutionary path for Swift involves some
> notion of dynamic typing, which would have a lot of the properties sought
> by this proposal (and the DynamicCallableProtocol one). If that is true—and
> I’m not at all convinced that it is—we shouldn’t accidentally fall into a
> suboptimal design by taking small, easy, steps.
>
>
> Given that you haven’t followed the discussion on the many threads we’ve
> had on this, and haven’t proposed a workable approach to this problem, I’m
> not sure upon what basis your fears and uncertainty and doubt are founded.
>
>
> A few meta-comments here. First of all, following all previous threads on
> a discussion is not realistic.
>
>
> I understand that, but you’re also accusing the proposal of being a
> suboptimal design made by looking at a series of small easy steps, instead
> of the right design for the long term.  I’m pointing out that it is hard to
> see the rationale for that sort of claim.
>
> This is part of the reason why we have different stages in a proposal’s
> lifetime, and is the responsibility of the proposal’s authors to capture
> alternatives and rationale in the proposal to make it self-contained.
>
>
> Agreed.  As I mentioned in my previous email, I definitely screwed that up
> by not capturing this discussion in the proposal.  Thank you again for
> pulling this perspective to the front of the discussion so I could fix that
> oversight.
>
> This proposal went through rapid iteration in the pitch phase and has now
> been elevated to a pull request to ask for formal review—you should expect
> more people to come on board having not read those threads. I appreciate
> that you have now captured more alternatives and rationale in the proposal.
>
>
> Agreed.  This is why I’ve been proactive about starting threads and trying
> to keep visibility on the proposal each time there is a significant
> change.  I really do value the discussion and feedback (both on the
> proposed direction but also the writing itself).
>
> Second, it is absolutely reasonable to disagree with the technical
> direction of a proposal without providing a complete solution to the
> problem that the proposal is attempting to solve. Some problems aren’t
> worth solving at all, or fully.
>
>
> Ok, but at some point, if there is no alternative proposed, then a strong
> opposition has the appearance of saying “we shouldn’t solve this problem”.
> It was my understanding that thought that this was a worthwhile problem to
> solve.
>
>
> *How Should Python Interoperability Work?*
> Going back to the central motivator for this proposal, I think that
> providing something akin to the Clang Importer provides the best
> interoperability experience: it would turn Python declarations into *real*
> Swift declarations, so that we get the various tooling benefits of having a
> strong statically-typed language.
>
>
> This could be an theoretically interesting refinement to this proposal but
> I’m personally very skeptical that this is every going to happen.  I’ve put
> the rationale into the alternatives section of the proposal.  I don’t
> explain it in the proposal in this way directly, but I believe it is far
> more likely for a Pythonista transplant into Swift to rewrite their code in
> Swift than it is to use Python type annotations.
>
>
> I assume that this belief is based on type annotations lack of traction in
> the Python community thus far?
>
>
> There are many parts to this, which have to do with the ObjC<->Swift
> situation being very different than the Python<->Swift situation:
>
> 1) The annotations don’t have significant traction in the Python
> community.
> 2) The Python annotations are not as powerful as ObjC generics are, and
> thus lack important expressive capability.
> 3) Many Python APIs are wrappers for C APIs.  “Swiftizing” a Python API in
> this case means writing a new Swift wrapper for the API, not adding type
> annotations.
> 4) The Python community doesn’t care about Swift, and are not motivated to
> do things to make Swift succeed.
> 5) There is no “clang equivalent” for Python (that I’m aware of) which
> close enough to the way Clang does for us to directly use.  The owners of
> the existing Python compiler/interpreter implementations are not going to
> be strongly motivated to change their stuff for us.
>
> Finally, just MHO, but I don’t expect a lot of “mix and match"
> Python/Swift apps to exist (where the developer owns both the Python and
> the Swift code), which is one case where type annotations are super awesome
> in ObjC.  IMO, the most important use-case is a Swift program that uses
> some Python APIs.
>
>
> Sure, the argument types will all by PyObject or PyVal,
>
>
> That’s the root of the problem.  Python has the “fully dynamic” equivalent
> of “id” in Objective-C, so we need to represent that somehow.  Even if we
> followed the implementation approach of the Clang importer, we would need
> some way to represent this dynamic case.  That type needs features like
> DynamicMemberLookup or AnyObject.  In my opinion, the DynamicMemberLookup
> approach is better in every way than AnyObject is.
>
>
> The AnyObject approach has the advantage of knowing the set of declared,
> reachable APIs:
>
> * Code completion shows all of the APIs that are possible to use via
> dynamic dispatch, with their signatures so can fill in the right # of
> arguments, see the names of the parameters, see documentation, etc.
> * Indexing/refactoring/goto definition all point you to the declarations
> that could be the targets of dynamic dispatch
>
> The DynamicMemberLookup approach is better for cases where you don’t have
> a declaration of the member you want to access. I suspect that’s not the
> common case.
>
>
> Your points are valid, but the advantages for Objective-C don’t obviously
> translate to Swift.  Note that ObjC (due to its heritage) has very long
> method names that are perhaps arguably designed to not conflict with each
> other often.  Python doesn’t have this heritage, and it has much shorter
> names, which means that we’ll get a lot more conflicts and a lot less
> “safety" out of this.
>
> AnyObject lookup also depends on a strange set of scoping heuristics that
> was designed to be similar to Clang’s “header import” scope.  It isn’t
> clear that this approach will work in Python, given that it doesn’t have an
> analogue of umbrella headers that import things that cross frameworks.
>
>
> Which approach do you think is the best way to handle the untyped
> “actually dynamic” case?
>
>
> AnyObject already exists in the language, and it fits the untyped
> “actually dynamic” case well. It does require having a declaration for the
> thing you want to reference, which I consider to be important: we can
> code-complete those declarations, goto-definition to see those
> declarations, index/refactor/look up documentation based on those
> declarations.
>
> I’d be more inclined to push for the ImplicitlyUnwrappedOptional ->
> Optional change if we did something to make AnyObject more prominent in
> Swift.
>
>
> I didn’t realize that you were thinking we would literally use AnyObject
> itself.  I haven’t thought fully through it, but I think this will provide
> several problems:
>
> 1) You’re mushing all of the ObjC and Python world’s together, making the
> ObjC interop worse just because you’re doing some Python stuff too.
> 2) You’re introducing ambiguity: does “ao = [1,2,3]” create an NSArray or
> a Python array?  How do string literals work? (The answer is obvious,
> Python loses). Maybe there is some really complicated bridging solution to
> these problems, but that causes its own massive complexity spiral.
> 3) You can’t realistically overload the Python operator set on AnyObject,
> which means you get a worse python experience.
> 4) AnyObject magic is currently limited to Apple platforms.  This would
> bring its problems to other platforms like Linux.
>
> There are probably other issues, but I haven’t thought through it.
>
>
> but the names are there for code completion (and indexing, etc.) to work,
> and one could certainly imagine growing the importer to support Python’s typing
> annotations <https://docs.python.org/3/library/typing.html>.
>
>
> You’re basing this on the flawed assumption that local variables will
> pervasively have types, which I can’t imagine being the case.  Even on
> "typable” API, I wouldn’t expect people to commonly get code completion
> results for reasons now explained in the proposal.
>
>
> Remember that one *does* get code completion results for member access
> into an AnyObject… lots of them… but the list filters down pretty fast when
> you type a few characters, and then you get a member access that’ll fill in
> stubs for (say) the arguments to the method you were trying to call. But
> you can’t get those code completion results without having declarations to
> complete to.
>
>
> Fair point, it’s unclear to me how useful this would be with python’s
> style of naming, but it could work.
>
> In truth, you don’t even need the compiler to be involved. The dynamic
> “subscript” operation could be implemented in a Swift library, and one
> could write a Python program to process a Python module and emit Swift
> wrappers that call into that subscript operation. You’ll get all of the
> tooling benefits with no compiler changes, and can tweak the wrapper
> generation however much you want, using typing annotations or other
> Python-specific information to create better wrappers over time.
>
>
> I’d love for you to sketch out how any of this works with an acceptable
> user experience, because I don't see anything concrete here.
>
>
> We don’t need the basic dynamic case in the language to do this
> experiment. Take the PyVal struct from the proposal. Now, write a Python
> script that loads some module Foo and uses Python’s inspect
> <https://docs.python.org/3/library/inspect.html> module to go find the
> classes, methods, etc., and pretty-print Swift code that uses PyVal. So
> this:
>
> def add_trick(self, trick):
>
> turns into
>
> extension PyVal {
>   func add_trick(_ trick: PyVal) -> PyVal {
>     /* do the magic to call into Python */
>   }
> }
>
> Using the inspect module, you can extract parameter names, default
> arguments, docstrings, and more to reflect the existing Python API as Swift
> API, packed into a bridging module.
>
> Note that we have a “flat” namespace of all Python methods on PyVal, which
> is basically what you get with AnyObject today. Swift tooling will provide
> code completion for member accesses into PyVal. Goto definition will jump
> to the pretty-printed declarations, which could have the docstrings
> formatted in comments and would show up in QuickHelp. The types are weak
> (everything is PyVal), but that’s what we expect from importing a
> dynamically-typed language.
>
>
> As I mention above, I expect this to expose significant scalability
> problems in the Swift compiler and it also defeats REPL/Playgrounds.  Being
> able to use the Swift REPL is really important for Python programmers.
>
> -Chris
>
>
> _______________________________________________
> swift-evolution mailing list
> swift-evolution at swift.org
> https://lists.swift.org/mailman/listinfo/swift-evolution
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-evolution/attachments/20171201/74902e9b/attachment.html>