[swift-evolution] Proposal: Introduce User-defined "Dynamic Member Lookup" Types

[Xiaodi Wu]

On Thu, Nov 30, 2017 at 2:24 AM, Douglas Gregor via swift-evolution <
swift-evolution at swift.org> wrote:

>
>
> On Nov 26, 2017, at 10:04 PM, Chris Lattner via swift-evolution <
> swift-evolution at swift.org> wrote:
>
> I’d like to formally propose the inclusion of user-defined dynamic member
> lookup types.
>
> Here is my latest draft of the proposal:
> https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438
> https://github.com/apple/swift-evolution/pull/768
>
> An implementation of this design is available here:
> https://github.com/apple/swift/pull/13076
>
> The implementation is straight-forward and (IMO) non-invasive in the
> compiler.
>
>
>
> I think better interoperability with Python (and other OO languages in
> widespread use) is a good goal, and I agree that the implementation of the
> feature described is straight-forward and not terribly invasive in the
> compiler.
>
> However, I do not think this proposal is going in the right direction for
> Swift. I have objections on several different grounds.
>
> *Philosophy*
> Swift is, unabashedly, a strong statically-typed language. We don’t allow
> implicit down casting, we require “as?” so you have to cope with the
> possibility of failure (or use “as!” and think hard about the “!”). Even
> the gaping hole that is AnyObject dispatch still requires the existence of
> an @objc declaration and produces an optional lookup result, so the user
> must contend with the potential for dynamic failure. Whenever we discuss
> adding more dynamic features to Swift, there’s a strong focus on
> maintaining that strong static type system.
>
> IMO, this proposal is a significant departure from the fundamental
> character of Swift, because it allows access to possibly-nonexistent
> members (as well as calls with incorrect arguments, in the related
> proposal) without any indication that the operation might fail. It’s easy
> to fall through these cracks for any type that supports
> DynamicMemberLookupProtocol—a single-character typo when using a
> DynamicMemberLookupProtocol-capable type means you’ve fallen out of the
> safety that Swift provides. I think that’s a poor experience for the Python
> interoperability case, but more on that in the Tooling section below.
>
> While we shouldn’t necessarily avoid a feature simply because it can be
> used distastefully, consider something like this:
>
> public extension NSObject :  DynamicMemberLookupProtocol,
> DynamicCallableProtocol { … }
>
> that goes directly to the Objective-C runtime to resolve member lookups
> and calls—avoiding @objc, bridging headers, and so on. It’s almost
> frighteningly convenient, and one could imagine some mixed
> Objective-C/Swift code bases where this would save a lot of typing (of
> code)… at the cost of losing static typing in the language. The presence of
> that one extension means I can no longer rely on the safety guarantees
> Swift normally provides, for any project that imports that extension and
> uses a subclass of NSObject. At best, we as a community decide “don’t do
> that”; at worse, some nontrivial fraction of the community decides that the
> benefits outweigh the costs (for this type or some other), and we can no
> longer say that Swift is a strong statically-typed language without adding
> “unless you’re using something that adopts DynamicMemberLookupProtocol”.
>

There are several commenters below to whom I would have liked to respond in
the fullness of time, but time constraints would make doing so prohibitive.
Since your message set off an abundance of discussion, I'll reply to the
points you make here and, along the way, ask your forbearance to bring up
and respond to some related concerns raised by others.

I agree that the prospect above seems not ideal at all. On reading Chris's
proposal, it never occurred to me that the intention was to support such
retroactive conformance to these special protocols. Admittedly, such
retroactive conformance is possible with all protocols--with the notable
exception of those that require compiler synthesis of requirements. But
Chris's protocols seemed magical enough (in the gut feeling sense) that I
naturally assumed that retroactive conformance was never on the table. We
would be justified in making that prohibition here, I think, although I'm
not sure if Chris as proposal author feels the same way.

Alternatively--and perhaps more elegantly--we could address this concern
head-on by having, instead of `DynamicMemberLookupProtocol` and
`DynamicCallable`, a magical class `DynamicObject` which all dynamic types
must inherit from. It would then be clear by design that Swift types cannot
be _retroactively dynamic_ in the sense that Chris proposes. I *think* the
vast majority of bridged use cases can tolerate being `final class` types
instead of `struct` types. I could be wrong though.

Now, as to the possibility of failure: I agree also that eliding the
possibility of lookup failure at the callsite requires further
consideration. Some might agree that restricting dynamic features only to
subclasses of a `DynamicObject` might be clear enough that we do not need
to go further in this regard. I think all will agree that inventing new (to
Swift) notations like "->" or "::" to denote dynamic lookup is rather
awkward and not ergonomic. I wonder if it would be instead sufficient to
require dynamic member lookup to return values of type `T!` (as in, IUOs).
IUOs are, after all, designed to deal with similar situations in bridging
from Obj-C, and are explicitly "transitional technology."

*Tooling*
> The Python interoperability enabled by this proposal *does* look fairly
> nice when you look at a small, correctly-written example. However,
> absolutely none of the tooling assistance we rely on when writing such code
> will work for Python interoperability. Examples:
>
> * As noted earlier, if you typo’d a name of a Python entity or passed the
> wrong number of arguments to it, the compiler will not tell you: it’ll be a
> runtime failure in the Python interpreter. I guess that’s what you’d get if
> you were writing the code in Python, but Swift is supposed to be *better*
> than Python if we’re to convince a community to use Swift instead.
> * Code completion won’t work, because Swift has no visibility into
> declarations written in Python
> * Indexing/jump-to-definition/lookup documentation/generated interface
> won’t ever work. None of the IDE features supported by SourceKit will work,
> which will be a significant regression for users coming from a
> Python-capable IDE.
>
> Statically-typed languages should be a boon for tooling, but if a user
> coming from Python to Swift *because* it’s supposed to be a better
> development experience actually sees a significantly worse development
> experience, we’re not going to win them over. It’ll just feel inconsistent.
>

As I wrote in the earlier thread, I think this is the wrong way to reason
about the use case--and a counterproductive one that effectively rejects
the legitimacy of dynamic language interop support instead of working
towards an optimal solution for it. Along those lines, some replies to your
message literally question whether the user case is worthwhile to support
at all, which I think entirely misses the mark.

As Chris writes in his proposal, there are several areas (such as data
science) where, to put it plainly, Python or another dynamic language is
significantly better than Swift due to a much larger ecosystem of
libraries, tools, and user communities, with sometimes decades of lead
time. It is nonsensical to talk about how Swift is "supposed to be better"
in any way whatsoever in that context. As diehard Swift users, we may be
confident that the virtues of Swift's syntax, static typing, compiler
smarts, protocol-based design, or whatever else offer opportunities for,
say, data science libraries and tools to be better in the future,
eventually. But to make this even possible involves first making Swift a
viable language in which to work with current data science tools. Your top
bullet point here reasons that one key flaw of Chris's proposal is that he
has not somehow figured out how to make dynamic Python calls give
compile-time Swift errors. If this is the minimum bar for Python interop,
then as far as I can tell, it seems isomorphic to rejecting interop with
dynamic languages altogether.

It goes without saying that, in bridging between languages X and Y, much of
X's native tooling will be inoperable, and much of Y's native tooling will
not work. The solution is to build additional tools where necessary, not to
argue that interop shouldn't be implemented in the first place.

*Dynamic Typing Features*
> It’s possible that the right evolutionary path for Swift involves some
> notion of dynamic typing, which would have a lot of the properties sought
> by this proposal (and the DynamicCallableProtocol one). If that is true—and
> I’m not at all convinced that it is—we shouldn’t accidentally fall into a
> suboptimal design by taking small, easy, steps. If we’re to include
> dynamic-typing facilities, we should look at more existing practice—C#
> ‘dynamic' is one such approach, but more promising would be some form of
> gradual typing a la TypeScript that let’s one more smoothly (and probably
> explicitly) shift between strong and weak typing.
>
> *How Should Python Interoperability Work?*
> Going back to the central motivator for this proposal, I think that
> providing something akin to the Clang Importer provides the best
> interoperability experience: it would turn Python declarations into *real*
> Swift declarations, so that we get the various tooling benefits of having a
> strong statically-typed language. Sure, the argument types will all by
> PyObject or PyVal, but the names are there for code completion (and
> indexing, etc.) to work, and one could certainly imagine growing the
> importer to support Python’s typing annotations
> <https://docs.python.org/3/library/typing.html>. But the important part
> here is that it doesn’t change the language model at all—it’s a compiler
> feature, separate from the language. Yes, the Clang importer is a big
> gnarly beast—but if the goal is to support N such importers, we can
> refactor and share common infrastructure to make them similar, perhaps
> introducing some kind of type provider infrastructure to allow one to write
> new importers as Swift modules.
>
> In truth, you don’t even need the compiler to be involved. The dynamic
> “subscript” operation could be implemented in a Swift library, and one
> could write a Python program to process a Python module and emit Swift
> wrappers that call into that subscript operation. You’ll get all of the
> tooling benefits with no compiler changes, and can tweak the wrapper
> generation however much you want, using typing annotations or other
> Python-specific information to create better wrappers over time.
>

I doubt seriously that there is any viable path to interoperating with a
much more established and extensive ecosystem which begins by requiring
that the more dominant ecosystem support Swift-specific annotations or
tooling. It would seem that, if we're to implement a Swift library to do
what you describe, it'd have to be one with extensive knowledge of both
Python and Swift, being able to parse entire Python modules as well as
create entire Swift ones. Since we haven't even designed a way of writing
code-generating macros for Swift in Swift, I struggle to see how this
hypothetical tool is going to be ever built, or whether sufficient people
exist with the expertise to do so, given that you'd need deep expertise
working in both languages in order to write the tool that permits you to
work in both languages.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-evolution/attachments/20171130/1e547ba7/attachment.html>