[swift-evolution] [Pitch] Introduce User-defined "Dynamic Member Lookup" Types

[Chris Lattner]

Hi All,

As a peer to the DynamicCallable proposal (https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d <https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d>), I’d like to get your feedback on making member lookup dynamically extensible.  My primary motivation is to improve interoperability with dynamic languages like Python, Perl, Ruby, Javascript, etc, but there are other use cases (e.g. when working with untyped JSON).

In addition to being a high impact on expressivity of Swift, I believe an implementation can be done in a way with changes that are localized, and thus not have significant impact on the maintainability of the compiler as a whole.  Once the pitch phase of this proposal helps refine the details, I’ll be happy to prepare an implementation for consideration.

In case it is useful, I’m working on cleaning up my current prototype Python bindings.  I’ll share them in the next day or two in case they are useful to provide context.  It is amazingly simple: less than 500 lines of Swift code (plus some small additional C header glue to work around clang importer limitations) enables impressive interoperability.  The only problems are the verbosity addressed by this proposal and the peer DynamicCallable proposal.


Here is the canonical proposal URL:
https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438 <https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438>

A snapshot of the proposal is included below in case it is useful.  Thanks in advance for help improving the proposal!

-Chris


Introduce User-defined "Dynamic Member Lookup" Types

Proposal: SE-NNNN <https://gist.github.com/lattner/NNNN-DynamicMemberLookup.md>
Author: Chris Lattner <https://github.com/lattner>
Review Manager: TBD
Status: Awaiting implementation
 <https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438#introduction>Introduction

This proposal introduces a new DynamicMemberLookupProtocol type to the standard library. Types that conform to it provide "dot" syntax for arbitrary names which are resolved at runtime. It is simple syntactic sugar which allows the user to write:

    a = someValue.someMember
    someValue.someMember = a
and have it be interpreted by the compiler as:

  a = someValue[dynamicMember: "someMember"]
  someValue[dynamicMember: "someMember"] = a
Many other languages have analogous features (e.g. the composition of Objective-C's explicit properties <https://developer.apple.com/library/content/documentation/General/Conceptual/DevPedia-CocoaCore/DeclaredProperty.html> and underlying messaging infrastructure <https://developer.apple.com/library/content/documentation/Cocoa/Conceptual/ObjCRuntimeGuide/Articles/ocrtHowMessagingWorks.html>). This sort of functionality is great for implementing dynamic language interoperability, dynamic proxy APIs <https://developer.apple.com/library/content/documentation/Cocoa/Conceptual/ObjCRuntimeGuide/Articles/ocrtForwarding.html>, and other APIs (e.g. for JSON processing).

Swift-evolution thread: Discussion thread topic for that proposal <https://lists.swift.org/pipermail/swift-evolution/>
 <https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438#motivation-and-context>Motivation and Context

Swift is well known for being exceptional at interworking with existing C and Objective-C APIs, but its support for calling APIs written in scripting languages like Python, Perl, and Ruby is quite lacking.

C and Objective-C are integrated into Swift by expending a heroic amount of effort into integrating Clang ASTs, remapping existing APIs in an attempt to feel "Swifty", and by providing a large number of attributes and customization points for changing the behavior of this integration when writing an Objective-C header. The end result of this massive investment of effort is that Swift provides a better experience when programming against these legacy APIs than Objective-C itself did.

When considering the space of dynamic languages, three things are clear: 1) there are several different languages of interest, and they each have significant interest in different quarters: for example, Python is big in data science and machine learning, Ruby is popular for building server side apps, and even Perl is in still widely used. 2) These languages have decades of library building behind them, sometimes with significant communities <https://pandas.pydata.org/> and 3) there are one or two orders of magnitude more users of these libraries than there are people currently using Swift.

While it is theoretically possible to expend the same level of effort on each of these languages and communities as has been spent on Objective-C, it is quite clear that this would both ineffective as well as bad for Swift: It would be ineffective, because the Swift community has not leverage over these communities to force auditing and annotation of their APIs. It would be bad for Swift because it would require a ton of language-specific support (and a number of third-party dependencies) onto the compiler and runtime, each of which makes the implementation significantly more complex, difficult to reason about, difficult to maintain, and difficult to test the supported permutations. In short, we'd end up with a mess.

Fortunately for us, these scripting languages provide an extremely dynamic programming model where almost everything is discovered at runtime, and many of them are explicitly designed to be embedded into other languages and applications. This aspect allows us to embed APIs from these languages directly into Swift with no language support at all - without not the level of effort, integration, and invasiveness that Objective-C has benefited from. Instead of invasive importer work, we can write some language-specific Swift APIs, and leave the interop details to that library.

This offers a significant opportunity for us - the Swift community can "embrace" these dynamic language APIs (making them directly available in Swift) which reduces the pain of someone moving from one of those languages into Swift. It is true that the APIs thus provided will not feel "Swifty", but if that becomes a significant problem for any one API, then the community behind it can evaluate the problem and come up with a solution (either a Swift wrapper for the dynamic language, or a from-scratch Swift reimplementation of the desired API). In any case, if/when we face this challenge, it will be a good thing: we'll know that we've won a significant new community of Swift developers.

While it is possible today to import (nearly) arbitrary dynamic language APIs into Swift today, the resultant API is unusable for two major reasons: member lookup is too verbose to be acceptable, and calling behavior is similarly too verbose to be acceptable. As such, we seek to provide two "syntactic sugar" features that solve this problem. These sugars are specifically designed to be dynamic language independent and, indeed, independent of dynamic languages at all: we can imagine other usage for the same primitive capabilities.

The two proposals in question are the introduction of the DynamicCallable <https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d> protocol and a related DynamicMemberLookupProtocol proposal (this proposal). With these two extensions, we think we can eliminate the need for invasive importer magic by making interoperability with dynamic languages ergonomic enough to be acceptable.

For example, consider this Python code:

class Dog:
    def __init__(self, name):
        self.name = name
        self.tricks = []    # creates a new empty list for each dog
        
    def add_trick(self, trick):
        self.tricks.append(trick)
we would like to be able to use this from Swift like this (the comments show the corresponding syntax you would use in Python):

  // import DogModule
  // import DogModule.Dog as Dog    // an alternate
  let Dog = Python.import(“DogModule.Dog")

  // dog = Dog("Brianna")
  let dog = Dog("Brianna")

  // dog.add_trick("Roll over")
  dog.add_trick("Roll over")

  // dog2 = Dog("Kaylee").add_trick("snore")
  let dog2 = Dog("Kaylee").add_trick("snore")
Of course, this would also apply to standard Python APIs as well. Here is an example working with the Python pickleAPI and the builtin Python function open:

  // import pickle
  let pickle = Python.import("pickle")

  // file = open(filename)
  let file = Python.open(filename)

  // blob = file.read()
  let blob = file.read()

  // result = pickle.loads(blob)
  let result = pickle.loads(blob)
This can all be expressed today as library functionality written in Swift, but without this proposal, the code required is unnecessarily verbose and gross. Without it (but with the related DynamicCallable proposal <https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d> the code would have explicit member lookups all over the place:

  // import pickle
  let pickle = Python.get(member: "import")("pickle")

  // file = open(filename)
  let file = Python.get(member: "open")(filename)

  // blob = file.read()
  let blob = file.get(member: "read")()

  // result = pickle.loads(blob)
  let result = pickle.get(member: "loads")(blob)

  // dog2 = Dog("Kaylee").add_trick("snore")
  let dog2 = Dog("Kaylee").get(member: "add_trick")("snore")
While this is a syntactic sugar proposal, we believe that this expands Swift to be usable in important new domains. In addition to dynamic language interoperability, this sort of functionality is useful for other APIs, e.g. when working with dynamically typed unstructured data like JSON, which could provide an API like jsonValue?.jsonField1?.jsonField2where each field is dynamically looked up.

 <https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438#proposed-solution>Proposed solution

We propose introducing this protocol to the standard library:

protocol DynamicMemberLookupProtocol {
  associatedtype DynamicMemberLookupValue

  subscript(dynamicMember name: String) -> DynamicMemberLookupValue { get set }
}
It also extends the language such that member lookup syntax (x.y) - when it otherwise fails (because there is no member y defined on the type of x) and when applied to a value which conforms to DynamicMemberLookupProtocol- is accepted and transformed into a call to the subscript in the protocol. This ensures that no member lookup on such a type ever fails.

 <https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438#example-usage>Example Usage

While there are many potential uses of this sort of API (e.g. resolving JSON members to named results, producing optional bindings) a motivating example comes from a prototype Python interoperability layer. There are many ways to implement this, and the details are not particularly important, but it is perhaps useful to know that this is directly useful to address the motivation section described above. Given a currency type of PyVal (and a conforming implementation named PyRef), an implementation may look like:

extension PyVal {
  subscript(dynamicMember member: String) -> PyVal {
    get {
      let result = PyObject_GetAttrString(borrowedPyObject, member)!
      return PyRef(owned: result)  // PyObject_GetAttrString returns +1 result.
    }
    set {
      PyObject_SetAttrString(borrowedPyObject, member,
                             newValue.toPython().borrowedPyObject)
    }
  }
}
 <https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438#source-compatibility>Source compatibility

This is a strictly additive proposal with no source breaking changes.

 <https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438#effect-on-abi-stability>Effect on ABI stability

This is a strictly additive proposal with no ABI breaking changes.

 <https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438#effect-on-api-resilience>Effect on API resilience

This has no impact on API resilience which is not already captured by other language features.

 <https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438#alternatives-considered>Alternatives considered

A few alternatives were considered:

 <https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438#add-ability-to-provide-read-only-members>Add ability to provide read-only members

The implementation above does not allow an implementation to statically reject members that are read-only. If this was important to add, we could add another protocol to model this, along the lines of:

protocol DynamicMemberLookupGettableProtocol {
  associatedtype DynamicMemberLookupValue

  // gettable only
  subscript(dynamicMember name: String) -> DynamicMemberLookupValue { get }
}

protocol DynamicMemberLookupProtocol : DynamicMemberLookupGettableProtocol {
  // gettable and settable.
  subscript(dynamicMember name: String) -> DynamicMemberLookupValue { get set }
  }
This would allow a type to implement one or the other based on their capabilities. This proposal starts with a very simple design based on the requirements of dynamic languages (which have no apparent immutability model), but if there is demand for this (e.g. because we want input JSON values to be gettable but not settalbe), the author is happy to switch to this more general model.

 <https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438#naming>Naming

There is a lot of grounds to debate naming of the protocol and methods in this type. Suggestions (along with rationale to support them) are more than welcome.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-evolution/attachments/20171114/2d451303/attachment-0001.html>