[swift-evolution] [Pitch] Introduce User-defined "Dynamic Member Lookup" Types

Wed Nov 15 14:36:43 CST 2017

Really interesting proposal,
I don't really have the need for Python interop at the moment so I
would be more interesting on the potential usages of this changes to
improve Swift code like the proxy and json accessors that you mention.
The JSON part I can see it, as you are basically accessing with
strings a keyed data structure.
But I'm having a hard time imagining how a proxy example would work as
there doesn't seem to be a mechanism to access Swift members from the
information received on the subscript. I pressume you can call your
proxied object subscript but that would mean that the proxied objects
need to be also DynamicMemberLookupProtocol and in any case at some
point down the chain somebody will have to dispatch from the "string"
to a member lookup with some switch or similar.
Am I missing something or is this just completely unrelated to the
proposal and I just left my imagination go to far? :P

On Wed, Nov 15, 2017 at 10:58 AM, Stefan Mayer-Popp via
swift-evolution <swift-evolution at swift.org> wrote:
> Hej Rick,
>
> i definitely agree with you. +1
>
> From my personal experience i can say that a good working language interoperability gives us tons of new possibilities.
>
> A big thing i like is that i could for example use already written projects in different languages which are maybe already in production and reuse their proven functionality without the requirement to transform the code to Swift.
> The complexity of the Swift language itself isn't even touched in any area a “casual” developer uses, i personally see no argument of not integrating this feature for Swift.
>
> I for myself can see a lot new possibilities and more reasons (and better arguments) for using Swift in other areas in my company like a normal phone app or some command-line tools.
> Symbiosis is great lets have some fun with Swython, Swuby or SwHP in future.
>
> --
> Stefan Mayer-Popp
>
>
>
>> On 15. Nov 2017, at 11:08, Rick Mann via swift-evolution <swift-evolution at swift.org> wrote:
>>
>> +1 to both proposals. I don't know enough to comment further, but I know this would help in the adoption of Swift on my team(s) (we call a lot of Python, although currently via exec-ing some bash scripts; it's a mess, but works well in the CLI world the stuff was written for).
>>
>>> On Nov 14, 2017, at 23:29 , Chris Lattner via swift-evolution <swift-evolution at swift.org> wrote:
>>>
>>> Hi All,
>>>
>>> As a peer to the DynamicCallable proposal (https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d), I’d like to get your feedback on making member lookup dynamically extensible.  My primary motivation is to improve interoperability with dynamic languages like Python, Perl, Ruby, Javascript, etc, but there are other use cases (e.g. when working with untyped JSON).
>>>
>>> In addition to being a high impact on expressivity of Swift, I believe an implementation can be done in a way with changes that are localized, and thus not have significant impact on the maintainability of the compiler as a whole.  Once the pitch phase of this proposal helps refine the details, I’ll be happy to prepare an implementation for consideration.
>>>
>>> In case it is useful, I’m working on cleaning up my current prototype Python bindings.  I’ll share them in the next day or two in case they are useful to provide context.  It is amazingly simple: less than 500 lines of Swift code (plus some small additional C header glue to work around clang importer limitations) enables impressive interoperability.  The only problems are the verbosity addressed by this proposal and the peer DynamicCallable proposal.
>>>
>>>
>>> Here is the canonical proposal URL:
>>> https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae5438
>>>
>>> A snapshot of the proposal is included below in case it is useful.  Thanks in advance for help improving the proposal!
>>>
>>> -Chris
>>>
>>>
>>> Introduce User-defined "Dynamic Member Lookup" Types
>>>
>>>      • Proposal: SE-NNNN
>>>      • Author: Chris Lattner
>>>      • Review Manager: TBD
>>>      • Status: Awaiting implementation
>>> Introduction
>>>
>>> This proposal introduces a new DynamicMemberLookupProtocol type to the standard library. Types that conform to it provide "dot" syntax for arbitrary names which are resolved at runtime. It is simple syntactic sugar which allows the user to write:
>>>
>>>    a = someValue.someMember
>>>
>>>    someValue.
>>> someMember = a
>>> and have it be interpreted by the compiler as:
>>>
>>>  a = someValue[dynamicMember: "someMember"
>>> ]
>>>  someValue[
>>> dynamicMember: "someMember"] = a
>>> Many other languages have analogous features (e.g. the composition of Objective-C's explicit properties and underlying messaging infrastructure). This sort of functionality is great for implementing dynamic language interoperability, dynamic proxy APIs, and other APIs (e.g. for JSON processing).
>>>
>>> Swift-evolution thread: Discussion thread topic for that proposal
>>>
>>> Motivation and Context
>>>
>>> Swift is well known for being exceptional at interworking with existing C and Objective-C APIs, but its support for calling APIs written in scripting languages like Python, Perl, and Ruby is quite lacking.
>>>
>>> C and Objective-C are integrated into Swift by expending a heroic amount of effort into integrating Clang ASTs, remapping existing APIs in an attempt to feel "Swifty", and by providing a large number of attributes and customization points for changing the behavior of this integration when writing an Objective-C header. The end result of this massive investment of effort is that Swift provides a better experience when programming against these legacy APIs than Objective-C itself did.
>>>
>>> When considering the space of dynamic languages, three things are clear: 1) there are several different languages of interest, and they each have significant interest in different quarters: for example, Python is big in data science and machine learning, Ruby is popular for building server side apps, and even Perl is in still widely used. 2) These languages have decades of library building behind them, sometimes with significant communities and 3) there are one or two orders of magnitude more users of these libraries than there are people currently using Swift.
>>>
>>> While it is theoretically possible to expend the same level of effort on each of these languages and communities as has been spent on Objective-C, it is quite clear that this would both ineffective as well as bad for Swift: It would be ineffective, because the Swift community has not leverage over these communities to force auditing and annotation of their APIs. It would be bad for Swift because it would require a ton of language-specific support (and a number of third-party dependencies) onto the compiler and runtime, each of which makes the implementation significantly more complex, difficult to reason about, difficult to maintain, and difficult to test the supported permutations. In short, we'd end up with a mess.
>>>
>>> Fortunately for us, these scripting languages provide an extremely dynamic programming model where almost everything is discovered at runtime, and many of them are explicitly designed to be embedded into other languages and applications. This aspect allows us to embed APIs from these languages directly into Swift with no language support at all - without not the level of effort, integration, and invasiveness that Objective-C has benefited from. Instead of invasive importer work, we can write some language-specific Swift APIs, and leave the interop details to that library.
>>>
>>> This offers a significant opportunity for us - the Swift community can "embrace" these dynamic language APIs (making them directly available in Swift) which reduces the pain of someone moving from one of those languages into Swift. It is true that the APIs thus provided will not feel "Swifty", but if that becomes a significant problem for any one API, then the community behind it can evaluate the problem and come up with a solution (either a Swift wrapper for the dynamic language, or a from-scratch Swift reimplementation of the desired API). In any case, if/when we face this challenge, it will be a good thing: we'll know that we've won a significant new community of Swift developers.
>>>
>>> While it is possible today to import (nearly) arbitrary dynamic language APIs into Swift today, the resultant API is unusable for two major reasons: member lookup is too verbose to be acceptable, and calling behavior is similarly too verbose to be acceptable. As such, we seek to provide two "syntactic sugar" features that solve this problem. These sugars are specifically designed to be dynamic language independent and, indeed, independent of dynamic languages at all: we can imagine other usage for the same primitive capabilities.
>>>
>>> The two proposals in question are the introduction of the DynamicCallable protocol and a related DynamicMemberLookupProtocol proposal (this proposal). With these two extensions, we think we can eliminate the need for invasive importer magic by making interoperability with dynamic languages ergonomic enough to be acceptable.
>>>
>>> For example, consider this Python code:
>>>
>>> class Dog
>>> :
>>>
>>> def __init__(self, name
>>> ):
>>>
>>> self.name =
>>> name
>>>
>>> self.tricks = []    # creates a new empty list for each dog
>>>
>>>
>>>
>>> def add_trick(self, trick
>>> ):
>>>
>>> self.tricks.append(trick)
>>> we would like to be able to use this from Swift like this (the comments show the corresponding syntax you would use in Python):
>>>
>>>  // import DogModule
>>>  // import DogModule.Dog as Dog    // an alternate
>>>  let Dog = Python.import(“DogModule.Dog")
>>>
>>>  // dog = Dog("Brianna")
>>>  let dog = Dog("Brianna")
>>>
>>>  // dog.add_trick("Roll over")
>>>  dog.add_trick("Roll over")
>>>
>>>  // dog2 = Dog("Kaylee").add_trick("snore")
>>>  let dog2 = Dog("Kaylee").add_trick("snore")
>>> Of course, this would also apply to standard Python APIs as well. Here is an example working with the Python pickleAPI and the builtin Python function open:
>>>
>>>  // import pickle
>>>  let pickle = Python.import("pickle"
>>> )
>>>
>>>
>>> // file = open(filename)
>>>  let file = Python.open
>>> (filename)
>>>
>>>
>>> // blob = file.read()
>>>  let blob = file.read
>>> ()
>>>
>>>
>>> // result = pickle.loads(blob)
>>>  let result = pickle.loads(blob)
>>> This can all be expressed today as library functionality written in Swift, but without this proposal, the code required is unnecessarily verbose and gross. Without it (but with the related DynamicCallable proposal the code would have explicit member lookups all over the place:
>>>
>>>  // import pickle
>>>  let pickle = Python.get(member: "import")("pickle"
>>> )
>>>
>>>
>>> // file = open(filename)
>>>  let file = Python.get(member: "open"
>>> )(filename)
>>>
>>>
>>> // blob = file.read()
>>>  let blob = file.get(member: "read"
>>> )()
>>>
>>>
>>> // result = pickle.loads(blob)
>>>  let result = pickle.get(member: "loads"
>>> )(blob)
>>>
>>>
>>> // dog2 = Dog("Kaylee").add_trick("snore")
>>>  let dog2 = Dog("Kaylee").get(member: "add_trick")("snore")
>>> While this is a syntactic sugar proposal, we believe that this expands Swift to be usable in important new domains. In addition to dynamic language interoperability, this sort of functionality is useful for other APIs, e.g. when working with dynamically typed unstructured data like JSON, which could provide an API like jsonValue?.jsonField1?.jsonField2where each field is dynamically looked up.
>>>
>>> Proposed solution
>>>
>>> We propose introducing this protocol to the standard library:
>>>
>>> protocol DynamicMemberLookupProtocol
>>> {
>>>
>>> associatedtype DynamicMemberLookupValue
>>>
>>>
>>>
>>> subscript(dynamicMember name: String) -> DynamicMemberLookupValue { get set
>>> }
>>> }
>>>
>>> It also extends the language such that member lookup syntax (x.y) - when it otherwise fails (because there is no member y defined on the type of x) and when applied to a value which conforms to DynamicMemberLookupProtocol- is accepted and transformed into a call to the subscript in the protocol. This ensures that no member lookup on such a type ever fails.
>>>
>>> Example Usage
>>>
>>> While there are many potential uses of this sort of API (e.g. resolving JSON members to named results, producing optional bindings) a motivating example comes from a prototype Python interoperability layer. There are many ways to implement this, and the details are not particularly important, but it is perhaps useful to know that this is directly useful to address the motivation section described above. Given a currency type of PyVal (and a conforming implementation named PyRef), an implementation may look like:
>>>
>>> extension PyVal
>>> {
>>>
>>> subscript(dynamicMember member: String) ->
>>> PyVal {
>>>
>>> get
>>> {
>>>
>>> let result = PyObject_GetAttrString(borrowedPyObject, member)!
>>>
>>>
>>> return PyRef(owned: result)  // PyObject_GetAttrString returns +1 result.
>>>    }
>>>
>>> set
>>> {
>>>
>>> PyObject_SetAttrString
>>> (borrowedPyObject, member,
>>>                             newValue.
>>> toPython().borrowedPyObject
>>> )
>>>    }
>>>  }
>>> }
>>>
>>> Source compatibility
>>>
>>> This is a strictly additive proposal with no source breaking changes.
>>>
>>> Effect on ABI stability
>>>
>>> This is a strictly additive proposal with no ABI breaking changes.
>>>
>>> Effect on API resilience
>>>
>>> This has no impact on API resilience which is not already captured by other language features.
>>>
>>> Alternatives considered
>>>
>>> A few alternatives were considered:
>>>
>>> Add ability to provide read-only members
>>>
>>> The implementation above does not allow an implementation to statically reject members that are read-only. If this was important to add, we could add another protocol to model this, along the lines of:
>>>
>>> protocol DynamicMemberLookupGettableProtocol
>>> {
>>>
>>> associatedtype DynamicMemberLookupValue
>>>
>>>
>>>
>>> // gettable only
>>>  subscript(dynamicMember name: String) -> DynamicMemberLookupValue { get
>>> }
>>> }
>>>
>>>
>>> protocol DynamicMemberLookupProtocol : DynamicMemberLookupGettableProtocol
>>> {
>>>
>>> // gettable and settable.
>>>  subscript(dynamicMember name: String) -> DynamicMemberLookupValue { get set
>>> }
>>>  }
>>>
>>> This would allow a type to implement one or the other based on their capabilities. This proposal starts with a very simple design based on the requirements of dynamic languages (which have no apparent immutability model), but if there is demand for this (e.g. because we want input JSON values to be gettable but not settalbe), the author is happy to switch to this more general model.
>>>
>>> Naming
>>>
>>> There is a lot of grounds to debate naming of the protocol and methods in this type. Suggestions (along with rationale to support them) are more than welcome.
>>>
>>> _______________________________________________
>>> swift-evolution mailing list
>>> swift-evolution at swift.org
>>> https://lists.swift.org/mailman/listinfo/swift-evolution
>>
>>
>> --
>> Rick Mann
>> rmann at latencyzero.com
>>
>>
>> _______________________________________________
>> swift-evolution mailing list
>> swift-evolution at swift.org
>> https://lists.swift.org/mailman/listinfo/swift-evolution
>
> _______________________________________________
> swift-evolution mailing list
> swift-evolution at swift.org
> https://lists.swift.org/mailman/listinfo/swift-evolution

-- 
Alejandro Martinez
http://alejandromp.com