[swift-evolution] DynamicMemberLookup proposal: status update
Nevin Brackett-Rozinsky
nevin.brackettrozinsky at gmail.com
Thu Jan 4 17:43:37 CST 2018
There’s a lot of information here and it’ll take some time to process it
all. My initial reaction is that a “strong type-alias” feature might help.
If one could write (strawman syntax):
strong typealias Dog = PyVal // A semantically independent new type
extension Dog {
// Declarations here are only available on “Dog”, not on “PyVal”
}
then most of the overload issues would evaporate.
Nevin
On Thu, Jan 4, 2018 at 3:52 PM, Chris Lattner via swift-evolution <
swift-evolution at swift.org> wrote:
> Hi everyone,
>
> With the holidays and many other things behind us, the core team had a
> chance to talk about python interop + the dynamic member lookup proposal
> recently.
>
> Here’s where things stand: we specifically discussed whether a
> counter-proposal of using “automatically generated wrappers” or “foreign
> classes” to solve the problem would be better. After discussion, the
> conclusion is no: the best approach appears to be DynamicMemberLookup/DynamicCallable
> or something similar in spirit to them. As such, I’ll be dusting off the
> proposal and we’ll eventually run it.
>
> For transparency, I’m attaching the analysis below of what a wrapper
> facility could look like, and why it doesn’t work very well for Python
> interop. I appologize in advance that this is sort of train-of-thought and
> not a well written doc.
>
> That said, it would be really great to get tighter integration between
> Swift and SwiftPM for other purposes! I don’t have time to push this
> forward in the short term though, but if someone was interested in pushing
> it forward, many people would love to see it discussed seriously.
>
> -Chris
>
>
> *A Swift automatic wrapper facility:*
>
> Requirements:
> - We want the be able to run a user defined script to generate wrappers.
> - This script can have arbitrary dependencies and should get updated when
> one of them change.
> - These dependencies won’t be visible to the Xcode build system, so the
> compiler will have to manage them.
> - In principle, one set of wrappers should be able to depend on another
> set, and wants “overlays”, so we need a pretty general model.
>
> I don’t think the clang modules based approach is a good way to go.
>
>
> *Proposed Approach: Tighter integration between SwiftPM and Swift*
>
> The model is that you should be able to say (strawman syntax):
>
> import Foo from http://github.com/whatever/mypackage
> import Bar from file:///some/path/on/my/machine
>
> and have the compiler ask SwiftPM to build and cache the specified module
> onto your local disk, then have the compiler load it like any other
> module. This means that “generated wrappers” is now a SwiftPM/llbuild
> feature, and we can use the SwiftPM “language” to describe things like:
>
> 1. Information about what command line invocation is required to generate
> the wrappers.
> 2. Dependency information so that the compiler can regenerate the wrappers
> when they are out of date.
> 3. Platform abstraction tools since things are in different locations on
> linux vs mac, Python 2 vs Python 3 is also something that would have to be
> handled somehow.
> 4. The directory could contain manually written .swift code, serving the
> function similar to “overlays” to augment the automatic wrappers generated.
>
> We care about Playgrounds and the REPL, and they should be able to work
> with this model.
>
> I think that this would be a very nice and useful feature.
>
>
> *Using Wrappers to implement Python Interop:*
>
> While such a thing would be generally useful, it is important to explore
> how well this will work to solve the actual problem at hand, since this is
> being pitched as an alternative to DynamicMemberLookup. Here is the
> example from the list:
>
> class BankAccount:
> def __init__(self, initial_balance: int = 0) -> None:
> self.balance = initial_balance
> def deposit(self, amount: int) -> None:
> self.balance += amount
> def withdraw(self, amount: int) -> None:
> self.balance -= amount
> def overdrawn(self) -> bool:
> return self.balance < 0
>
> my_account = BankAccount(15)
> my_account.withdraw(5)print(my_account.balance)
>
>
> The idea is to generate a wrapper like this (potentially including the
> type annotations as a refinement):
>
> typealias BankAccount = PyVal
> extension PyVal { // methods on BankAccount
> init(initial_balance: PyVal) { … }
> func deposit(amount: PyVal) -> PyVal { … }
> func withdraw(amount: PyVal) -> PyVal { … }
> func overdrawn() -> PyVal { … }
> }
>
> my_account = BankAccount(initial_balance: 15)
> my_account.withdraw(amount: 5)
> print(my_account.balance)
>
>
>
> It is worth pointing out that this approach is very analogous to the “type
> providers” feature that Joe pushed hard for months ago, it is just a
> different implementation approach. The proposal specifically explains why
> this isn’t a great solution here:
> https://gist.github.com/lattner/b016e1cf86c43732c8d82f90e5ae54
> 38#introduce-f-style-type-providers-into-swift
>
> That said, while there are similarities, there are also differences with
> type providers. Here are the problems that I foresee:
>
>
> *1) This design still requires DynamicMemberLookup*
>
> This is because Python doesn’t have property declarations for the wrapper
> generator to process. The code above shows this on the last line: since
> there is no definition of the “balance" property, there will be no “balance
> member” declared in the PyVal extension. A wrapper generator can generate
> a decl for something it can’t “see”. You can see this in a simpler example:
>
> class Car(object):
> def __init__(self):
> self.speed = 100
>
>
> We really do want code like this to work:
>
> let mini = Car()
> print(mini.speed)
>
>
> How will the wrapper generator produce a decl for ‘speed’ when no decl
> exists? Doug agreed through offline email that "we’d need to fall back to
> foo[dynamicMember: “speed”] or something like DynamicMemberLookup.” to
> handle properties.
>
>
> *2) **Dumping properties and methods into the same scope won’t work*
>
> I didn’t expect this, and I’m not sure how this works, but apparently we
> accept this:
>
> struct S {
> var x: Int
> func x(_ a: Int) -> Int { return a }
> }
>
>
> let x = S(x: 1)
> x.x // returns 1, not a curried method. Bug or feature?
> x.x(42) // returns 42
>
>
> That said, we reject this:
>
> struct S {
> var x: Int
> func x() -> Int { ... } // invalid redeclaration of x
> }
>
> which means that we’re going to have problems if we find a way to generate
> property decls (e.g. in response to @property in Python).
>
> Even if we didn’t, we’d still have a problem if we wanted to incorporate
> types because we reject this:
>
> struct S {
> var x: Int
> var x: Float // invalid redeclaration of X.
> }
>
>
> This means that even if we have property declarations (e.g. due to use of
> the Python @property marker for computed properties) we cannot actually
> incorporate type information into the synthesized header, because multiple
> classes have all their members munged together and will conflict.
>
> Further, types in methods can lead to ambiguities too in some cases, e.g.
> if you have:
>
> extension PyVal { // Class Dog
> func f(a: Float) -> Float { ... }
> }
> extension PyVal { // Class Cat
> func f(a: Int) {}
> }
>
> print(myDog.f(a: 1))
>
>
> This compiles just fine, but prints out “()" instead of the result of your
> Dog method, because we select the second overload. In other cases, I
> suspect you’d fail to compile due to ambiguities. This is a case where
> types are really harmful for Python, and one of the reasons that the Python
> types do not affect runtime behavior at all.
>
>
> *3) It’s unclear how to incorporate initializers into this model.*
>
> The example above included this as suggested. It looks nice on the
> surface, but presents some problems:
>
> typealias BankAccount = PyVal
> extension PyVal { // methods on BankAccount
> init(initial_balance: Int) { … }
> }
>
> This has a couple of problems. First of all, in Python classes are
> themselves callable values, and this break that. Second this mooshes all
> of the initializers for all of the Python classes onto PyVal.
>
> While it might seem that this would make code completion for initializers
> ugly, this isn’t actually a problem. After all, we’re enhancing code
> completion to know about PyVal already, so we can clearly use this trivial
> local information to filter the list down.
>
> The actual problem is that multiple Python classes can have the same
> initializer, and we have no way to set the ‘self’ correctly in this case.
> Consider:
>
> class BankAccount:
> def __init__(self, initial_balance: int = 0) -> None: …
>
> class Gyroscope:
> def __init__(self, initial_balance: int = 0) -> None: …
>
> These will require generating one extension:
>
> extension PyVal { // methods on BankAccount
> init(initial_balance: Int) {
> self.init( /* what Python class do we allocate and pass here?
> BankAccount or Gyroscope? */ )
> }
> }
>
> I can think of a couple of solutions to this problem, I’d appreciate
> suggestions for other ones:
>
> *3A) Classes turn into typealiases, initializers get name mangled:*
>
> something like:
>
> typealias BankAccount = PyVal
> extension PyVal {
> init(BankAccount_initial_balance: PyVal) { … }
> }
> ...
> let my_account = BankAccount(BankAccount_initial_balance: 15)
>
> I don’t like this: this is ugly for clients, and BankAccount is still
> itself not a value.
>
>
> *3B) Classes turn into global functions:*
>
> something like:
>
> func BankAccount(initial_balance: PyVal) -> PyVal {… }
> ...
> let my_account = BankAccount(initial_balance: 15)
>
> This makes the common cases work, but breaks currying and just seems weird
> to me. Maybe this is ok? This also opens the door for:
>
> // type annotation for clarity only.
>
> let BankAccount: PyVal = BankingModule.BankAccount
>
> which makes “BankAccount” itself be a value (looked up somehow on the
> module it is defined in).
>
> *3C) Classes turn into types that are callable with subscripts?*
>
> We don’t actually support static subscripts right now (seems like a silly
> limitation…):
>
> struct BankAccount {
> // error, subscripts can’t be static.
> static subscript(initial_balance initial_balance: PyVal) -> PyVal { … }
> }
> …
> let my_account = BankAccount[initial_balance: 15]
>
>
> But we could emulate them with:
>
> struct BankAccountType {
> subscript(initial_balance initial_balance: PyVal) -> PyVal { … }
> }
> var BankAccount: BankAccountType { return BankAccountType() }
> …
> let my_account = BankAccount[initial_balance: 15]
>
>
> this could work, but the square brackets are “gross”, and BankAccount is
> not a useful Python metatype. We could improve the call-side syntax by
> introducing a “call decl” or a new “operator()” language feature like:
>
> extension BankAccountType {
> func () (initial_balance: PyVal) -> PyVal { … }
> }
> var BankAccount: BankAccountType { return BankAccountType() }
> …
> let my_account = BankAccount(initial_balance: 15)
>
>
> but this still doesn’t solve the “BankAccount as a python value” problem.
>
> *4) Throwing / Failability of APIs*
>
> Python, like many languages, doesn’t require you to specify whether APIs
> can throw or not, and because it is dynamically typed, member lookup itself
> is failable. That said, just like in C++, it is pretty uncommon for
> exceptions to be thrown from APIs, and our Error Handling design wants
> Swift programmers to think about error handling, not just slap try on
> everything.
>
> The Python interop prototype handles this by doing this:
>
> extension PyVal {
> /// Return a version of this value that may be called. It throws a
> Swift
> /// error if the underlying Python function throws a Python exception.
> public var throwing : ThrowingPyVal {
> return ThrowingPyVal(self)
> }
> }
>
> .. and both PyVal and ThrowingPyVal are callable. The implementation of
> DynamicCallable on ThrowingPyVal throws a Swift error but the
> implementation on PyVal does not (similar mechanic exists for
> DynamicMemberLookup).
>
> This leads to a nice model: the PyVal currency type never throws a Swift
> error (it aborts on exception) so users don’t have to use “try” on
> everything. However, if they *want* to process an error, they can. Here is
> an example from the tutorial:
>
> do {
> let x = *try* Python.open.throwing("/file/that/doesnt/exist")
> print(x)
> } catch let err {
> print("file not found, just like we expected!")
>
> // Here is the error we got:
> print(err)
> }
>
> In practice, this works really nicely with the way that “try” is required
> for throwing values, but produces a warning when you use it unnecessarily.
>
> Coming back to wrappers, it isn’t clear how to handle this: a direct port
> of this would require synthesizing all members onto both PyVal
> and ThrowingPyVal. This causes tons of bloat and undermines the goal of
> making the generated header nice.
>
> -Chris
>
>
>
> _______________________________________________
> swift-evolution mailing list
> swift-evolution at swift.org
> https://lists.swift.org/mailman/listinfo/swift-evolution
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-evolution/attachments/20180104/624dddb4/attachment.html>
More information about the swift-evolution
mailing list