[swift-evolution] [Pitch] Introduce user-defined dynamically "callable" types
Howard Lovatt
howard.lovatt at gmail.com
Sat Nov 11 19:23:42 CST 2017
In Java you can have annotation processors, user supplied compiler extensions, that are extensively used for making frameworks easier to use and for general interfacing to ‘foreign stuff’.
Would a limited form of user supplied compiler extension be an alternative, an import processor. Special import syntax, import beginning with ‘Foreign’, would be recognised by the compiler and a user supplied import processor would be called, in the case of the example for Python. The import processor would generate the glue code. Below is a basic example to give a feel for what I am proposing.
(Syntax, names, etc. in code below just the 1st that came into my head!)
— Howard.
======================================================
import Foreign.Python.Foo // Special import tag `Foreign` calls a user supplied compiler extension that parses the imported Python and generates something along these lines (its up to the importer the exact code generated!):
protocol _ForeignPythonCallable {
func call(method: Any, arguments: Any...) throws -> Any
}
extension _ForeignPythonCallable {
func call(method: Any, arguments: Any...) throws -> Any {
return "method: \(method), arguments: \(arguments)" // Method lookup and calling code for Python.
}
}
class Foo: _ForeignPythonCallable {
// Could override `call` if more efficient than complete general lookup was possible.
enum _ForeignPythonMethodNames { // This might be a C struct so that it matches perfectly what the python interpreter is expecting.
case bar
}
enum _ForeignPythonMethodBarArguments { // This might be a C struct so that it matches perfectly what the python interpreter is expecting.
case x(Any)
case noName1(Any)
case y(Any)
}
func bar(x: Any, _ noName1: Any, y: Any) -> Any {
do {
return try call(method: _ForeignPythonMethodNames.bar, arguments: _ForeignPythonMethodBarArguments.x(x), _ForeignPythonMethodBarArguments.noName1(noName1), _ForeignPythonMethodBarArguments.y(y))
} catch {
fatalError("Method `bar` does not throw, therefore Python importer bug.")
}
}
}
// Then the consumer of `Foo` uses as per normal Swift:
let foo = Foo()
foo.bar(x: 1, 23, y: 17)
class TypedFoo: Foo { // For many dynamic languages this wouldn't be possible to automatically generate via the import, therefore would have to be hand written.
func bar(x: Int, _ noName1: Int, y: Int) -> String {
return super.bar(x: x, noName1, y: y) as! String
}
}
let typedFoo = TypedFoo()
typedFoo.bar(x: 1, 23, y: 17)
> On 12 Nov 2017, at 11:15 am, Andrew Bennett via swift-evolution <swift-evolution at swift.org> wrote:
>
> HI, this proposal looks really interesting!
>
> I have a few questions:
>
> Clarity on the proposal's intent
> Nice cheap bridges, or lowering barriers to bridging?
>
> I can see this providing a nice quick interface to Python from Swift, but I'm not sure if the exposed interface will be very Swifty (you probably have a much better idea of what is Swifty ;) than I do though). It seems you want it to be possible for everything to be dynamically exposed, I've used similar with Lua's meta methods, and I found it to be very powerful, you could basically implement inheritance in the language, which wasn't necessarily a good thing in retrospect.
>
> Is it common for the the argument labels in other languages to be open ended, or are labels typically finite? If the answer is finite, why not use a Swift method as the wrapper?
> Do you want duck typing, and would it be better to expose this via a protocol?
>
> It seems like in almost every case you could do something like this:
>
> func myMethod<X: PythonConvertible & CanQuack, Y: PythonConvertible>(a: X? = nil, b: Y) {
> pythonBridge.call("myMethod", arguments: ["a": X, "b": Y])
> }
>
> It might be good to add some use-cases (a popular Python library perhaps) to the proposal where this type of bridge would be insufficient :).
>
> It seems like this proposal pushes the responsibility of Swifty-ness and type-safety to the caller. At some point you'll have to write a type-safe bridging layer, or write your entire program in non-Swifty code ("The most obvious way to write code should also behave in a safe manner"). Is the main goal to lower the barrier to Python and other dynamic languages? or is it to provide a cheap nice Swifty bridge? I have the above concerns about the latter.
>
> Alternative sugar
>
> Ruby has Keyword Arguments for similar sugar:
>
> def foo(regular, hash={})
> puts "the hash: #{hash}"
>
>
> I'm sure you're aware of it, but I'll explain for completeness, any trailing argument labels are stored and passed as a hash:
>
> foo(regular, bar: "hello", bas: 123) # outputs 'the hash: [bar: "hello", bas: 123]'
> Have you considered an alternative like this? For example:
>
> func myMethod(regular: Int, store: @argcapture [String: PythonConvertible]) -> PythonConvertible
>
> I'm sure you have good reasons, it might make the implementation bleed out into other parts of the codebase. It would be good to include it in the proposal alternatives section though. At the moment most of the "alternatives" in the proposal just seem to be extensions to the same solution :)
>
> Clarity
> Perhaps just that things are more clear to me now
>
> If my extrapolation is correct a user will implement a single type that will allow a subset of a good portion of another language to be exposed (object method and property bridging). I'm guessing that the dynamic member proposal you're planning will not work with methods, it will require a property, I think this helps explain some of the motivations. It might be nice to have a more complete example that includes dynamic members. I didn't find it clear from the proposal that it would only be necessary to implement this protocol once per language.
>
> Thanks for considering my questions,
> Andrew Bennett
>
>
>> On Sat, Nov 11, 2017 at 4:37 AM, Chris Lattner via swift-evolution <swift-evolution at swift.org> wrote:
>> Hello all,
>>
>> I have a couple of proposals cooking in a quest to make Swift interoperate with dynamically typed languages like Python better. Instead of baking in hard coded support for one language or the other, I’m preferring to add a few small but general purpose capabilities to Swift. This is the first, which allows a Swift type to become “callable”.
>>
>> The proposal is here:
>> https://gist.github.com/lattner/a6257f425f55fe39fd6ac7a2354d693d
>>
>> I’ve also attached a snapshot below, but it will drift out of date as the proposal is refined. Feedback and thoughts are appreciated, thanks!
>>
>> -Chris
>>
>>
>>
>>
>>
>> Introduce user-defined dynamically "callable" types
>> Proposal: SE-NNNN
>> Author: Chris Lattner
>> Review Manager: TBD
>> Status: Awaiting implementation
>> Introduction
>>
>> This proposal introduces a new DynamicCallable protocol to the standard library. Types that conform to it are "callable" with the function call syntax. It is simple syntactic sugar which allows the user to write:
>>
>> a = someValue(keyword1: 42, "foo", keyword2: 19)
>> and have it be interpreted by the compiler as:
>>
>> a = someValue.dynamicCall(arguments: [
>> ("keyword1", 42), ("", "foo"), ("keyword2", 19)
>> ])
>> Other languages have analogous features (e.g. Python "callables"), but the primary motivation of this proposal is to allow elegant and natural interoperation with dynamic languages in Swift.
>>
>> Swift-evolution thread: Discussion thread topic for that proposal
>>
>> Motivation
>>
>> Swift is well known for being exceptional at interworking with existing C and Objective-C APIs, but its support for calling APIs written in scripting langauges like Python, Perl, and Ruby is quite lacking. These languages provide an extremely dynamic programming model where almost everything is discovered at runtime.
>>
>> Through the introduction of this proposal, and the related DynamicMemberLookupProtocol proposal, we seek to fix this problem. We believe we can make many common APIs feel very natural to use directly from Swift without all the complexity of implementing something like the Clang importer. For example, consider this Python code:
>>
>> class Dog:
>> def __init__(self, name):
>> self.name = name
>> self.tricks = [] # creates a new empty list for each dog
>>
>> def add_trick(self, trick):
>> self.tricks.append(trick)
>> we would like to be able to use this from Swift like this (the comments show the corresponding syntax you would use in Python):
>>
>> // import DogModule
>> // import DogModule.Dog as Dog // an alternate
>> let Dog = Python.import(“DogModule.Dog")
>>
>> // dog = Dog("Brianna")
>> let dog = Dog("Brianna")
>>
>> // dog.add_trick("Roll over")
>> dog.add_trick("Roll over")
>>
>> // dog2 = Dog("Kaylee").add_trick("snore")
>> let dog2 = Dog("Kaylee").add_trick("snore")
>> Of course, this would also apply to standard Python APIs as well. Here is an example working with the Python pickleAPI and the builtin Python function open:
>>
>> // import pickle
>> let pickle = Python.import("pickle")
>>
>> // file = open(filename)
>> let file = Python.open(filename)
>>
>> // blob = file.read()
>> let blob = file.read()
>>
>> // result = pickle.loads(blob)
>> let result = pickle.loads(blob)
>> This can all be expressed today as library functionality written in Swift, but without this proposal, the code required is unnecessarily verbose and gross. Without it (but with the related dynamic member lookup proposal) the code would have a method name (like call) all over the code:
>>
>> // import pickle
>> let pickle = Python.import("pickle") // normal method in Swift, no change.
>>
>> // file = open(filename)
>> let file = Python.open.call(filename)
>>
>> // blob = file.read()
>> let blob = file.read.call()
>>
>> // result = pickle.loads(blob)
>> let result = pickle.loads.call(blob)
>>
>> // dog2 = Dog("Kaylee").add_trick("snore")
>> let dog2 = Dog.call("Kaylee").add_trick.call("snore")
>> While this is a syntactic sugar proposal, we believe that this expands Swift to be usable in important new domains. This sort of capability is also highly precedented in other languages, and is a generally useful language feature that could be used for other purposes as well.
>>
>> Proposed solution
>>
>> We propose introducing this protocol to the standard library:
>>
>> protocol DynamicCallable {
>> associatedtype DynamicCallableArgument
>> associatedtype DynamicCallableResult
>>
>> func dynamicCall(arguments: [(String, DynamicCallableArgument)]) throws -> DynamicCallableResult
>> }
>> It also extends the language such that function call syntax - when applied to a value of DynamicCallable type - is accepted and transformed into a call to the dynamicCall member. The dynamicCall method takes a list of tuples: the first element is the keyword label (or an empty string if absent) and the second value is the formal parameter specified at the call site.
>>
>> Before this proposal, the Swift language has two types that participate in call syntax: functions and metatypes (for initialization). Neither of those may conform to protocols at the moment, so this introduces no possible ambiguity into the language.
>>
>> It is worth noting that this does not introduce the ability to provide dynamicly callable static/class members. We don't believe that this is important given the goal of supporting dynamic languages like Python, but if there is a usecase discovered in the future, it could be explored as future work. Such future work should keep in mind that call syntax on metatypes is already meaningful, and that ambiguity would have to be resolved somehow.
>>
>> Discussion
>>
>> While the signature for dynamicCall is highly general we expect the most common use will be clients who are programming against concrete types that implement this proposal. One very nice aspect of this is that, as a result of Swift's existing subtyping mechanics, implementations of this type can choose whether they can actually throw an error or not. For example, consider this silly implementation:
>>
>> struct ParameterSummer : DynamicCallable {
>> func dynamicCall(arguments: [(String, Int)]) -> Int {
>> return arguments.reduce(0) { $0+$1.1 }
>> }
>> }
>>
>> let x = ParameterSummer()
>> print(x(1, 7, 12)) // prints 20
>> Because ParameterSummer's implementation of dynamicCall does not throw, the call site is known not to throw either, so the print doesn't need to be marked with try.
>>
>> Example Usage
>>
>> A more realistic (and motivating) example comes from a prototype Python interop layer. While the concrete details of this use case are subject to change and not important for this proposal, it is perhaps useful to have a concrete example to see how this comes together.
>>
>> That prototype currently has two types which model Python values, one of which handles Python exceptions and one of which does not. Their conformances would look like this, enabling the use cases described in the Motivation section above:
>>
>> extension ThrowingPyRef: DynamicCallable {
>> func dynamicCall(arguments: [(String, PythonConvertible)]) throws
>> -> PythonConvertible {
>> // Make sure state errors are not around.
>> assert(PyErr_Occurred() == nil, "Python threw an error but wasn't handled")
>>
>> // Count how many keyword arguments are in the list.
>> let numKeywords = arguments.reduce(0) {
>> $0 + ($1.0.isEmpty ? 0 : 1)
>> }
>>
>> let kwdict = numKeywords != 0 ? PyDict_New() : nil
>>
>> // Non-keyword arguments are passed as a tuple of values.
>> let argTuple = PyTuple_New(arguments.count-numKeywords)!
>> var nonKeywordIndex = 0
>> for (keyword, argValue) in arguments {
>> if keyword.isEmpty {
>> PyTuple_SetItem(argTuple, nonKeywordIndex, argValue.toPython())
>> nonKeywordIndex += 1
>> } else {
>> PyDict_SetItem(kwdict!, keyword.toPython(), argValue.toPython())
>> }
>> }
>>
>> // Python calls always return a non-null value when successful. If the
>> // Python function produces the equivalent of C "void", it returns the None
>> // value. A null result of PyObjectCall happens when there is an error,
>> // like 'self' not being a Python callable.
>> guard let resultPtr = PyObject_Call(state, argTuple, kwdict) else {
>> throw PythonError.invalidCall(self)
>> }
>>
>> let result = PyRef(owned: resultPtr)
>>
>> // Translate a Python exception into a Swift error if one was thrown.
>> if let exception = PyErr_Occurred() {
>> PyErr_Clear()
>> throw PythonError.exception(PyRef(borrowed: exception))
>> }
>>
>> return result
>> }
>> }
>>
>> extension PyRef: DynamicCallable {
>> func dynamicCall(arguments: [(String, PythonConvertible)])
>> -> PythonConvertible {
>> // Same as above, but internally aborts instead of throwing Swift
>> // errors.
>> }
>> }
>> Source compatibility
>>
>> This is a strictly additive proposal with no source breaking changes.
>>
>> Effect on ABI stability
>>
>> This is a strictly additive proposal with no ABI breaking changes.
>>
>> Effect on API resilience
>>
>> This has no impact on API resilience which is not already captured by other language features.
>>
>> Alternatives considered
>>
>> A few alternatives were considered:
>>
>> Add ability to reject parameter labels
>>
>> The implementation above does not allow an implementation to staticly reject argument labels. If this was important to add, we could add another protocol to model this, along the lines of:
>>
>> /// A type conforming just to this protocol would not accept parameter
>> /// labels in its calls.
>> protocol DynamicCallable {
>> associatedtype DynamicCallableArgument
>> associatedtype DynamicCallableResult
>>
>> func dynamicCall(arguments: [DynamicCallableArgument]) throws -> DynamicCallableResult
>> }
>>
>> /// A type conforming to this protocol does allow optional parameter
>> /// labels.
>> protocol DynamicCallableWithKeywordsToo : DynamicCallable {
>> func dynamicCall(arguments: [(String, DynamicCallableArgument)]) throws -> DynamicCallableResult
>> }
>> This would allow a type to implement one or the other based on their capabilities. This proposal is going with a very simple design, but if there is demand for this, the author is happy to switch.
>>
>> Staticly checking for exact signatures
>>
>> This protocol does not allow a type to specify an exact signature for the callable - a specific number of parameters with specific types. If we went down that route, the best approach would be to introduce a new declaration kind (which would end up being very similar to get-only subscripts) since, in general, a type could want multiple concrete callable signatures, and those signatures should participate in overload resolution.
>>
>> While such a feature could be interesting for some use cases, it is almost entirely orthogonal from this proposal: it addresses different use cases and does not solve the needs of this proposal. It does not address our needs because even a variadic callable declaration would not provide access to the keyword argument labels we need.
>>
>> Direct language support for Python
>>
>> We considered implementing something analogous to the Clang importer for Python, which would add a first class Python specific type(s) to Swift language or standard library. We rejected this option because it would be significantly more invasive in the compiler, would set the precedent for all other dynamic languages to get first class language support, and because that first class support doesn't substantially improve the experience of working with Python over existing Swift with a couple small "generally useful" extensions like this one.
>>
>> Naming
>>
>> The most fertile ground for bikeshedding is the naming of the protocol and the members. We welcome other ideas and suggestions for naming, but here are some thoughts on obvious options to consider:
>>
>> We considered but rejected the name CustomCallable, because the existing Custom* protocols in the standard library (CustomStringConvertible, CustomReflectable, etc) provide a way to override and custom existing builtin abilities of Swift. In contrast, this feature grants a new capability to a type.
>>
>> We considered but rejected a name like ExpressibleByCalling to fit with the ExpressibleBy* family of protocols (like ExpressibleByFloatLiteral, ExpressibleByStringLiteral, etc). This name family is specifically used by literal syntax, and calls are not literals. Additionally the type itself is not "expressible by calling" - instead, instances of the type may be called.
>>
>> On member and associated type naming, we intentionally gave these long and verbose names so they stay out of the way of user code completion. The members of this protocol are really just compiler interoperability glue. If there was a Swift attribute to disable the members from showing up in code completion, we would use it (such an attribute would also be useful for the LiteralConvertible and other compiler magic protocols).
>>
>>
>>
>> _______________________________________________
>> swift-evolution mailing list
>> swift-evolution at swift.org
>> https://lists.swift.org/mailman/listinfo/swift-evolution
>>
>
> _______________________________________________
> swift-evolution mailing list
> swift-evolution at swift.org
> https://lists.swift.org/mailman/listinfo/swift-evolution
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-evolution/attachments/20171112/002855d6/attachment.html>
More information about the swift-evolution
mailing list