[swift-evolution] [Proposal] Foundation Swift Archival & Serialization

[Matthew Johnson]

> On Mar 17, 2017, at 1:15 PM, Itai Ferber via swift-evolution <swift-evolution at swift.org> wrote:
> 
> On 15 Mar 2017, at 22:58, Zach Waldowski wrote:
> 
> 
> Another issue of scale - I had to switch to a native mail client as replying inline severely broke my webmail client. ;-)
> 
> Again, lots of love here. Responses inline.
> On Mar 15, 2017, at 6:40 PM, Itai Ferber via swift-evolution <swift-evolution at swift.org> wrote:
> Proposed solution
> We will be introducing the following new types:
> 
> protocol Codable: Adopted by types to opt into archival. Conformance may be automatically derived in cases where all properties are also Codable.
> 
> FWIW I think this is acceptable compromise. If the happy path is derived conformances, only-decodable or only-encodable types feel like a lazy way out on the part of a user of the API, and builds a barrier to proper testing.
> 
> [snip]
> 
> Structured types (i.e. types which encode as a collection of properties) encode and decode their properties in a keyed manner. Keys may be String-convertible or Int-convertible (or both), and user types which have properties should declare semantic key enums which map keys to their properties. Keys must conform to the CodingKey protocol:
> public protocol CodingKey { <##snip##> }
> 
> A few things here:
> 
> The protocol leaves open the possibility of having both a String or Int representation, or neither. What should a coder do in either case? Are the representations intended to be mutually exclusive, or not? The protocol design doesn’t seem particularly matching with the flavor of Swift; I’d expect something along the lines of a CodingKey enum and the protocol CodingKeyRepresentable. It’s also possible that the concerns of the two are orthogonal enough that they deserve separate container(keyedBy:) requirements.
> 
> The general answer to "what should a coder do" is "what is appropriate for its format". For a format that uses exclusively string keys (like JSON), the string representation (if present on a key) will always be used. If the key has no string representation but does have an integer representation, the encoder may choose to stringify the integer. If the key has neither, it is appropriate for the Encoder to fail in some way.
> 
> On the flip side, for totally flat formats, an Encoder may choose to ignore keys altogether, in which case it doesn’t really matter. The choice is up to the Encoder and its format.
> 
> The string and integer representations are not meant to be mutually exclusive at all, and in fact, where relevant, we encourage providing both types of representations for flexibility.
> 
> As for the possibility of having neither representation, this question comes up often. I’d like to summarize the thought process here by quoting some earlier review (apologies for the poor formatting from my mail client):
> 
> 
> If there are two options, each of which is itself optional, we have 4 possible combinations. But! At the same time we prohibit one combination by what? Runtime error? Why not use a 3-case enum for it? Even further down the rabbit whole there might be a CodingKey<> specialized for a concrete combination, like CodingKey<StringAndIntKey> or just CodingKey<StringKey>, but I’m not sure whether our type system will make it useful or possible…
> 
> public enum CodingKeyValue {
> case integer(value: Int)
> case string(value: String)
> case both(intValue: Int, stringValue: String)
> }
> public protocol CodingKey {
> init?(value: CodingKeyValue)
> var value: CodingKeyValue { get }
> }
> 
> I agree that this certainly feels suboptimal. We’ve certainly explored other possibilities before sticking to this one, so let me try to summarize here:
> 
> * Having a concrete 3-case CodingKey enum would preclude the possibility of having neither a stringValue nor an intValue. However, there is a lot of value in having the key types belong to the type being encoded (more safety, impossible to accidentally mix key types, private keys, etc.); if the CodingKey type itself is an enum (which cannot be inherited from), then this prevents differing key types.
> * Your solution as presented is better: CodingKey itself is still a protocol, and the value itself is the 3-case enum. However, since CodingKeyValue is not literal-representable, user keys cannot be enums RawRepresentable by CodingKeyValue. That means that the values must either be dynamically returned, or (for attaining the benefits that we want to give users — easy representation, autocompletion, etc.) the type has to be a struct with static lets on it giving the CodingKeyValues. This certainly works, but is likely not what a developer would have in mind when working with the API; the power of enums in Swift makes them very easy to reach for, and I’m thinking most users would expect their keys to be enums. We’d like to leverage that where we can, especially since RawRepresentable enums are appropriate in the vast majority of use cases.
> * Three separate CodingKey protocols (one for Strings, one for Ints, and one for both). You could argue that this is the most correct version, since it most clearly represents what we’re looking for. However, this means that every method now accepting a CodingKey must be converted into 3 overloads each accepting different types. This explodes the API surface, is confusing for users, and also makes it impossible to use CodingKey as an existential (unless it’s an empty 4th protocol which makes no static guarantees and the others inherit from).
> * [The current] approach. On the one hand, this allows for the accidental representation of a key with neither a stringValue nor an intValue. On the other, we want to make it really easy to use autogenerated keys, or autogenerated key implementations if you provide the cases and values yourself. The nil value possibility is only a concern when writing stringValue and intValue yourself, which the vast majority of users should not have to do.
> * Additionally, a key word in that sentence bolded above is “generally”. As part of making this API more generalized, we push a lot of decisions to Encoders and Decoders. For many formats, it’s true that having a key with no value is an error, but this is not necessarily true for all formats; for a linear, non-keyed format, it is entirely reasonable to ignore the keys in the first place, or replaced them with fixed-format values. The decision of how to handle this case is left up to Encoders and Decoders; for most formats (and for our implementations), this is certainly an error, and we would likely document this and either throw or preconditionFailure. But this is not the case always.
> * In terms of syntax, there’s another approach that would be really nice (but is currently not feasible) — if enums were RawRepresentable in terms of tuples, it would be possible to give implementations for String, Int, (Int, String), (String, Int), etc., making this condition harder to represent by default unless you really mean to.
> 
> Hope that gives some helpful background on this decision. FWIW, the only way to end up with a key having no intValue or stringValue is manually implementing the CodingKey protocol (which should be exceedingly rare) and implementing the methods by not switching on self, or some other method that would allow you to forget to give a key neither value.
> 
> 
> Speaking of the mutually exclusive representations - what above serializations that doesn’t code as one of those two things? YAML can have anything be a “key”, and despite that being not particularly sane, it is a use case.
> 
> We’ve explored this, but at the end of the day, it’s not possible to generalize this to the point where we could represent all possible options on all possible formats because you cannot make any promises as to what’s possible and what’s not statically.
> 
> We’d like to strike a balance here between strong static guarantees on one end (the extreme end of which introduces a new API for every single format, since you can almost perfectly statically express what’s possible and what isn’) and generalization on the other (the extreme end of which is an empty protocol because there really are encoding formats which are mutually exclusive). So in this case, this API would support producing and consuming YAML with string or integer keys, but not arbitrary YAML.
> 
> 
> For most types, String-convertible keys are a reasonable default; for performance, however, Int-convertible keys are preferred, and Encoders may choose to make use of Ints over Strings. Framework types should provide keys which have both for flexibility and performance across different types of Encoders. It is generally an error to provide a key which has neither a stringValue nor an intValue.
> Could you speak a little more to using Int-convertible keys for performance? I get the feeling int-based keys parallel the legacy of NSCoder’s older design, and I don’t really see anyone these days supporting non-keyed archivers. They strike me as fragile. What other use cases are envisioned for ordered archiving than that?
> 
> We agree that integer keys are fragile, and from years (decades) of experience with NSArchiver, we are aware of the limitations that such encoding offers. For this reason, we will never synthesize integer keys on your behalf. This is something you must put thought into, if using an integer key for archival.
> 
> However, there are use-cases (both in archival and in serialization, but especially so in serialization) where integer keys are useful. Ordered encoding is one such possibility (when the format supports it, integer keys are sequential, etc.), and is helpful for, say, marshaling objects in an XPC context (where both sides are aware of the format, are running the same version of the same code, on the same device) — keys waste time and bandwidth unnecessarily in some cases.
> 
> Integer keys don’t necessarily imply ordered encoding, however. There are binary encoding formats which support integer-keyed dictionaries (read: serialized hash maps) which are more efficient to encode and decode than similar string-keyed ones. In that case, as long as integer keys are chosen with care, the end result is more performant.
> 
> But again, this depends on the application and use case. Defining integer keys requires manual effort because we want thought put into defining them; they are indeed fragile when used carelessly.
> 
> 
> [snip]
> 
> Keyed Encoding Containers
> 
> Keyed encoding containers are the primary interface that most Codable types interact with for encoding and decoding. Through these, Codable types have strongly-keyed access to encoded data by using keys that are semantically correct for the operations they want to express.
> 
> Since semantically incompatible keys will rarely (if ever) share the same key type, it is impossible to mix up key types within the same container (as is possible with Stringkeys), and since the type is known statically, keys get autocompletion by the compiler.
> 
> open class KeyedEncodingContainer<Key : CodingKey> {
> 
> Like others, I’m a little bummed about this part of the design. Your reasoning up-thread is sound, but I chafe a bit on having to reabstract and a little more on having to be a reference type. Particularly knowing that it’s got a bit more overhead involved… I /like/ that NSKeyedArchiver can simply push some state and pass itself as the next encoding container down the stack.
> 
> There’s not much more to be said about why this is a class that I haven’t covered; if it were possible to do otherwise at the moment, then we would.
> 
It is possible using a manually written type-erased wrapper along the lines of AnySequence and AnyCollection.  I don’t recall seeing a rationale for why you don’t want to go this route.  I would still like to hear more on this topic.

> As for why we do this — this is the crux of the whole API. We not only want to make it easy to use a custom key type that is semantically correct for your type, we want to make it difficult to do the easy but incorrect thing. From experience with NSKeyedArchiver, we’d like to move away from unadorned string (and integer) keys, where typos and accidentally reused keys are common, and impossible to catch statically.
> encode<T : Codable>(_: T?, forKey: String) unfortunately not only encourages code like encode(foo, forKey: "foi") // whoops, typo, it is more difficult to use a semantic key type: encode(foo, forKey: CodingKeys.foo.stringValue). The additional typing and lack of autocompletion makes it an active disincentive. encode<T : Codable>(_: T?, forKey: Key) reverses both of these — it makes it impossible to use unadorned strings or accidentally use keys from another type, and nets shorter code with autocompletion: encode(foo, forKey: .foo)
> 
> The side effect of this being the fact that keyed containers are classes is suboptimal, I agree, but necessary.
> 
> 
> 
> open func encode<Value : Codable>(_ value: Value?, forKey key: Key) throws
> 
> Does this win anything over taking a Codable?
> 
> Taking the concrete type over an existential allows for static dispatch on the type within the implementation, and is a performance win in some cases.
> 
> 
> open func encode(_ value: Bool?, forKey key: Key) throws
> open func encode(_ value: Int?, forKey key: Key) throws
> open func encode(_ value: Int8?, forKey key: Key) throws
> open func encode(_ value: Int16?, forKey key: Key) throws
> open func encode(_ value: Int32?, forKey key: Key) throws
> open func encode(_ value: Int64?, forKey key: Key) throws
> open func encode(_ value: UInt?, forKey key: Key) throws
> open func encode(_ value: UInt8?, forKey key: Key) throws
> open func encode(_ value: UInt16?, forKey key: Key) throws
> open func encode(_ value: UInt32?, forKey key: Key) throws
> open func encode(_ value: UInt64?, forKey key: Key) throws
> open func encode(_ value: Float?, forKey key: Key) throws
> open func encode(_ value: Double?, forKey key: Key) throws
> open func encode(_ value: String?, forKey key: Key) throws
> open func encode(_ value: Data?, forKey key: Key) throws
> 
> What is the motivation behind abandoning the idea of “primitives” from the Alternatives Considered? Performance? Being unable to close the protocol?
> 
> Being unable to close the protocol is the primary reason. Not being able to tell at a glance what the concrete types belonging to this set are is related, and also a top reason.
> 
Looks like we have another strong motivating use case for closed protocols.  I hope that will be in scope for Swift 5.  

It would be great for the auto-generated documentation and “headers" to provide a list of all public or open types inheriting from a closed class or conforming to a closed protocol (when we get them).  This would go a long way towards addressing your second reason.

> 
> What ways is encoding a value envisioned to fail? I understand wanting to allow maximum flexibility, and being symmetric to `decode` throwing, but there are plenty of “conversion” patterns the are asymmetric in the ways they can fail (Date formatters, RawRepresentable, LosslessStringConvertible, etc.).
> 
> Different formats support different concrete values, even of primitive types. For instance, you cannot natively encode Double.nan in JSON, but you can in plist. Without additional options on JSONEncoder, encode(Double.nan, forKey: …) will throw.
> 
> 
> /// For `Encoder`s that implement this functionality, this will only encode the given object and associate it with the given key if it encoded unconditionally elsewhere in the archive (either previously or in the future).
> open func encodeWeak<Object : AnyObject & Codable>(_ object: Object?, forKey key: Key) throws
> 
> Is this correct that if I send a Cocoa-style object graph (with weak backrefs), an encoder could infinitely recurse? Or is a coder supposed to detect that?
> 
> encodeWeak has a default implementation that calls the regular encode<T : Codable>(_: T, forKey: Key); only formats which actually support weak backreferencing should override this implementation, so it should always be safe to call (it will simply unconditionally encode the object by default).
> 
> 
> open var codingKeyContext: [CodingKey]
> }
> [snippity snip]
> Alright, those are just my first thoughts. I want to spend a little time marinating in the code from PR #8124 before I comment further. Cheers! I owe you, Michael, and Tony a few drinks for sure.
> 
> Hehe, thanks :)
> 
> 
> Zach Waldowski
> zach at waldowski.me
> 
> _______________________________________________
> swift-evolution mailing list
> swift-evolution at swift.org
> https://lists.swift.org/mailman/listinfo/swift-evolution

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-evolution/attachments/20170317/f85504b1/attachment.html>