[swift-evolution] [Proposal] Foundation Swift Archival & Serialization

[Itai Ferber]

On 15 Mar 2017, at 22:58, Zach Waldowski wrote:

> Another issue of scale - I had to switch to a native mail client as 
> replying inline severely broke my webmail client. ;-)
>
> Again, lots of love here. Responses inline.
>
>> On Mar 15, 2017, at 6:40 PM, Itai Ferber via swift-evolution 
>> <swift-evolution at swift.org> wrote:
>> Proposed solution
>> We will be introducing the following new types:
>>
>> protocol Codable: Adopted by types to opt into archival. Conformance 
>> may be automatically derived in cases where all properties are also 
>> Codable.
> FWIW I think this is acceptable compromise. If the happy path is 
> derived conformances, only-decodable or only-encodable types feel like 
> a lazy way out on the part of a user of the API, and builds a barrier 
> to proper testing.
>> [snip]
>>
>> Structured types (i.e. types which encode as a collection of 
>> properties) encode and decode their properties in a keyed manner. 
>> Keys may be String-convertible or Int-convertible (or both), and user 
>> types which have properties should declare semantic key enums which 
>> map keys to their properties. Keys must conform to the CodingKey 
>> protocol:
>> public protocol CodingKey { <##snip##> }
>
> A few things here:
>
> The protocol leaves open the possibility of having both a String or 
> Int representation, or neither. What should a coder do in either case? 
> Are the representations intended to be mutually exclusive, or not? The 
> protocol design doesn’t seem particularly matching with the flavor 
> of Swift; I’d expect something along the lines of a CodingKey enum 
> and the protocol CodingKeyRepresentable. It’s also possible that the 
> concerns of the two are orthogonal enough that they deserve separate 
> container(keyedBy:) requirements.
The general answer to "what should a coder do" is "what is appropriate 
for its format". For a format that uses exclusively string keys (like 
JSON), the string representation (if present on a key) will always be 
used. If the key has no string representation but does have an integer 
representation, the encoder may choose to stringify the integer. If the 
key has neither, it is appropriate for the `Encoder` to fail in some 
way.

On the flip side, for totally flat formats, an `Encoder` may choose to 
ignore keys altogether, in which case it doesn’t really matter. The 
choice is up to the `Encoder` and its format.

The string and integer representations are not meant to be mutually 
exclusive at all, and in fact, where relevant, we encourage providing 
both types of representations for flexibility.

As for the possibility of having neither representation, this question 
comes up often. I’d like to summarize the thought process here by 
quoting some earlier review (apologies for the poor formatting from my 
mail client):

>> If there are two options, each of which is itself optional, we have 4 
>> possible combinations. But! At the same time we prohibit one 
>> combination by what? Runtime error? Why not use a 3-case enum for it? 
>> Even further down the rabbit whole there might be a CodingKey<> 
>> specialized for a concrete combination, like 
>> CodingKey<StringAndIntKey> or just CodingKey<StringKey>, but I’m 
>> not sure whether our type system will make it useful or possible…
>>
>> public enum CodingKeyValue {
>>   case integer(value: Int)
>>   case string(value: String)
>>   case both(intValue: Int, stringValue: String)
>> }
>> public protocol CodingKey {
>>   init?(value: CodingKeyValue)
>>   var value: CodingKeyValue { get }
>> }
> I agree that this certainly feels suboptimal. We’ve certainly 
> explored other possibilities before sticking to this one, so let me 
> try to summarize here:
>
> * Having a concrete 3-case CodingKey enum would preclude the 
> possibility of having neither a stringValue nor an intValue. However, 
> there is a lot of value in having the key types belong to the type 
> being encoded (more safety, impossible to accidentally mix key types, 
> private keys, etc.); if the CodingKey type itself is an enum (which 
> cannot be inherited from), then this prevents differing key types.
> * Your solution as presented is better: CodingKey itself is still a 
> protocol, and the value itself is the 3-case enum. However, since 
> CodingKeyValue is not literal-representable, user keys cannot be enums 
> RawRepresentable by CodingKeyValue. That means that the values must 
> either be dynamically returned, or (for attaining the benefits that we 
> want to give users — easy representation, autocompletion, etc.) the 
> type has to be a struct with static lets on it giving the 
> CodingKeyValues. This certainly works, but is likely not what a 
> developer would have in mind when working with the API; the power of 
> enums in Swift makes them very easy to reach for, and I’m thinking 
> most users would expect their keys to be enums. We’d like to 
> leverage that where we can, especially since RawRepresentable enums 
> are appropriate in the vast majority of use cases.
> * Three separate CodingKey protocols (one for Strings, one for Ints, 
> and one for both). You could argue that this is the most correct 
> version, since it most clearly represents what we’re looking for. 
> However, this means that every method now accepting a CodingKey must 
> be converted into 3 overloads each accepting different types. This 
> explodes the API surface, is confusing for users, and also makes it 
> impossible to use CodingKey as an existential (unless it’s an empty 
> 4th protocol which makes no static guarantees and the others inherit 
> from).
> * [The current] approach. On the one hand, this allows for the 
> accidental representation of a key with neither a stringValue nor an 
> intValue. On the other, we want to make it really easy to use 
> autogenerated keys, or autogenerated key implementations if you 
> provide the cases and values yourself. The nil value possibility is 
> only a concern when writing stringValue and intValue yourself, which 
> the vast majority of users should not have to do.
>   * Additionally, a key word in that sentence bolded above is 
> “generally”. As part of making this API more generalized, we push 
> a lot of decisions to Encoders and Decoders. For many formats, it’s 
> true that having a key with no value is an error, but this is not 
> necessarily true for all formats; for a linear, non-keyed format, it 
> is entirely reasonable to ignore the keys in the first place, or 
> replaced them with fixed-format values. The decision of how to handle 
> this case is left up to Encoders and Decoders; for most formats (and 
> for our implementations), this is certainly an error, and we would 
> likely document this and either throw or preconditionFailure. But this 
> is not the case always.
> * In terms of syntax, there’s another approach that would be really 
> nice (but is currently not feasible) — if enums were 
> RawRepresentable in terms of tuples, it would be possible to give 
> implementations for String, Int, (Int, String), (String, Int), etc., 
> making this condition harder to represent by default unless you really 
> mean to.

Hope that gives some helpful background on this decision. FWIW, the only 
way to end up with a key having no `intValue` or `stringValue` is 
manually implementing the `CodingKey` protocol (which should be 
_exceedingly_ rare) and implementing the methods by not switching on 
`self`, or some other method that would allow you to forget to give a 
key neither value.

> Speaking of the mutually exclusive representations - what above 
> serializations that doesn’t code as one of those two things? YAML 
> can have anything be a “key”, and despite that being not 
> particularly sane, it is a use case.
We’ve explored this, but at the end of the day, it’s not possible to 
generalize this to the point where we could represent all possible 
options on all possible formats because you cannot make any promises as 
to what’s possible and what’s not statically.

We’d like to strike a balance here between strong static guarantees on 
one end (the extreme end of which introduces a new API for every single 
format, since you can almost perfectly statically express what’s 
possible and what isn’) and generalization on the other (the extreme 
end of which is an empty protocol because there really are encoding 
formats which are mutually exclusive). So in this case, this API would 
support producing and consuming YAML with string or integer keys, but 
not arbitrary YAML.

>> For most types, String-convertible keys are a reasonable default; for 
>> performance, however, Int-convertible keys are preferred, and 
>> Encoders may choose to make use of Ints over Strings. Framework types 
>> should provide keys which have both for flexibility and performance 
>> across different types of Encoders. It is generally an error to 
>> provide a key which has neither a stringValue nor an intValue.
>>
>
> Could you speak a little more to using Int-convertible keys for 
> performance? I get the feeling int-based keys parallel the legacy of 
> NSCoder’s older design, and I don’t really see anyone these days 
> supporting non-keyed archivers. They strike me as fragile. What other 
> use cases are envisioned for ordered archiving than that?
We agree that integer keys are fragile, and from years (decades) of 
experience with `NSArchiver`, we are aware of the limitations that such 
encoding offers. For this reason, we will never synthesize integer keys 
on your behalf. This is something you must put thought into, if using an 
integer key for archival.

However, there are use-cases (both in archival and in serialization, but 
especially so in serialization) where integer keys are useful. Ordered 
encoding is one such possibility (when the format supports it, integer 
keys are sequential, etc.), and is helpful for, say, marshaling objects 
in an XPC context (where both sides are aware of the format, are running 
the same version of the same code, on the same device) — keys waste 
time and bandwidth unnecessarily in some cases.

Integer keys don’t necessarily imply ordered encoding, however. There 
are binary encoding formats which support integer-keyed dictionaries 
(read: serialized hash maps) which are more efficient to encode and 
decode than similar string-keyed ones. In that case, as long as integer 
keys are chosen with care, the end result is more performant.

But again, this depends on the application and use case. Defining 
integer keys requires manual effort because we want thought put into 
defining them; they are indeed fragile when used carelessly.

>> [snip]
>>
>> Keyed Encoding Containers
>>
>> Keyed encoding containers are the primary interface that most Codable 
>> types interact with for encoding and decoding. Through these, Codable 
>> types have strongly-keyed access to encoded data by using keys that 
>> are semantically correct for the operations they want to express.
>>
>> Since semantically incompatible keys will rarely (if ever) share the 
>> same key type, it is impossible to mix up key types within the same 
>> container (as is possible with Stringkeys), and since the type is 
>> known statically, keys get autocompletion by the compiler.
>>
>> open class KeyedEncodingContainer<Key : CodingKey> {
>
> Like others, I’m a little bummed about this part of the design. Your 
> reasoning up-thread is sound, but I chafe a bit on having to 
> reabstract and a little more on having to be a reference type. 
> Particularly knowing that it’s got a bit more overhead involved… I 
> /like/ that NSKeyedArchiver can simply push some state and pass itself 
> as the next encoding container down the stack.
There’s not much more to be said about why this is a `class` that I 
haven’t covered; if it were possible to do otherwise at the moment, 
then we would.

As for _why_ we do this — this is the crux of the whole API. We not 
only want to make it easy to use a custom key type that is semantically 
correct for your type, we want to make it difficult to do the easy but 
incorrect thing. From experience with `NSKeyedArchiver`, we’d like to 
move away from unadorned string (and integer) keys, where typos and 
accidentally reused keys are common, and impossible to catch statically.
`encode<T : Codable>(_: T?, forKey: String)` unfortunately not only 
encourages code like `encode(foo, forKey: "foi") // whoops, typo`, it is 
_more difficult_ to use a semantic key type: `encode(foo, forKey: 
CodingKeys.foo.stringValue)`. The additional typing and lack of 
autocompletion makes it an active disincentive. `encode<T : Codable>(_: 
T?, forKey: Key)` reverses both of these — it makes it impossible to 
use unadorned strings or accidentally use keys from another type, and 
nets shorter code with autocompletion: `encode(foo, forKey: .foo)`

The side effect of this being the fact that keyed containers are classes 
is suboptimal, I agree, but necessary.

>>     open func encode<Value : Codable>(_ value: Value?, forKey key: 
>> Key) throws
>
> Does this win anything over taking a Codable?
Taking the concrete type over an existential allows for static dispatch 
on the type within the implementation, and is a performance win in some 
cases.

>>     open func encode(_ value: Bool?,   forKey key: Key) throws
>>     open func encode(_ value: Int?,    forKey key: Key) throws
>>     open func encode(_ value: Int8?,   forKey key: Key) throws
>>     open func encode(_ value: Int16?,  forKey key: Key) throws
>>     open func encode(_ value: Int32?,  forKey key: Key) throws
>>     open func encode(_ value: Int64?,  forKey key: Key) throws
>>     open func encode(_ value: UInt?,   forKey key: Key) throws
>>     open func encode(_ value: UInt8?,  forKey key: Key) throws
>>     open func encode(_ value: UInt16?, forKey key: Key) throws
>>     open func encode(_ value: UInt32?, forKey key: Key) throws
>>     open func encode(_ value: UInt64?, forKey key: Key) throws
>>     open func encode(_ value: Float?,  forKey key: Key) throws
>>     open func encode(_ value: Double?, forKey key: Key) throws
>>     open func encode(_ value: String?, forKey key: Key) throws
>>     open func encode(_ value: Data?,   forKey key: Key) throws
>
> What is the motivation behind abandoning the idea of “primitives” 
> from the Alternatives Considered? Performance? Being unable to close 
> the protocol?
Being unable to close the protocol is the primary reason. Not being able 
to tell at a glance what the concrete types belonging to this set are is 
related, and also a top reason.

> What ways is encoding a value envisioned to fail? I understand wanting 
> to allow maximum flexibility, and being symmetric to `decode` 
> throwing, but there are plenty of “conversion” patterns the are 
> asymmetric in the ways they can fail (Date formatters, 
> RawRepresentable, LosslessStringConvertible, etc.).
Different formats support different concrete values, even of primitive 
types. For instance, you cannot natively encode `Double.nan` in JSON, 
but you can in plist. Without additional options on `JSONEncoder`, 
`encode(Double.nan, forKey: …)` will throw.

>>     /// For `Encoder`s that implement this functionality, this will 
>> only encode the given object and associate it with the given key if 
>> it encoded unconditionally elsewhere in the archive (either 
>> previously or in the future).
>>     open func encodeWeak<Object : AnyObject & Codable>(_ object: 
>> Object?, forKey key: Key) throws
>
> Is this correct that if I send a Cocoa-style object graph (with weak 
> backrefs), an encoder could infinitely recurse? Or is a coder supposed 
> to detect that?
`encodeWeak` has a default implementation that calls the regular 
`encode<T : Codable>(_: T, forKey: Key)`; only formats which actually 
support weak backreferencing should override this implementation, so it 
should always be safe to call (it will simply unconditionally encode the 
object by default).

>>     open var codingKeyContext: [CodingKey]
>> }
>> [snippity snip]
>>
> Alright, those are just my first thoughts. I want to spend a little 
> time marinating in the code from PR #8124 before I comment further. 
> Cheers! I owe you, Michael, and Tony a few drinks for sure.
Hehe, thanks :)

> Zach Waldowski
> zach at waldowski.me


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-evolution/attachments/20170317/8688be9f/attachment.html>