<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Hi Gwendal,<div class=""><br class=""></div><div class="">I hear your frustration. Some comments inline.<br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On May 31, 2017, at 5:36 AM, Gwendal Roué <<a href="mailto:gwendal.roue@gmail.com" class="">gwendal.roue@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><meta http-equiv="Content-Type" content="text/html charset=utf-8" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div dir="auto" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Itai,</div><div dir="auto" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><br class=""></div><div dir="auto" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">(This email is not technical)</div><div dir="auto" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><br class=""></div><div dir="auto" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">I'm not claiming that SE-0166 should be able to address all archival formats. I've been talking about GRDB to show at least one format that SE-0166 doesn't cover well. And should SE-0166 be fixed to support SQL (in the GRDB fashion), this does not mean that other developers won't eventually fight with SE-0166 until they understand it does not fit their bill.</div></div></div></blockquote><div>I’ll respond to the technical portion of this thread in the other email, but let me at least provide some background here. When working on this feature, we thought for a <i class="">very</i> long time about what we were looking to support with this feature, and how (feel free to take a look at the Alternatives Considered section of the proposal, though of course, there were more attempts and approaches before that).</div><div>The majority of this thought was put into figuring out what the proper abstractions were for applying this new API — how can we abstract over different archival and serialization formats in a way that makes this useful?</div><div><br class=""></div><div>In truth, if you try to abstract over all archival and serialization formats, the abstraction that you get is... the empty set. :) There are simply so many different things at odds with one another across different formats (JSON supports null values, plist does not; numbers are arbitrary precision in JSON, but not in plist or MessagePack or others; plist and MessagePack and others support binary data blobs, but JSON does not; etc.) that if you try to abstract over them all, you end up with nothing useful — an empty protocol that covers nothing.</div><div><br class=""></div><div>So the key here is to try to strike a pragmatic balance between supporting some of the most common archival and serialization formats in a way that makes them useful, even if we have to handle special cases in some of them (e.g. null values in plist, binary data in JSON, etc.). It’s true that we cannot support them all, but in fact, we’re not looking to, because it would weaken the API.</div><div><br class=""></div><div>I will respond to the comments specific to GRDB in the other thread, but this is a bit of background. Yes, there will always developers who will not be able to fit a serialization format into this API because it is fundamentally different in a way that cannot fit with the rest of the formats we’re looking to support. There’s nothing to be done about that. But, you mention this yourself.</div><br class=""><blockquote type="cite" class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div dir="auto" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">But there's something very special with SE-0166:</div><div dir="auto" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><br class=""></div><div dir="auto" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">It's in the standard library, with all the backward-compatibility constraints that come with such a position.</div><div dir="auto" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><br class=""></div><div dir="auto" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">IT'S BLESSED WITH CODE GENERATION.</div><div dir="auto" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><br class=""></div><div dir="auto" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">I don't know if you, Michael LeHew, Tony Parker, and the core team, realize the importance of this insanely great privilege granted to this proposal.</div></div></div></blockquote><div>Believe me, I do, because we considered a <i class="">lot</i> of different approaches before settling on this. We wanted to avoid code generation for this reason — it has a privileged place within the compiler, it generates code which the user may not be able to introspect, etc.</div><div>At the end of the day, though, we decided on this option because it provided the best user experience as part of the language in the vast majority of cases. There’s a lot to be said for that, and you mention this yourself, too.</div><br class=""><blockquote type="cite" class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div dir="auto" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">The lack of introspection and macros in Swift makes SE-0166 immensely attractive for a whole category of libraries.</div><div dir="auto" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><br class=""></div><div dir="auto" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">When SE-0166 is lacking, should those libs ignore it, and lose CODE GENERATION, which means looking like it's still Swift 3?</div><div dir="auto" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><br class=""></div><div dir="auto" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Should those libs claim SE-0166 conformance, and raise runtime errors for invalid inputs (where "invalid" does not mean "invalid data", or "invalid code", but "impossible to fit in SE-0166" <=> "invalid library")?</div></div></div></blockquote>That being said, let’s separate the capabilities of the Codable API itself from the code generated by the compiler for it. While the code generation is a huge convenience for the majority of simple cases, it does just that — generate code for the <i class="">simple</i> cases. We cannot arbitrarily generate code to match arbitrary applications. Much more is possible with custom encode/decode implementations and custom CodingKeys than you might imagine, rather than just sticking to the default, compiler-generated implementation. (Data migration, format-specific encoded representations, multiple sets of CodingKeys, etc.)</div><div><br class=""></div><div>If a library finds use for the Codable APIs only for the code generation, then I think that’s likely misapplication of the API. Attempting to use the Codable API to fit a square peg into a round hole will be frustrating because, well, it was designed for a singular purpose.</div><div>The code generation that comes with Codable is meant for archival and serialization, not for arbitrary introspection. You’re right in that there is an overlap here (and I think the key pain point is that we need better tools for doing introspection, macros, and compile-time metaprogramming), but this is not a problem that this API is meant to solve. If a library cannot use the code generated by the Codable API, it’s not an "invalid library" — it’s just a poor fit. A full-featured compile-time macro/introspection system will require further thought and discussion, but the current Codable feature will eventually fit into that.</div><div><br class=""></div><div><blockquote type="cite" class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div dir="auto" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">I'd like to hear a little better than that :-) GRDB is a library of unusual quality (sorry for the auto-congratulation). Until now, fatal errors thrown by GRDB were always a sign of programmer mistake. Not of defects in the foundations. I wish this would remain true.</div><div dir="auto" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><br class=""></div><div dir="auto" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Less caveats and runtime/fatal errors mean less user frustration.</div><div dir="auto" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><br class=""></div><div dir="auto" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Less caveats also mean less documentation to write. Ideally, this should be enough: <a href="https://github.com/groue/GRDB.swift/tree/Swift4#codable-records" class="">https://github.com/groue/GRDB.swift/tree/Swift4#codable-records</a></div></div></div></blockquote><div>There are always going to be cases where certain input cannot be limited in the type system. Consider JSON, which does not support NaNs or Infinity. There is no way to statically prevent that input via the type system, so yes, there has to be a runtime error thrown for that. That doesn’t make JSON invalid for this API; we just add a strategy to the encoder to allow users to customize what happens there. These errors don’t have to be fatal errors — we have EncodingError.invalidValue to cover just this issue; it is just not possible to cover all possible properties and all possible values of inputs statically.</div><div><br class=""></div><div>It’s up to Encoders and Decoders to make the decisions that allow them to fit nicely within this. If they cannot, then it might be that they are not a good fit for this API.</div><div><br class=""></div><blockquote type="cite" class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div dir="auto" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Gwendal</div><div dir="auto" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Just a guy that write Swift apps and libraries</div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><blockquote type="cite" class=""><div class="">Le 30 mai 2017 à 20:49, Itai Ferber <<a href="mailto:iferber@apple.com" class="">iferber@apple.com</a>> a écrit :</div><br class="Apple-interchange-newline"><div class="">
<meta http-equiv="Content-Type" content="text/xhtml; charset=utf-8" class="">
<div class="">
<div style="font-family:sans-serif" class=""><div style="white-space:normal" class=""><p dir="auto" class="">Hi Gwendal,</p><p dir="auto" class="">There are no stupid questions — everything helps hammer out this API, so I appreciate you taking the time to look at this so deeply.<br class="">
I have to confess that I’m not familiar with this concept, but let’s take a look:</p>
<pre style="background-color: rgb(247, 247, 247); border-top-left-radius: 5px; border-top-right-radius: 5px; border-bottom-right-radius: 5px; border-bottom-left-radius: 5px; margin-left: 15px; margin-right: 15px; max-width: 90vw; overflow-x: auto; padding: 5px;" bgcolor="#F7F7F7" class=""><code style="background-color:#F7F7F7; border-radius:3px; margin:0; padding:0" bgcolor="#F7F7F7" class=""><span style="color: #008800; font-weight: bold" class="">if</span> <span style="color: #008800; font-weight: bold" class="">let</span> <span style="color: #996633" class="">valueType</span> = T.<span style="color: #008800; font-weight: bold" class="">self</span> <span style="color: #008800; font-weight: bold" class="">as</span>? DatabaseValueConvertible.<span style="color: #008800; font-weight: bold" class="">Type</span> {
<span style="color: #888888" class="">// if column is missing, trigger the "missing key" error or return nil.</span>
} <span style="color: #008800; font-weight: bold" class="">else</span> <span style="color: #008800; font-weight: bold" class="">if</span> <span style="color: #008800; font-weight: bold" class="">let</span> <span style="color: #996633" class="">complexType</span> = T.<span style="color: #008800; font-weight: bold" class="">self</span> <span style="color: #008800; font-weight: bold" class="">as</span>? RowConvertible.<span style="color: #008800; font-weight: bold" class="">Type</span> {
<span style="color: #888888" class="">// if row scope is missing, trigger the "missing key" error or return nil.</span>
} <span style="color: #008800; font-weight: bold" class="">else</span> {
<span style="color: #888888" class="">// don't know what to do</span>
<span style="color: #007020" class="">fatalError</span>(<span style="background-color: #fff0f0" class="">"unsupported"</span>)
}
</code></pre><p dir="auto" class="">Is it appropriate for a type which is neither <code style="background-color:#F7F7F7; border-radius:3px; margin:0; padding:0 0.4em" bgcolor="#F7F7F7" class="">DatabaseValueConvertible</code> nor <code style="background-color:#F7F7F7; border-radius:3px; margin:0; padding:0 0.4em" bgcolor="#F7F7F7" class="">RowConvertible</code> to be decoded with your decoder? If not, then this warrants a <code style="background-color:#F7F7F7; border-radius:3px; margin:0; padding:0 0.4em" bgcolor="#F7F7F7" class="">preconditionFailure</code> or an error of some sort, right? In this case, that would be valid.</p><p dir="auto" class="">You also mention that "it’s still impossible to support other Codable types" — what do you mean by this? Perhaps there’s a way to accomplish what you’re looking to do.<br class="">
In any case, one option (which is not recommended unless if there are other avenues to solve this by) is to perform a "dry run" decoding. Attempt to decode the type with a dummy decoder to see what container it will need, then prepare your approach and do it again for real. Obviously, this isn’t a clean way to do it if we can find alternatives, but it’s an option.</p><p dir="auto" class="">— Itai</p><p dir="auto" class="">On 29 May 2017, at 4:51, Gwendal Roué via swift-evolution wrote:</p>
</div>
<div style="white-space:normal" class=""><blockquote style="border-left:2px solid #777; color:#777; margin:0 0 5px; padding-left:5px" class=""><p dir="auto" class="">Hello,<br class="">
<br class="">
I have already asked stupid questions about SE-0167 and SE-0166, but this time I hope this is a real one.<br class="">
<br class="">
According so SE-0166, codable types themselves instantiate a single value decoder, or a keyed container:<br class="">
<br class="">
public struct Farm : Codable {<br class="">
public init(from decoder: Decoder) throws {<br class="">
let container = try decoder.container(keyedBy: CodingKeys.self<br class="">
...<br class="">
}<br class="">
}<br class="">
<br class="">
public enum Animal : Int, Codable {<br class="">
public init(from decoder: Decoder) throws<br class="">
let intValue = try decoder.singleValueContainer().decode(Int.self)<br class="">
...<br class="">
}<br class="">
}<br class="">
<br class="">
According to SE-0167, decoder decode non-trivial types in their decode(_:forKey:) and decodeIfPresent(_:forKey:) methods:<br class="">
<br class="">
func decode<T>(_ type: T.Type, forKey key: Key) throws -> T where T : Decodable<br class="">
func decodeIfPresent<T>(_ type: T.Type, forKey key: Key) throws -> T? where T : Decodable<br class="">
<br class="">
My trouble is that the decoder does not know whether the Decodable type will ask for a keyed container, or for a single value container.<br class="">
<br class="">
Why is it a problem?<br class="">
<br class="">
In the context of decoding of SQL rows, keys may refer to different things, depending on whether we are decoding a *value*, or a *complex object*:<br class="">
<br class="">
- for values, keys are column names, as everybody can expect<br class="">
- for complex objects, keys are names of "row scopes". Row scopes are a concept introduced by GRDB.swift and allows a type that knows how to consume `SELECT * FROM table1` to consume as well the results of `SELECT table1.*, table2.* FROM table1 JOIN table2` through a "scope" that presents the row in the shape expected by the consumer (here, only columns from table1).<br class="">
<br class="">
This is supposed to allow support for types that contain both nested types and values (one of the goals of SE-0166 and SE-0167):<br class="">
<br class="">
struct Compound : Codable {<br class="">
let someStruct: SomeStruct // object that feeds on the "someStruct" scope<br class="">
let name: String // value that feeds on the "name" column<br class="">
}<br class="">
<br class="">
The two decoding methods decode(_:forKey:) and decodeIfPresent(_:forKey:) can't be implemented nicely, because they don't know whether the decodable type will ask for a keyed container or a single value container, and thus they don't know whether they should look for the presence of a row scope, or of a column:<br class="">
<br class="">
A workaround is to perform runtime checks on the GRDB protocols adopted by T, as below. But it's still impossible to support other codable types:<br class="">
<br class="">
if let valueType = T.self as? DatabaseValueConvertible.Type {<br class="">
// if column is missing, trigger the "missing key" error or return nil.<br class="">
} else if let complexType = T.self as? RowConvertible.Type {<br class="">
// if row scope is missing, trigger the "missing key" error or return nil.<br class="">
} else {<br class="">
// don't know what to do<br class="">
fatalError("unsupported")<br class="">
}<br class="">
<br class="">
Do you have any advice?<br class="">
<br class="">
Gwendal Roué<br class="">
<br class="">
<br class="">
_______________________________________________<br class="">
swift-evolution mailing list<br class="">
<a href="mailto:swift-evolution@swift.org" class="">swift-evolution@swift.org</a><br class="">
<a href="https://lists.swift.org/mailman/listinfo/swift-evolution" style="color:#777" class="">https://lists.swift.org/mailman/listinfo/swift-evolution</a></p>
</blockquote></div>
<div style="white-space:normal" class="">
</div>
</div>
</div>
</div></blockquote></div><br class=""></div></div></blockquote></div><br class=""></div></body></html>