[swift-users] file io in Swift 2.2 for Linux (would like to be pointed in the right direction)

[Matthew Johnson]

> On May 20, 2016, at 10:47 AM, Jens Alfke via swift-users <swift-users at swift.org> wrote:
> 
> 
>> On May 20, 2016, at 1:16 AM, Quinn The Eskimo! via swift-users <swift-users at swift.org <mailto:swift-users at swift.org>> wrote:
>> 
>> One of the nice things about the file system is that it has reasonable performance and error characteristics such that you /can/ deal with it synchronously.  Revel in that fact!
> 
> That’s only really true at small scales, for local filesystems, under light I/O loads. This happens to be true most of the time* for typical Mac/iOS app needs, but it’s not universal.
> 
> I believe most people in the Swift community today come from a Mac/iOS background, but as Linux support matures, and with IBM’s backing, that’s going to be less and less true in the future. Me, I’m one of the oddball “mobile guys” at a database company [Couchbase] whose customers [eBay, PayPal, Citi, etc.] very commonly have multi-terabyte data sets.

Couchbase looks like a great product.  Cool to see someone from Couchbase is participating in Swift evolution!

> 
>> Consider the NSXMLDocument example again.  This high-level one-shot API is layered on top of a low-level one-shot API.
> 
> The trouble with this is that those one-shot APIs tend to suck when working with data coming from the network, because you have to buffer the entire data set in memory, and you can’t start parsing it until the entire response is finished. If it does finish — there are protocols like XMPP/Jabber that use indefinitely-long streams of structured data like XML and can’t be parsed at all with one-shot APIs. I’ve recently had to replace NSJSONSerialization with a 3rd-party streaming JSON parser library, because the latency of reading large JSON documents over HTTP was getting in the way of performance and memory requirements.
> 
> Sorry if this is off-topic! What’s relevant is that I’d hate to see Apple’s historical bias toward one-shot I/O get in the way of Swift’s Foundation framework having a kick-ass stream API.

+1.  There are many reasons to prefer streaming implementations.  It is also possible to write high-level APIs with relatively simple interfaces that use streaming behind the scenes.  The fact that this is not always done is not a good argument against providing robust streaming APIs.  In fact, it should be considered a challenge to improve the state of the art.

For example, I shared an example of a Ruby DSL I wrote that generated streaming XML parsers (in an Objective-C API and some C in the implementation) from the DSL which defined the structure of the document and how it maps to model objects.  This was much easier to use than any DOM-based API (whether XML, JSON, or whatever) and was more efficient.  It did not require the entire document to be in memory at once and did not construct all kinds of intermediate objects not needed by the model (dictionaries and arrays, etc).  

If / when Swift gets macros we should be able to implement something very similar directly in Swift without needing an intermediate code generation step.  It may also be possible to do this a better way in Swift.  

The main point I am making here is that taking advantage of the benefits of streaming APIs does not need to be difficult at the application level. 

> 
> —Jens
> 
> * But not always. I can attest that during OS startup, when dozens of processes are contending for the disk, reading a 10MB file can take an achingly long time by normal standards. This caused lots of “fun” during performance tuning of the PubSub framework. Also, remember the days when devs at Apple still had NFS-based home directories?
> _______________________________________________
> swift-users mailing list
> swift-users at swift.org
> https://lists.swift.org/mailman/listinfo/swift-users

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-users/attachments/20160520/2379a2f1/attachment.html>