[swift-server-dev] HTTP Parser

Paulo Faria paulo at zewo.io
Sun Nov 6 14:07:47 CST 2016 

I agree with you Ben. (:

Drawing from what Helge said and my own experience with parsers, I think the best design for parsers is the streaming (or incremental) parser. Streaming parsers work by the following mechanism. You send chunks of data and when the parser has gathered enough data to construct the parsed value it gives you the request. The pseudo-code would be something like this:

let parser = RequestParser()

while !stream.closed {
  let chunk = stream.read(bufferSize)

  try parser.parse(chunk) { request in
      // do whatever with the request
  }

  ...
}

So we read a chunk from the tcp stream, for example, and then pass that chunk to the parser. If the parser doesn’t have enough information to create a request, the call to `parse` returns and then the loop continues. We read from the stream again and send it to the parser. Let’s say we have enough information to construct a request now. When that happens the `parse` function calls the closure sending the constructed request. The closure is non escaping, which means that `parse` will only return after the closure returns. This means the parser acts basically like an “if” statement activating the closure when there’s a request or just moving on if it doesn’t.

The `parse` function could return an optional Request instead of taking a closure. But this would not be ideal if the data that is sent to the parser contains enough information to produce more than one request. With the closure design, if the parser has enough information to construct two or more request, all it has to do is send each request to the closure. If we used the optional approach we would have return the first parsed request, then store the second one and we would only be able to return that second request if the `parse` function is called again, and we have no guarantees about that. So we could effectively lose requests this way.

So instead of the optional we could still return an array of requests in case we have parsed more than one request.	The pseudo-code would be something like this:

let parser = RequestParser()

while !stream.closed {
  let chunk = stream.read(bufferSize)

  for request in try parser.parse(chunk) {
      // do whatever with the request
  }

  ...
}

This would have the same effect of the closure-based API, but it wouldn’t be as optimal because it would require the parser to store all parsed requests in an array before it returns it. With the closure-based API we don’t need to store the parsed requests. We just send a request to the closure as soon as we construct one.

The streaming parser design is pretty good because it allows you to provide convenience APIs which you can use for one-shot parsing. Where you know (or expect) to have binary data that fully represents a request. Then you can provide this API on top of the streaming parser.

let data = getBinaryDataRepresentingAFullRequestFromSomewhere()
let request = try RequestParser.parse(data)

This scenario is very unlikely when it comes to HTTP parsing in real applications, but if it’s needed for say unit testing purposes, we have a nice way to do that.

This design applies to HTTP request and HTTP responses and it works the same way for both. I only used requests here because that’s what we would be doing on the HTTP server side. On the HTTP client side it would be the same mechanism, but it would parse responses instead of requests.

> On Nov 6, 2016, at 4:09 PM, Benoit Person <benoit.person at gmail.com> wrote:
> 
> I am new here and mostly following along (so take all I say with a
> "newbie" pov ;) ), but is there actual value in discussing whether or
> not the implementation should be Swift-based or a wrapper around C
> right now?
> 
> Let's focus on designing a Swift-friendly API, unconstrained by any
> "implementation detail" (C-wrapper or pure Swift), and when we have
> that, make an educated decision on whether or not it can be built by
> wrapping a C library to speed up delivery and iterations on the API?
> 
> Ben
> 
> On 6 November 2016 at 18:09, Logan Wright via swift-server-dev
> <swift-server-dev at swift.org> wrote:
>> 
>> Here are some of the most basic reasons why I would prefer to use swift implementations.
>> 
>> - Easier for Swift contributors to enter and participate
>> - More readable code
>> - Faster in long run
>> - More concise code
>> - Fully embraces Swift language
>> - Easier to maintain with control over the library
>> 
>> Again, I am not saying this all needs to happen immediately, but I think assuming that we'll only ever use c libraries in perpetuity is an extremely short sighted vision of how the server apis can grow. We should also keep in mind that when talking about HTTP, this isn't a from scratch rewrite as seems to be suggested several times. There are a few (likely more) parsers written in pure swift that we could mix and match to get something solid into the library. The fact that these libraries exist makes wrapping c seem like a step backwards in the progression of Swift which I am against. A perfectly reasonable compromise (as suggested before) seems to be initially wrapping some c code while leaving room in the design for Swift implementations long term if necessary.
>> 
>> I'd still like to get more voices, we seem to have heard from about 7 people or so, and from what I can tell, the count is about 4/3. Anyone following along care to weigh in a little bit so the same few people aren't rehashing the same talking points.
>> 
>> - Logan
>> 
>> On Sun, Nov 6, 2016 at 7:37 AM Paulo Faria via swift-server-dev <swift-server-dev at swift.org> wrote:
>>> 
>>> Hey Joannis! Nice to see you here.
>>> 
>>> I agree with you 100℅ we shouldn't rush the design of the APIs. I didn't mean that at all. I meant exactly the opposite. We should spend as much time as possible on the API design, and wrapping a C library would allows to distribute our total time of development focusing a lot more on the API design than on the development and maintenance of the internals. We should definitely design the API with the best of Swift using protocols and everything else you mentioned, and wraping C doesn't restrain us in any way to do that. There's nothing we can't do on the API level because we're wrapping C. What Chris Bailey mentioned about my mentality before is totally right. I believe we should spend as much time as possible designing the APIs without any constraint from, for example, Foundation or any C library, and in the Foundation case if really needed we should compromise. I just know from experience that wrapping C libs doesn't constraint at all the "Swiftyness" of the user-facing API. This is pure conjecture as I didn't really dig in, but I believe Swift's standard library itself uses libicu for String's internal on Linux. If this is wrong someone please correct me. I also believe Foundation wraps libcurl for the HTTP stuff.
>>> 
>>> 
>>> ---- On Sun, 06 Nov 2016 07:54:15 -0200 joannis at orlandos.nl wrote ----
>>> 
>>> 
>>> 
>>> I think that if you wrap C libraries you'll get stuck in the wrong mentality. Working with C libraries encourages thinking without protocols. Protocols like `ExpressibleByDictionaryLiteral` and `Sequence` are really easy to implement and make usage of conforming types really beautiful. I think that a lot of APIs will be designed with the wrong mentality, at least initially, because they're alien to Swift. And that'll create APIs for these libraries that are not at the level where they could be in a language like Swift.
>>> 
>>> I don't think we need to rush ourselves with trying to create the libraries rapidly. If you want to have a common set of libraries in which the entire community will cooperate I think Swift is the only way to go. Many people have already written big libraries and frameworks in Swift. And I myself would never remove my own Swift code in favour of C-reliant code. And I doubt it's just me. I think that people whom prefer pure Swift will stick with (their own) existing functionality and rather build on top of a C reliant codebase.
>>> 
>>> Besides that all I feel like there's no need to rush it. A good set of libraries needs time. If you have a million people working on something for a week it will never be as good as a hundred people for a year or two. A good API/library needs time to be thought out and will need more than a few nights of rest before I think the idea alone should be considered complete. I think this counts for C-based and pure Swift libraries.
>>> 
>>> And if we'd just take the time to develop something great it will have a much greater impact on the Server Side Swift world than it would otherwise.
>>> 
>>> Joannis
>>> 
>>> On 6 Nov 2016, at 10:44, Chris Bailey via swift-server-dev <swift-server-dev at swift.org> wrote:
>>> 
>>> Whilst it might not feel like it, it seems to be that we're all pulling in the same direction.
>>> 
>>> Tony said:
>>>>> I think if the API design feels “alien” to Foundation, then we either need to update the Foundation API to feel like it fits in with Swift or design the server API to fit in with Foundation.
>>> 
>>>>> I’ll keep pushing for this throughout the process. It’s really important that we have a coherent stack of software from top to bottom. Ending up with different model objects for URLRequest used in the same app via two frameworks would be really unfortunate.
>>> 
>>> I think this completely aligns with what everyone is saying - we want/need a coherent API stack with seamless interop between any new APIs and the existing ones. The only thing we need to work on is the approach to achieving that.
>>> 
>>> For me, and I think that this is the approach that Paulo is pushing for, is that we should first look at "what do we think is the right approach", without any constraints. Once we understand that, and any gap there is between that and the Foundation approach, we can determine how we close that gap - whether we can do that by evolving APIs, or whether it requires compromises for the sake of developer experience and application code portability, etc.
>>> 
>>> Chris
>>> 
>>> 
>>> 
>>> 
>>> From:        Paulo Faria via swift-server-dev <swift-server-dev at swift.org>
>>> To:        Tony Parker <anthony.parker at apple.com>
>>> Cc:        Swift Server Dev <swift-server-dev at swift.org>
>>> Date:        04/11/2016 23:08
>>> Subject:        Re: [swift-server-dev] HTTP Parser
>>> Sent by:        swift-server-dev-bounces at swift.org
>>> ________________________________
>>> 
>>> 
>>> 
>>>> With respect to adopting other core Swift ideas like value types: we turned 20 classes into value types last year. This included a ton of work to adopt standard library protocols and improve type safety. It is an absolutely huge surface amount of Foundation’s total API surface area. It’s a large statement about how much we value making Foundation’s API consistent for Swift.
>>> 
>>> Yeah! I really love the effort you’re putting into making Foundation more “Swifty”. That doesn’t go unnoticed.
>>> 
>>> 
>>> On Nov 4, 2016, at 8:49 PM, Tony Parker <anthony.parker at apple.com> wrote:
>>> 
>>> However, I do not believe there is such a fundamental conceptual mismatch here that we cannot preserve one of the most useful aspects of developing with the iOS, macOS SDKs - consistent types and low impedance mismatch between API at all levels of the stack.
>>> 
>>> I definitely value "consistent types and low impedance mismatch between API at all levels of the stack”. I just hope the APIs are designed with the best of Swift in mind. Then after we get a good design, we can see if it fits Foundation, and then make the adaptations there. I think this approach would make Foundation follow along and become even more “Swiftier” with time. Of course this is easier said than done, but I believe that’s what would make the ecosystem as a whole become better for Swift._______________________________________________
>>> swift-server-dev mailing list
>>> swift-server-dev at swift.org
>>> https://lists.swift.org/mailman/listinfo/swift-server-dev
>>> 
>>> 
>>> _______________________________________________
>>> swift-server-dev mailing list
>>> swift-server-dev at swift.org
>>> https://lists.swift.org/mailman/listinfo/swift-server-dev
>>> 
>>> 
>>> _______________________________________________
>>> swift-server-dev mailing list
>>> swift-server-dev at swift.org
>>> https://lists.swift.org/mailman/listinfo/swift-server-dev
>> 
>> 
>> _______________________________________________
>> swift-server-dev mailing list
>> swift-server-dev at swift.org
>> https://lists.swift.org/mailman/listinfo/swift-server-dev

More information about the swift-server-dev mailing list