[swift-server-dev] Next HTTP API meeting

[Johannes Weiß]

Hi Helge,


> On 4 Apr 2017, at 14:54, Helge Heß via swift-server-dev <swift-server-dev at swift.org> wrote:
> 
> On 04 Apr 2017, at 15:37, Johannes Weiß <johannesweiss at apple.com> wrote:
>>> On 31 Mar 2017, at 15:27, Helge Heß <me at helgehess.eu> wrote:
>>> On 30 Mar 2017, at 16:31, Helge Heß <me at helgehess.eu> wrote:
>>>> Personally I find the API a little weird (because the response/request ordering is backwards in the source) :-). But we have to get started somewhere.
>>>> 
>>>> Notes:
>>> 
>>> + there is not way to handle back pressure with that API
>> 
>> our implementation handles back-pressure as part of the HTTP library. So you're correct in that a client of that API can't directly control back-pressure. Personally I feel it's better for that to live inside the HTTP library implementation.
> 
> So you have one queue filling a buffer-list, pausing the dispatch source if a buffer is full, and then another queue taking buffers of the list and running the HTTPBodyHandler callbacks?

you can think of it that way, yes.


> But how does it know that the callback is actually done processing the chunk, there is no done callback or something :-)

yes, agreed, there should be a done callback for the body processor. Missed that when compiling the proposal.


> To me this still looks like there would be no back pressure handling at all, only between the queues, which kinda looks useless. Unless the body handler is synchronous and doesn’t return before processing the chunk, which defeats the idea of async.

I agree that should be added to the API I proposed. It still makes a big difference without the completion handler for the chunk handler but I agree there's applications for which it just wouldn't work.

 
> Please elaborate :-)
> 
> 
>>> + it provides no way to stop the body processor
>>> 
>>> E.g. if you determined that you don’t want to handle the
>>> content after a while. Something like
>>> 
>>> typealias HTTPBodyHandler = ( HTTPBodyChunk, inout Bool ) -> Void
>> 
>> given that the underlying HTTP parser library needs to parse everything anyway, we just ignore everything that we don't care about. So it'll still be handed to the client but the client would just ignore everything by doing nothing.
> 
> So if I upload a 10000TB file to iCloud, you still /dev/null process it? That doesn’t seem sensible to me :-)

well, you need to parse it anyway, so you can't ignore the data. But yes, you can optimise it by communicating your desire to ignore the rest into the HTTP library. The question is how many applications actually want to stream a very large amount of data to just ignore it. But I'm not opposed at all to include that into the API.

As I said at the beginning: This API is something that I compiled from a library we use internally. It's not an exact copy of the code but close enough. But we shouldn't take it verbatim, it was more meant as a basis for discussion. And many points you raise are absolutely valid, like having a completion handler for the chunk handler. So please keep the comments coming, more than happy to improve it.

Cheers,
  Johannes