[swift-server-dev] Prototype of the discussed HTTP API Spec

[Johannes Weiss]

> On 1 Jun 2017, at 6:27 pm, Michael Chiu <hatsuneyuji at icloud.com> wrote:
> 
> 
> Hi Johannes,
> 
>> Yes, Linux, macOS, FreeBSD and so on offer only a 1:1 threading model from the OS (Windows I think has Fibers). But libmill/dill/venice implement something akin to user-level threads themselves, you don't need kernel support for that at all (to schedule them cooperatively). Check out the man pages of setjmp and longjmp. Or check out Go (goroutines), Lua (coroutines), ... These are all basically cooperatively scheduled threads.
>> 
>> In other words: With setjmp() you can make a snapshot of the current environment and with longjmp() you can replace the current environment with one that you previously saved. That's like cooperatively switching something like a user-level thread/coroutine/green thread.
>> 
>> Run for example the code in this stackoverflow example: https://stackoverflow.com/a/14685524
>> 
>> This document explains it pretty well too: http://libdill.org/structured-concurrency.html
> 
> This is off topic but interesting stuff, your claim is generally true except when any one of the green threads invoke a blocking system call (in this case, kevent/epoll),
> setjmp(), longjmp() only take cares the context swiftching _within_ a real thread and make it act like a lot of concurrent(not really) threads, in orther words it only take care of the time sharing problem.

Of course, that's why I wrote "cooperatively scheduled" and "cooperatively switching". Cooperative scheduling means that a thread cannot be switched if it doesn't cooperate (ie. voluntarily yields). The opposite is 'preemptive scheduling' when the scheduler can switch threads without the cooperation of the threads.

If one of those coroutines doesn't cooperate with the system (ie. calls a blocking system call), we'll see everything stalling. Therefore those systems normally offer special read/write/... calls that appear blocking but in reality just signal 'interest in reading/writing' and then yield the thread. Again, check out libdill/mill/venice/node. You'll still be able to call regular system calls but as soon as you call a blocking one, the system grinds to a halt until that system call returns.


> So let’s say if we have a sigle thread, and green threads A. B. C. D, despite they can have independent stack pointer, they are still using the same thread.
> 
> When a blocking system call invoked, you actually making an interrupt and cause the real thread associate with the current user thread (fiber, coroutine…) enter the kernel (as context switching), and will only return back to normal routine when the system call returns, so when the real thread is in the kernel, non of the associated “user threads” can do anything nor perform context switching  from user space since the user-space lost control on the thread.

I do understand what happens if you call blocking system calls on coroutines but thank you.


>>>>> in fact all the example you listed above all uses events api internally. Hence I don’t think if an api will block a kernel thread is a good argument here.
>>>> 
>>>> kernel threads are a finite resource and most modern networking APIs try hard to only spawn a finite number of kernel threads way smaller than the number of connections handled concurrently. If you use Dispatch as your concurrency mechanism, your thread pool will have a maximum size of 64 threads by default on Darwin. (Sure you can spawn more using (NS)Thread from Foundation or pthreads or so)
>>> 
>>> Yes Kernel threads are finite resources especially in 1:1 model but I’m not sure how is it relevant. My concern on not include a synchronous API is that it make people impossible to write synchronous code, with server side swift tools, despite blocking or not, which they might want to. I’m not saying sync is better, I’m just saying we could give them a chance.
>> 
>> No one's taking anything away from you. Everything you have today will still be available. But I believe the APIs (which is what we're designing here) a web app uses should in today's world in Swift be asynchronous.
>> 
>> Of course to implement the asynchronous API, synchronous system calls will be used (eg. kevent/epoll). But the user-facing API that is currently proposed is async-only in order for it to be implementable in a performant way. If we were to put synchronous functions in the user-facing API, then we'll struggle to implement them in a performant way).
>> 
>> Imagine the function to write a HTTP body chunk looked like this:
>> 
>> func writeBodyChunk(_ data: Data) throws -> Void
>> 
>> then the user can expect this to only return when the data has been written successfully and that the connection was dropped if it throws.
>> But the implementation now has a problem: What to do if we can't write the bytes immediately? The only option we have is block this very thread and wait until we have written the bytes. Then we can return and let the user know if the write worked or not.
>> 
>> Comparing this to
>> 
>> func writeBodyChunk(_ data: Data, completion: (Error?) -> Void) -> Void
>> 
>> we can now register the attempt to write the data and move on with the calling thread. When the data has been written we invoke the completion handler and everything's good.
> 
> Well the expected behavior is that is the data cannot write immediately it will throw an error ‘temporary unavailable’ and the user can write again if he want to, exactly the same approach in my cpp code.

I'm really sorry but this is really not an API a user would want to write a web app. Your first version of the C++ example just dropped the bytes, whilst that indeed looks very straight forward in code it unfortunately doesn't work.

If you bring the user into the situation that something's temporarily unavailable, they'll probably retry, what else can they do?

while true {
    do {
        try writeBodyChunk(...)
        break
    } catch let e as TemporarilyUnavailable {
        /* what to do next? retry? */
    } catch let e {
        /* a real error */
    }
}

and that again blocks a thread until the write happens which might take a very long time. The only thing we achieve is more burnt CPU cycles.


>>>>> And even if such totally non-blocking programming model it will be expensive since the kernel is constantly scheduling a do-nothing-thread. (( if the io thread of a server-side application need to do something constantly despite there’s no user and no connection it sounds like a ghost story to me )).  
>>>> 
>>>> what is the do-nothing-thread? The IO thread will only be scheduled if there's something to do and then normally the processing starts on that very thread. In systems like Netty they try very hard to reduce hopping between different threads and to only spawn a few. Node.js is the extreme which handles everything on one thread. It will be able to do thousands of connections with only one thread.
>>>> 
>>> 
>>> The kernel has no idea is a thread have anything to do unless it sleeps/enterKernel, unless a thread fits in these requirements, it will always scheduled by the kernel.
>> 
>> but that's exactly what epoll/kevent do. They enter the kernel and tell the kernel what the user-space needs next.
>> 
>> The thread is now scheduled only if an event that kevent/epoll are waiting for turns up. And when kevent/epoll then return, most of the time the user space handler is submitted as a callback.
> 
> That’s why I said “a totally non blocking programming model is very expensive”, since both kevent and epoll are __Blocking__ system calls, so an api that doesn’t block _any_ kernel threads automatically exclude kevent and epoll.

Of course, look back at my very first email I sent to you in this conversation:

<quote>
Not really, what you do when you use kqueue/(e)poll/select is that only said calls are blocking and you set your file descriptors to non-blocking.
</quote>

Ie. kevent(I mistakenly wrote kqueue)/epoll/select are blocking and the other APIs you use a non-blocking. 


The important bit it that only a defined number of threads (say equal to the number of CPUs you have) call kevent/epoll/select. If you use one kevent/epoll/select call on a thread per connection you can make your life a lot easier by just sticking to synchronous IO because that's the same. 



>> well, it means the write hasn't happened. You will need to do the write again and that is normally done when kevent/epoll tell you to. And that's what I mean by inversion of control.
>> 
> 
> If you read the code carefully you will see additional write invoked (the cpp one)

Yes, you now changed your cpp one to include swrite(), let me paste that here

--- SNIP ---
void swrite(int fd, void * buf, size_t size)
{
    int idx = 0;
    while (idx < size) {
        size_t nwrite = write(fd, (void *)((uintptr_t)buf + idx), size - idx);
    
        if (nwrite == -1) {
            if (errno == EAGAIN)
                continue;
            else {
                close(fd);
                break;
            }
        }

        idx += nwrite;
    }
}
--- SNAP ---

what this is doing is turning it into a tight busy loop. This will just burn CPU for absolutely nothing. You will still be blocking a thread. It would be much cheaper if you would just do blocking IO here.

I just don't get what you're trying to tell me. You're always blocking a real thread, sometimes by blocking and with the swrite() by tight looping. But besides the wasted CPU cycles it's exactly equivalent to just calling a blocking write, you'll block this thread until the socket becomes writable. That could take nanoseconds, seconds, minutes, hours or longer.


>>>> Foundation/Cocoa is I guess the Swift standard library and they abandon synchronous&blocking APIs completely. I don't think we should create something different (without it being better) than what people are used to.
>>>> 
>>>> Again, there are two options for IO at the moment:
>>>> 1) synchronous & blocking kernel threads
>>>> 2) asynchronous/inversion of control & not blocking kernel threads
>>>> 
>>>> Even though I would love a synchronous programming model, I'd chose option (2) because the drawbacks of (1) are just too big. The designers of Foundation/Cocoa/Netty/Node.js/many more have made the same decision. Not saying all other options aren't useful but I'd like the API to be implementable with high-performance and not requiring the implementors to block a kernel thread per connection.
>>> 
>>> To be honest I will choose 2 as well. But we are in not a 2 choose 1 situation. The main difference between we and netty/node.js is that ppl use them to, write a server, what we do is, writing something ppl use to write something like netty and node.js. So it is reasonable to think there’s demand on a lower-level, synchronous api, despite the possible “drawbacks” they might encounter.
>> 
>> This is the HTTP group so people will only write web servers with it, the API that was proposed it definitely not meant to implement anything netty or node like. It's to implement web apps in Swift.
>> 
>> There is however also a Networking/Transport group which will be more low-level than this (I assume) and there we do need to consider the lower-level APIs. And those will contain blocking system calls, namely kevent/epoll (if it won't be based on top of DispatchSources/DispatchIO which do the eventing out of the box, obviously also implemented with kevent/epoll).
>> 
>> 
>>> Maybe we have some misunderstanding here. I’m not saying a synchronous api that happens to be able to handle a vector of sockets in single call without blocking anything, I’m saying a synchronous api that can just do one simple thing, which is, read/write in a synchronous way despite block or not, if it will block, just let them know by throwing an exception, the api call itself, will not block anything that way. 
>> 
>> there may well be a misunderstanding here. No one wants to take all synchronous APIs away from you. They are available in Swift today and will remain there tomorrow.
>> 
>> But we're trying to design a HTTP API that is implementable with reasonable performance and that I believe should be done by only offering async APIs.
> 
> That’s why I said “Ppl can always fall back to C API but that won’t do any good for serverside swift”. What the only thing I think a very simple synchronous API should at least provide is that people can use it and play nicely with the provided API without breaking anything. (read/write that's transparent to the library, messing up socket options etc).

I can't follow you here, sorry.


-- Johannes

> Cheers,
> Michael
> 
> 
> 
>