[swift-corelibs-dev] libdispatch epoll port

[Daniel A. Steffen]

> On Dec 17, 2015, at 13:41, Pierre Habouzit via swift-corelibs-dev <swift-corelibs-dev at swift.org> wrote:
> 
> 
> -Pierre
> 
>> On Dec 17, 2015, at 1:35 PM, Pierre Habouzit <pierre at habouzit.net> wrote:
>> 
>>> On Dec 17, 2015, at 12:40 PM, Tony Parker via swift-corelibs-dev <swift-corelibs-dev at swift.org> wrote:
>>> 
>>> Hi Dzianis,
>>> 
>>>> On Dec 17, 2015, at 12:36 PM, Dzianis Fedarenka via swift-corelibs-dev <swift-corelibs-dev at swift.org> wrote:
>>>> 
>>>>>> On Dec 10, 2015, at 12:42 AM, Joakim Hassila via swift-corelibs-dev <swift-corelibs-dev at swift.org> wrote: 
>>>>>> 
>>>>>> Hi, 
>>>>>> 
>>>>>>> On 8 dec. 2015, at 16:56, Pierre Habouzit <pierre at habouzit.net> wrote: 
>>>>>>> 
>>>>>>> FWIW, this is my personal, let’s call it enlightened, opinion, based on my knowledge of dispatch and my past extensive system programming experience with Linux before I joined Apple. 
>>>>>>> 
>>>>>>> I think that long term, the best way to maintain a Linux libdispatch port is to go away from the libkqueue that tries to emulate kqueue fully, where dispatch only needs a small subset of the surface of kqueue. Given how source.c is written today, this is not a very small undertaking, but eventually dispatch source map to epoll_ctl(EPOLLONESHOT) very very well. 
>>>>>> 
>>>>>> That makes sense, could simplify the implementation (and keep thing cleaner). Then the follow up question is of course how to split/manage source.c (as Daniel pointed out there is the merging issue). 
>>>>> we can decide when/if someone tries to tackle it. I humbly recognize that I have no great idea of how to do so.
>>>> 
>>>> I have some experience in event multiplexing programming for linux. So it looks like interesting project for me. There is some conceptual questions which I think should be discussed:
>>>> 
>>>> 1) Obviously, kqueue and epoll have a little different semantics. For example: in linux timers, signals and socket can be presented as file descriptor and processed uniformly. Is there any chance that community will agree to develop separate API for linux?
>>> 
>>> For what it’s worth, we went ahead and based CFRunLoop.c on Linux on top of epoll: 
>>> 
>>> https://github.com/apple/swift-corelibs-foundation/blob/master/CoreFoundation/RunLoop.subproj/CFRunLoop.c
>>> 
>>> https://github.com/apple/swift-corelibs-foundation/commit/d594de1bdd7f10a558e30b92809420303ded0a6a#diff-9739b4f43fc59b19e677f9e3f835d159
>>> 
>>> I think it makes total sense for dispatch’s SPI for CF to simply return an eventfd.
>> 
>> it’s exactly what we want for runloop tied queues. The mach port that is used for this on Darwin receives messages only to break out of the mach_msg() call, but the handler of the message is a void function: _dispatch_wakeup_runloop_thread().
>> 
>> The good news is that a mach_port is an uint32_t and eventfd would be an int, so as far as storage is concerned, everything is fine.
>> 
>> I would have the _dispatch_get_main_queue_port_4CF / _dispatch_runloop_root_queue_get_port_4CF return an eventfd, and adapt the code that disposes of it. This is a IMO straightforward patch that should be written e.g. that way:
>> 
>> #if HAVE_MACH
>> // current OS X Code
>> #elif HAVE_EVENTFD
>> // linux port
>> #else
>> #error should not happen
>> #endif
>> 
>> And also have:
>> 
>> DISPATCH_COCOA_COMPAT be set to one on linux (until it is, you don’t get the main queue and runloop tied queues).
>> 
>> 
>> The one murky thing is that someone has to *consume* what’s in that eventfd, today, it’s implicit with mach because MiG will call dispatch’s _dispatch_wakeup_runloop_thread() for it (corresponding to the wakeup_runloop_thread routine in protocol.defs)

actually that is never called, the only thing that is used is the mig server routine _dispatch_send_wakeup_runloop_thread, the client routine is just there so that the mig client code links…

>> , but for linux, it’s probably best if CF knows that it’s an eventfd and it has to eventfd_read() from it to consume the event before it’s calling _dispatch_runloop_root_queue_perform_4CF(). The alternative is for _dispatch_runloop_root_queue_perform_4CF() to do that read in a non blocking way, but for the cases when several things have been queued on the runloop queue and have been coalesced in a single eventfd delivery, it’s a bit dumb to pay a syscall per dequeue.
>> 
>> On Mach the coalescing happens because the port has a queue width of 1 and incoming messages are dropped when the port is full.
> 
> Actually alternatively this could be done (no error handling for clarity, but it should have some!):
> 
> 
> static bool
> _dispatch_runloop_queue_drain_one(dispatch_queue_t dq)
> {
> 	if (!dq->dq_items_tail) {
> #ifdef __linux__
> 		eventfd_read((int)dq->do_ctxt, &(eventfd_t)0);
> 		if (!dq->dq_items_tail) {
> 			return false;
> 		}
> #else
> 		return false;
> #endif
> 	}
> 	...
> }
> 
> IOW: consume the eventfd when we think the queue is empty, and check again it really is, that way you even have the eventfd break you out of the epoll while the queue is full which is probably nice (even if CF is supposed to already track this anyway, but I don’t know CFRunloop well so Tony should tell which alternative is better between my former mail or that one).

for the main queue, the current runloop / libdispatch interaction model is that __CFRunLoopRun() dequeues the message from the port obtained from _dispatch_get_main_queue_port_4CF(), and that _dispatch_main_queue_callback_4CF() consumes the message in question (aside from its storage).
If we wanted to call the mig client function mentioned above, this is where we would do it.
The main queue is then drained partially or completely, with another wakeup message generated in the partial case.

It seems that this model would make sense for eventfd as well and would keep the divergence to a minimum, though you’d likely need to emulate the coalescing behavior of the qlimit 1 wakeup port by dequeueing all additional wakeup messages on the eventfd, probably in _dispatch_main_queue_callback_4CF

for the runloop queues, the model is slightly different, the wakeup messages are supposed to be consumed by __CFRunLoopRun() and _dispatch_runloop_queue_drain_one called in a loop until it returns false, without any additional wakeup messages being generated.

Daniel