[swift-corelibs-dev] libdispatch roadmap and api addition proposal

[Daniel A. Steffen]

> On Dec 8, 2015, at 6:11, Joakim Hassila <Joakim.Hassila at orc-group.com> wrote:
> 
> Hi Daniel,
> 
> 
>> On 7 dec. 2015, at 23:11, Daniel A. Steffen <das at apple.com> wrote:
>> 
>> FWIW I’ve updated the macosforge svn repo trunk to match with github swift-corelibs-libdispatch trunk (sans the PRs, excecpt for my buildsystem one), but going forward we are likely going to retire the macosforge repository in favor of the github one.
> 
> That seems very reasonable and would make sense I think, there doesn’t seem to be much rationale for overlap.
> 
>> 
>>> 
>>> I have a few questions on how (particularly Apple folks) view this going forward:
>>> 
>>> First, the previous port to Linux/Solaris of libdispatch was dependent on libkqueue and more importantly on libpthread_workqueue (to have some heuristics for managing the number of threads when lacking kernel support).
>>> 
>>> How do you view this, would you consider integrating support for libpthread_workqueue, or would you have another preference for how to manage this on other platforms (Linux for starters, but essentially any lacking the pthread_workqueue interface)?
>> 
>> yes, staying with libpthread_workqueue is the focus of the current Linux porting effort, but it may make sense to move to something more native over time, e.g. like on FreeBSD where a version of the kernel workqueue was implemented natively.
> 
> Ok, that’s great - previously there was a discussion to actually integrate libpthread_workqueue at least directly into the libdispatch project to reduce the number of dependencies to get a reasonably working libdispatch running - currently Mark Heily put it up on GitHub as well at https://github.com/mheily/libpwq - it has been quite dormant for the last few years, but I think that is largely due to things working reasonably well.
> 
> So would such more close integration be desirable to make things build more out of the box, or would you prefer to only use it if found during build time on the current host? (I would probably prefer the first option, as it essentially just provides support for functionality that the underlying platform lacks - the current libpwq supports a few platforms…).

That seems like a good idea in principle, I agree that it makes good technical sense given libdispatch is presumably the only client of this library, but short term continuing to keep it separate will likely be easiest (for boring non-technical reasons)

In particular I’ll have to figure out what the situation would be with us continuing to take changes internally from the github repo after importing a whole contributed project into it (as opposed to incremental patches to the existing sourcebase), ideally I would really prefer to not significantly diverge from our internal repo to make that process as straightforward as possible (essentially a git merge…)


> 
>> Secondly, we have extended the public libdispatch API internally with one more flavor of dispatching, let’s call it ‘dispatch_async_inline’ - the semantics being ‘perform the processing of the work synchronously if we wouldn’t block the calling thread, if we would block, instead perform the work as a normal dispatch_async’.
>>> 
>>> Would such a change be considered to be integrated, or should we keep our internal diffs indefinitely? Just to understand if it is worth the effort with a nicely packaged pull request or not...
>>> 
>>> The rationale for the API is that we are quite latency sensitive and want to use inline processing up until the point where we can’t keep up with the available work, at which point we would switch to asynchronous processing seamlessly (we have multiple producers). This means that the thread calling this API can be stolen for a significant amount of time (emptying the queue it was assigned to), but when the system is under ‘light' load, we don’t need to incur the wakeup penalty for a completely asynchronous dispatch.
>> 
>> sounds familiar, have we talked to about this in the past somewhere ?
> 
> Well, it could well be that we touched upon it on the old libdispatch mailing list a few years ago (I did change my surname from Johansson -> Hassila as well as the company mail address, so it might have thrown things off for you :-). I did primarily spend some time in helping clean things up for usage on Solaris at that time though.
> 
>> 
>> we actually have something quite similar internal to the library already: _dispatch_barrier_trysync_f
>> 
>>      https://github.com/apple/swift-corelibs-libdispatch/blob/master/src/queue.c#L3089
>> 
>> but it currently (intentionally) ignores anything about the target queue hierarchy of the queue passed in (e.g. it will allow the sync even if the target queue is busy or suspended), so is not suitable as a general facility.
>> 
>> There are various technical reasons why we don’t believe this primitive in all generality is a good idea, Pierre is writing up an email about that so I won’t go into details here.
> 
> Thanks! I will reply to that separately.
> 
> Cheers,
> 
> Joakim
> 
> 
> ________________________________
> 
> This e-mail is confidential and may contain legally privileged information. It is intended only for the addressees. If you have received this e-mail in error, kindly notify us immediately by telephone or e-mail and delete the message from your system.