<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title></title>
</head>
<body>
<div name="messageBodySection" style="font-size: 14px; font-family: -apple-system, BlinkMacSystemFont, sans-serif;">
<div>I submitted the first pull request. It is still a pretty large patch but most of it is defined(FreeBSD) conditionals in the right places.<br /></div>
<div><br /></div>
<div><a href="https://github.com/apple/swift-corelibs-libdispatch/pull/319">https://github.com/apple/swift-corelibs-libdispatch/pull/319</a><br /></div>
<div><br /></div>
<div>kevent is very (very, very, very!) ugly right now and 99% sure broken on Darwin. FreeBSD and Darwin diverge on a ton of type definitions on kevent structures. I had to add a lot of casting to get it working on FreeBSD which almost certainly broken it on Darwin. Maybe it should be split into a kevent_mach for Apple platforms and another for FreeBSD, but that would cause code duplication problems.</div>
<div><br /></div>
<div>As said on the pull request, the Xcode project depends on a lot of Apple internal sources which means I simply cannot build it on my Mac to make reviewing and fixing issues on Darwin a lot easier.</div>
</div>
<div name="messageReplySection" style="font-size: 14px; font-family: -apple-system, BlinkMacSystemFont, sans-serif;"><br />
On Oct 16, 2017, 4:39 PM -0200, Pierre Habouzit <phabouzit@apple.com>, wrote:<br />
<blockquote type="cite" style="margin: 5px 5px; padding-left: 10px; border-left: thin solid #1abc9c;">
<div>
<blockquote type="cite" class="" style="margin: 5px 5px; padding-left: 10px; border-left: thin solid #e67e22;">
<div class="">On Oct 16, 2017, at 8:03 AM, Rogiel Sulzbach via swift-corelibs-dev <<a href="mailto:swift-corelibs-dev@swift.org" class="">swift-corelibs-dev@swift.org</a>> wrote:</div>
<br class="Apple-interchange-newline" />
<div class="">
<div class="">
<div name="messageBodySection" style="font-size: 14px; font-family: -apple-system, BlinkMacSystemFont, sans-serif;" class="">Thank you for all the help. I will try to take a look at this during the week.
<div class=""><br class="" /></div>
<div class="">I am not even running it in swift yet, I just got it to compile (without actually working) so that I could get Foundation to build without having to disable the entire URL system, but I would like to use dispatch.</div>
<div class=""><br class="" /></div>
<div class="">The reason I am going after runloops is because it was the first test to fail and many tests seem to depend on it. It seemed like a logical place to start porting.</div>
</div>
</div>
</div>
</blockquote>
<div><br class="" /></div>
It really isn'tm it's a high level construct that needs a TON of dispatch working for it to work, I'd start with simpler tests first. and sample code you write here and there, that only use pure dispatch, no runloops.</div>
<div><br class="" /></div>
<div>Focus on sources and the thread pool, this is where your issues will arise first. Once that is stable, you can move to runloops as it is built atop of the rest.<br class="" />
<div><br class="" /></div>
<br class="" />
<blockquote type="cite" class="" style="margin: 5px 5px; padding-left: 10px; border-left: thin solid #e67e22;">
<div class="">
<div class="">
<div name="messageBodySection" style="font-size: 14px; font-family: -apple-system, BlinkMacSystemFont, sans-serif;" class="">
<div class="">I think i get it now. Ill have to manually pass the fd to the kevent implementation. Makes sense. As soon as I get things starting to work i'll fork and keep a separate branch with these changes to ease review.</div>
</div>
</div>
</div>
</blockquote>
<div><br class="" /></div>
<div>It's also for your sake, if during review was ask you to do things differently or find some issue, then it's very likely that there will be ripple effect in the rest of your porting.</div>
<div><br class="" /></div>
<div>I encourage you to do focused changes (not small in terms of number of files touched, but small in terms of scope). It'll also allow us to check that it doesn't regress other ports which is always a concern ;)</div>
<br class="" />
<blockquote type="cite" class="" style="margin: 5px 5px; padding-left: 10px; border-left: thin solid #e67e22;">
<div class="">
<div class="">
<div name="messageBodySection" style="font-size: 14px; font-family: -apple-system, BlinkMacSystemFont, sans-serif;" class=""></div>
<div name="messageReplySection" style="font-size: 14px; font-family: -apple-system, BlinkMacSystemFont, sans-serif;" class=""><br class="" />
On Oct 12, 2017, 9:08 PM -0300, Pierre Habouzit via swift-corelibs-dev <<a href="mailto:swift-corelibs-dev@swift.org" class="">swift-corelibs-dev@swift.org</a>>, wrote:<br class="" />
<blockquote type="cite" style="margin: 5px 5px; padding-left: 10px; border-left: thin solid #3498db;" class="">oh and last, while working on FreeBSD, we are killing support for pthread_workqueue_additem_np, pthread_workqueue_setdispatch_np and similar functions that are ancient and do not support all the required prioirties. in your port, please do not try to repair this support if it is broken, because we're killing it anyway, that would be wasted time.
<div class=""><br class="" /></div>
<div class="">The minimal workqueue implementation that we'll support requires implementation of:</div>
<div class="">- _pthread_workqueue_init</div>
<div class="">- _pthread_workqueue_addthreads<br class="" />
<div class=""><br class="" /></div>
<div class="">Until FreeBSD supports this, the same internal workqueue implementation Linux has can be used. These interfaces above are not that different in terms of kernel requirements from the old interfaces and shouldn't be too difficult to support upstream long term from the original pthread_workqueue_additem_np era APIs FreeBSD had support for originally.</div>
<div class=""><br class="" /></div>
<div class="">Anything older than that is about to be removed.</div>
<div class=""><br class="" />
<div class="">-Pierre</div>
<div class=""><br class="" />
<blockquote type="cite" class="" style="margin: 5px 5px; padding-left: 10px; border-left: thin solid #d35400;">
<div class="">On Oct 12, 2017, at 4:56 PM, Pierre Habouzit via swift-corelibs-dev <<a href="mailto:swift-corelibs-dev@swift.org" class="">swift-corelibs-dev@swift.org</a>> wrote:</div>
<br class="Apple-interchange-newline" />
<div class="">
<div style="font-family: SFMono-Light; font-size: 11px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<blockquote type="cite" class="" style="margin: 5px 5px; padding-left: 10px; border-left: thin solid #34495e;">
<div class="">On Oct 12, 2017, at 4:52 PM, Pierre Habouzit via swift-corelibs-dev <<a href="mailto:swift-corelibs-dev@swift.org" class="">swift-corelibs-dev@swift.org</a>> wrote:</div>
<br class="Apple-interchange-newline" />
<div class="">
<div class="" style="font-family: SFMono-Light; font-size: 11px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">
<blockquote type="cite" class="" style="margin: 5px 5px; padding-left: 10px; border-left: thin solid #2ecc71;">
<div class="">On Oct 12, 2017, at 7:48 AM, Rogiel Sulzbach <<a href="mailto:rogiel@rogiel.com" class="">rogiel@rogiel.com</a>> wrote:</div>
<br class="Apple-interchange-newline" />
<div class="">
<div class="">
<div name="messageBodySection" class="" style="font-size: 14px; font-family: -apple-system, BlinkMacSystemFont, sans-serif;">
<div class="" style="margin: 0px; padding: 0px; color: rgb(51, 51, 51); font-family: Arial, sans-serif;">I am trying to get libdispatch working on FreeBSD but I am hitting a wall: I cannot get my head around how the runloop implementation dispatches a event to the <b class="">kqueue</b> once a new job is submitted. So far I was able to get the library to fully compile but most tests are timing out due to a issue getting the kqueue event handler to awake when the <b class="">poke</b> method is called.</div>
<p class="" style="margin: 10px 0px 0px; padding: 0px; color: rgb(51, 51, 51); font-family: Arial, sans-serif;">Looking at the Linux implementation (<a href="https://github.com/apple/swift-corelibs-libdispatch/blob/master/src/queue.c#L6119)" rel="nofollow" class="external-link" style="color: rgb(59, 115, 175); text-decoration: none; cursor: pointer;">https://github.com/apple/swift-corelibs-libdispatch/blob/master/src/queue.c#L6119)</a> it appears to be creating a new <b class="">eventfd</b> descriptor and calls <b class="">eventfd_write</b> on the <b class="">poke</b> method. A <b class="">eventfd_read</b> call is only made in the <b class="">event_epoll.c</b> implementation file BUT I am unable to find where the specific fd (created in <b class="">queue.c</b>) is forwarded to the epool handler. In other words, <ins class="">how is the event queue implementation informed of that file descriptor?</ins></p>
<p class="" style="margin: 10px 0px 0px; padding: 0px; color: rgb(51, 51, 51); font-family: Arial, sans-serif;">On the Darwin side of things, it appears to use a mix of <b class="">kqueue</b> (on the event handling side) and mach ports (on the signaling side), but i still cannot understand how the <b class="">event_kevent.c</b> implementation becomes aware of that port allocated on <b class="">queue.c</b>.</p>
<p class="" style="margin: 10px 0px 0px; padding: 0px; color: rgb(51, 51, 51); font-family: Arial, sans-serif;">In order do get dispatch working on FreeBSD I need to be able to understand the flow that happens between <b class="">_dispatch_runloop_queue_class_poke</b> and a event getting fired in the kqueue implementation (which I believe should be happening in <b class="">_dispatch_kq_poll</b>).</p>
<p class="" style="margin: 10px 0px 0px; padding: 0px; color: rgb(51, 51, 51); font-family: Arial, sans-serif;"><br class="" /></p>
<p class="" style="margin: 10px 0px 0px; padding: 0px;"><font color="#333333" face="Arial, sans-serif" class="">Pierre Habouzit replied to me on the bug tracker:</font></p>
<blockquote class="" style="margin: 5px 5px; padding-left: 10px; border-left: thin solid #9b59b6;">
<p class="" style="margin: 10px 0px 0px; padding: 0px;"><span class="" style="color: rgb(51, 51, 51); font-family: Arial, sans-serif; background-color: rgb(255, 255, 255);">The equivalent of the eventfd on kevent systems is to use an EVFILT_USER. It is very likely that the current implementation using kevent is badly broken on FreeBSD it has never run for a very long time. I have no access to FreeBSD at this time to help though but I can help with questions on the list.</span></p>
</blockquote>
<font color="#333333" face="Arial, sans-serif" class="">Indeed, the entire library needed changes in 39 files to get compiling, let alone working! But I am getting there!</font></div>
</div>
</div>
</blockquote>
<div class=""><br class="" /></div>
<div class="">39 seems a bit excessive so I'd encourage you when you submit this awesome porting effort to try to check in individual "porting fixes" in separate branches.</div>
<div class=""><br class="" /></div>
<div class="">You should also try to get it working at the C level before you try to integrate with Swift, runloop queues are not required to get the basic library working</div>
<br class="" />
<blockquote type="cite" class="" style="margin: 5px 5px; padding-left: 10px; border-left: thin solid #2ecc71;">
<div class="">
<div class="">
<div name="messageBodySection" class="" style="font-size: 14px; font-family: -apple-system, BlinkMacSystemFont, sans-serif;">
<div class=""><font color="#333333" face="Arial, sans-serif" class=""><br class="" /></font></div>
<div class=""><font color="#333333" face="Arial, sans-serif" class="">If i am understanding it correctly, I must send a EVFILT_USER and upon receiving that event I get the corresponding unote and call dux_merge_evt. Right?</font></div>
</div>
</div>
</div>
</blockquote>
<br class="" /></div>
<div class="" style="font-family: SFMono-Light; font-size: 11px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">Hmm actually when answering on Jira I had missed that you were talking about runllops. However, the equivalent of eventfd on kevent will be EVFILT_USER.</div>
<div class="" style="font-family: SFMono-Light; font-size: 11px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><br class="" /></div>
<div class="" style="font-family: SFMono-Light; font-size: 11px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">You will need to first work on the implementation of CFRunloop itself to understand how it "blocks". It will likely park in kevent() with a kqueue fd.</div>
<div class="" style="font-family: SFMono-Light; font-size: 11px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><br class="" /></div>
<div class="" style="font-family: SFMono-Light; font-size: 11px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">The CFRunloop in question will prepare an EVFILT USER that represents "availability" of dispatch items to drain in the runloop queue. the difficulty is that unlike epoll of portsets on the Mac, the runloop queue has to know your kqueue fd. So you need to devise a way to pass it down.</div>
<div class="" style="font-family: SFMono-Light; font-size: 11px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><br class="" /></div>
<div class="" style="font-family: SFMono-Light; font-size: 11px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">Once you have done that _dispatch_runloop_queue_class_poke() on FreeBSD will issue a NOTE_TRIGGER of the EVFILT_USER which will break the CFRunloo parked in kqueue() out, and this will recognize the EVFILT_USER as being the hint that you need to drain it.</div>
<div class="" style="font-family: SFMono-Light; font-size: 11px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><br class="" /></div>
<div class="" style="font-family: SFMono-Light; font-size: 11px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">I would suggest that on FreeBSD you add a _dispatch_runloop_queue_set_kqueue() which you will store where the fd/mach port is stored on other platforms.</div>
</div>
</blockquote>
<div class=""><br class="" /></div>
it also means that you likely need to export some function shared between Swift & Dispatch to "recognize" the knote in question, you'll need to pick an ident and a udata field. You probably want to have defines in <dispatch/private.h> when on FreeBSD for this purpose, since both dispatch & CF/Swift will need to know how to form the knote in question.</div>
<div style="font-family: SFMono-Light; font-size: 11px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br class="" /></div>
<div style="font-family: SFMono-Light; font-size: 11px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">-Pierre</div>
<span style="font-family: SFMono-Light; font-size: 11px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">_______________________________________________</span><br style="font-family: SFMono-Light; font-size: 11px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="" />
<span style="font-family: SFMono-Light; font-size: 11px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">swift-corelibs-dev mailing list</span><br style="font-family: SFMono-Light; font-size: 11px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="" />
<a href="mailto:swift-corelibs-dev@swift.org" style="font-family: SFMono-Light; font-size: 11px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">swift-corelibs-dev@swift.org</a><br style="font-family: SFMono-Light; font-size: 11px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="" />
<a href="https://lists.swift.org/mailman/listinfo/swift-corelibs-dev" style="font-family: SFMono-Light; font-size: 11px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">https://lists.swift.org/mailman/listinfo/swift-corelibs-dev</a></div>
</blockquote>
</div>
<br class="" /></div>
</div>
</blockquote>
<div class=""></div>
</div>
</div>
_______________________________________________<br class="" />
swift-corelibs-dev mailing list<br class="" />
<a href="mailto:swift-corelibs-dev@swift.org" class="">swift-corelibs-dev@swift.org</a><br class="" />
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev<br class="" /></div>
</blockquote>
</div>
<br class="" /></blockquote>
<div></div>
</div>
</body>
</html>