<html><head><meta http-equiv="Content-Type" content="text/html charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><br class=""><div><blockquote type="cite" class=""><div class="">On May 18, 2017, at 12:54 PM, Pushkar N Kulkarni via swift-dev <<a href="mailto:swift-dev@swift.org" class="">swift-dev@swift.org</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><font face="Verdana,Arial,Helvetica,sans-serif" size="2" class="">Over the past few months, we've seen random and intermittent failures in the URLSession tests in TestFoundation in the CI builds running on Ubuntu 16.04. Surprisingly, these failures never occur on PR builds. We've tried to reproduce them locally (on matching Ubuntu levels) and fixed locally observed issues, assuming those were the issues that broke the CI. <br class=""><font face="Verdana,Arial,Helvetica,sans-serif" size="2" class=""><font class=""><font class=""><div class="socmaildefaultfont" dir="ltr"><div class="socmaildefaultfont" dir="ltr"><div class="socmaildefaultfont" dir="ltr"><div dir="ltr" style="font-style: normal;" class=""><br class=""></div><div dir="ltr" style="font-style: normal;" class="">However, the tests continue to intermittently fail in the CI and we seemed to have reached a point where we simply aren't able to reproduce these failures locally - we tried 2/4/16CPU with 8G/16G memory running Ubuntu 16.04 on bare metal. Even if we are able to reproduce, as is in the case of a segfault, there's no way to ascertain that the local failures have the same causes (e.g backtrace for a segfault) as the ones in the CI. </div><div dir="ltr" style="font-style: normal;" class=""><br class=""></div><div dir="ltr" style="font-style: normal;" class="">I have two questions in this context:</div><div dir="ltr" style="font-style: normal;" class="">1. Is there a way to collect failure data from the CI environment - core dumps on a crash or thread stacks on a timeout (I don't think XCTest helps) ? </div></div></div></div></font></font></font></font></div></blockquote>Please send me email with build you are seeing failure with, and where on the system the core dump is located.<br class=""><blockquote type="cite" class=""><div class=""><font face="Verdana,Arial,Helvetica,sans-serif" size="2" class=""><font face="Verdana,Arial,Helvetica,sans-serif" size="2" class=""><font class=""><font class=""><div class="socmaildefaultfont" dir="ltr"><div class="socmaildefaultfont" dir="ltr"><div class="socmaildefaultfont" dir="ltr"><div dir="ltr" style="font-style: normal;" class="">2. Is there a way to obtain more information about the environment - is it virtualised, # of CPU cores, memory size, CPU/memory available for each build, what's the load (CPU/memory utilisation)? </div></div></div></div></font></font></font></font></div></blockquote>I updated the CI Jobs to display following info for each build. Please let me know if you would like see additional info.</div><div><br class=""></div><div><pre style="box-sizing: border-box; white-space: pre-wrap; word-wrap: break-word; margin-top: 0px; margin-bottom: 0px; color: rgb(192, 197, 206); font-size: 13px; background-color: rgb(43, 48, 59);" class="">========================
OS Info:
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.10
Release: 16.10
Codename: yakkety
CPU Info:
CPU op-mode(s): 32-bit, 64-bit
CPU(s): 48
Thread(s) per core: 2
Core(s) per socket: 12
Socket(s): 2
CPU family: 6
CPU MHz: 2600.000
CPU max MHz: 2600.0000
CPU min MHz: 1200.0000
Free:
total used free shared buff/cache available
Mem: 65860844 3029364 48991112 47452 13840368 62111280
Swap: 999420 271228 728192
Uptime:
19:39:19 up 195 days, 1:51, 0 users, load average: 18.37, 25.89, 42.99
Number of executors: 4
========================</pre><div class=""><br class=""></div><div class=""><a href="https://ci.swift.org/job/oss-swift-incremental-RA-linux-ubuntu-16_10/3733/consoleFull" class="">https://ci.swift.org/job/oss-swift-incremental-RA-linux-ubuntu-16_10/3733/consoleFull</a></div></div><div><br class=""></div><div>Thanks,</div><div>Mishal Shah<br class=""><blockquote type="cite" class=""><div class=""><font face="Verdana,Arial,Helvetica,sans-serif" size="2" class=""><font face="Verdana,Arial,Helvetica,sans-serif" size="2" class=""><font class=""><font class=""><div class="socmaildefaultfont" dir="ltr"><div class="socmaildefaultfont" dir="ltr"><div class="socmaildefaultfont" dir="ltr"><div dir="ltr" style="font-style: normal;" class=""><br class=""></div><div dir="ltr" style="font-style: normal;" class="">In general, I'd be really helpful to have some suggestions about how to go about debugging failures that are being observed ONLY in the CI.</div><div dir="ltr" style="font-style: normal;" class=""><br class=""></div><div dir="ltr" style="font-style: normal;" class="">Thanks!<br class=""><font face="Sans Serif, Verdana, Arial, Helvetica, sans-serif" class="">Pushkar N Kulkarni,</font></div>
<div dir="ltr" style="font-style: normal;" class=""><font face="Sans Serif, Verdana, Arial, Helvetica, sans-serif" class="">IBM Runtimes</font></div><div dir="ltr" style="font-style: normal;" class=""><font face="Sans Serif, Verdana, Arial, Helvetica, sans-serif" class=""><br class=""></font></div><div dir="ltr" class=""><font face="serif, Times New Roman, Times, serif" class=""><i class="">Simplicity is prerequisite for reliability - Edsger W. Dijkstra</i></font></div>
<div dir="ltr" style="font-style: normal; font-size: 10.5pt; font-family: Arial;" class=""><br class=""></div></div></div></div></font></font></font></font><br class="">
_______________________________________________<br class="">swift-dev mailing list<br class=""><a href="mailto:swift-dev@swift.org" class="">swift-dev@swift.org</a><br class="">https://lists.swift.org/mailman/listinfo/swift-dev<br class=""></div></blockquote></div><br class=""></body></html>