[swift-dev] [swift-lldb-dev] [Swift CI] Build Failure: OSS - Swift Package - Ubuntu 15.10 (master) #1930

Daniel Dunbar daniel_dunbar at apple.com
Fri Aug 19 16:26:15 CDT 2016 

> On Aug 19, 2016, at 2:22 PM, Todd Fiala <todd.fiala at gmail.com> wrote:
> 
> First off, I think the whole LLDB team can agree on the simple principle that tests that sometimes fail is not something we’re shooting for, so we’re on the same page with the general sentiment.
> 
> But, rather than talking in sweeping generalizations, let’s take this exact failure as an example.  It was a socket attempting to connect.  It uses general socket timeouts.  If it’s using a default, it will typically be something like 30 seconds.  Per my previous comment, this is not something we see fail with any kind of frequency.  We just had a failure on it now.  Is the solution to say we want 35 seconds?  1 minute?  2 minutes?  We’re using end-to-end testing here, and therefore we are subject to how a real system behaves.  If we crank that up, there could be a time where we overload the server further, and the socket facility just fails.  What do you do then, claim the test as unworthy?
> 
> The challenge we run into on LLDB is that, while we do have a growing number of unit tests that are strict input/output verification, far too much of a functioning debugger’s operations are not covered adequately by that.  So we have a large body of end to end tests.  Sometimes those fail under heavy load.  Sometimes those issues are ours - we have hidden races that are only exposed when the machine is grinding to a halt.  Sometimes it is that the OS fails to handle requests for resources that the debugger needs.
> 
> So it isn’t as simple as saying “file a bug to make the LLDB test suite more reliable.”  (I wish it were!)

This is specifically why I proposed a concrete strategy like running a stress test of the test suite. That is actionable, and should shake some bugs I would hope?

Another actionable strategies (although one I'm not a big fan of) is enhance the test suite to automatically re-attempt flaky tests.

> 
> On the positive side, we have more resources going into improving our quality in the upcoming months, and we now have a dedicated quality engineer working with us.  I expect this will help things as we move forward.  And we’re looking to sink some of the system testing that currently exists end-to-end in LLDB to a more input-output style testing that we can do without needing the whole of LLDB (e.g. DWARF verification, DWARF type round-tripping through the compiler, etc. - Greg and Sean can speak more to what we’re doing there).

Great!

 - Daniel

> 
> The net effect is that we should be hitting less of this noise.
> 
> -Todd
> On August 19, 2016 at 2:10:14 PM, Daniel Dunbar via swift-lldb-dev (swift-lldb-dev at swift.org <mailto:swift-lldb-dev at swift.org>) wrote:
> 
>> The amount of flakiness in the LLDB tests is disturbing... is there a high-level bug tracking improving this? It seems like it might be worth running a stress test of the tests on a loaded machine to try and shake out such problems.
>> 
>>  - Daniel
>> 
>>> On Aug 19, 2016, at 1:58 PM, Todd Fiala via swift-lldb-dev <swift-lldb-dev at swift.org <mailto:swift-lldb-dev at swift.org>> wrote:
>>> 
>>>> 
>>>> On Aug 19, 2016, at 1:42 PM, Enrico Granata <egranata at apple.com <mailto:egranata at apple.com>> wrote:
>>>> 
>>>>> 
>>>>> On Aug 19, 2016, at 11:20 AM, Todd Fiala via swift-lldb-dev <swift-lldb-dev at swift.org <mailto:swift-lldb-dev at swift.org>> wrote:
>>>>> 
>>>>> It looks like this may be the packaging build:
>>>>> 
>>>>>>> 
>>>>>>> oss-swift-package-linux-ubuntu-15_10
>>>>> 
>>>>> It was this build:
>>>>> 
>>>>> https://ci.swift.org/job/oss-swift-package-linux-ubuntu-15_10/1930/consoleFull#465392916fca400bf-2f4a-462e-b517-e058d770b2d7 <https://ci.swift.org/job/oss-swift-package-linux-ubuntu-15_10/1930/consoleFull#465392916fca400bf-2f4a-462e-b517-e058d770b2d7>
>>>>> 
>>>>> It has since gone blue.
>>>>> 
>>>>> I don’t think I recall ever seeing that test fail before in an intermittent way, but that might be what happened here.
>>>>> 
>>>> 
>>>> Given the failure mode, it does sound likely:
>>>> 
>>>> 
>>>> File "/usr/lib/python2.7/socket.py", line 206, in accept sock, addr
>>>> = self._sock.accept() timeout: timed out
>>>> 
>>>> 
>>>> 
>>> 
>>> If we see that again, we may be able to tweak the socket timeout to allow for heavily loaded test CI.
>>> 
>>>>> -Todd
>>>>> 
>>>>>> On Aug 19, 2016, at 11:15 AM, Kate Stone via swift-lldb-dev <swift-lldb-dev at swift.org <mailto:swift-lldb-dev at swift.org>> wrote:
>>>>>> 
>>>>>> Where is this failure being reported?  I’m not seeing anything on swift-3.0-branch or master on ci.swift.org <http://ci.swift.org/>.
>>>>>> 
>>>>>> Kate Stone k8stone at apple.com <mailto:k8stone at apple.com>
>>>>>>  Xcode Low Level Tools
>>>>>> 
>>>>>>> On Aug 19, 2016, at 12:15 AM, mishal_shah via swift-lldb-dev <swift-lldb-dev at swift.org <mailto:swift-lldb-dev at swift.org>> wrote:
>>>>>>> 
>>>>>>> + swift-lldb-dev group.  
>>>>>>> 
>>>>>>>> On Aug 18, 2016, at 9:47 PM, Daniel Dunbar <daniel_dunbar at apple.com <mailto:daniel_dunbar at apple.com>> wrote:
>>>>>>>> 
>>>>>>>> Failing in lldb:
>>>>>>>> 
>>>>>>>> Command invoked:
>>>>>>>> /home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/lldb/test/dotest.py
>>>>>>>> --executable
>>>>>>>> /home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/build/buildbot_linux/lldb-linux-x86_64/bin/lldb
>>>>>>>> --rerun-all-issues -C
>>>>>>>> /home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/build/buildbot_linux/llvm-linux-x86_64/bin/clang
>>>>>>>> -s 2016-08-18-15_15_07 --results-port 41187 --inferior -p
>>>>>>>> TestStubReverseConnect.py
>>>>>>>> /home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/lldb/packages/Python/lldbsuite/test
>>>>>>>> --event-add-entries worker_index=14:int Configuration: arch=x86_64
>>>>>>>> compiler=/home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/build/buildbot_linux/llvm-linux-x86_64/bin/clang
>>>>>>>> ----------------------------------------------------------------------
>>>>>>>> Collected 2 tests
>>>>>>>> ======================================================================
>>>>>>>> ERROR: test_reverse_connect_works_llgs
>>>>>>>> (TestStubReverseConnect.TestStubReverseConnect)
>>>>>>>> ----------------------------------------------------------------------
>>>>>>>> Traceback (most recent call last): File
>>>>>>>> "/home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/lldb/packages/Python/lldbsuite/test/decorators.py",
>>>>>>>> line 121, in wrapper func(*args, **kwargs) File
>>>>>>>> "/home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/lldb/packages/Python/lldbsuite/test/decorators.py",
>>>>>>>> line 121, in wrapper func(*args, **kwargs) File
>>>>>>>> "/home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/lldb/packages/Python/lldbsuite/test/tools/lldb-server/commandline/TestStubReverseConnect.py",
>>>>>>>> line 89, in test_reverse_connect_works_llgs
>>>>>>>> self.reverse_connect_works() File
>>>>>>>> "/home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/lldb/packages/Python/lldbsuite/test/tools/lldb-server/commandline/TestStubReverseConnect.py",
>>>>>>>> line 67, in reverse_connect_works (stub_socket, address) =
>>>>>>>> self.listener_socket.accept() File "/usr/lib/python2.7/socket.py",
>>>>>>>> line 206, in accept sock, addr = self._sock.accept() timeout: timed
>>>>>>>> out
>>>>>>>> Config=x86_64-/home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/build/buildbot_linux/llvm-linux-x86_64/bin/clang-3.9
>>>>>>>> ----------------------------------------------------------------------
>>>>>>>> Ran 2 tests in 22.006s RESULT: FAILED (0 passes, 0 failures, 1
>>>>>>>> errors, 1 skipped, 0 expected failures, 0 unexpected successes)
>>>>>>>> Session logs for test failures/errors/unexpected successes can be
>>>>>>>> found in directory '2016-08-18-15_15_07'
>>>>>>>> 
>>>>>>>>  <>[TestStubReverseConnect.py
>>>>>>>> FAILED] 
>>>>>>>> Command invoked: /usr/bin/python
>>>>>>>> /home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/lldb/test/dotest.py
>>>>>>>> --executable
>>>>>>>> /home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/build/buildbot_linux/lldb-linux-x86_64/bin/lldb
>>>>>>>> --rerun-all-issues -C
>>>>>>>> /home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/build/buildbot_linux/llvm-linux-x86_64/bin/clang
>>>>>>>> -s 2016-08-18-15_15_07 --results-port 41187 --inferior -p
>>>>>>>> TestStubReverseConnect.py
>>>>>>>> /home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/lldb/packages/Python/lldbsuite/test
>>>>>>>> --event-add-entries worker_index=14:int
>>>>>>>> 
>>>>>>>>  - Daniel
>>>>>>>> 
>>>>>>>>> On Aug 18, 2016, at 9:18 PM, no-reply at swift.org <mailto:no-reply at swift.org> wrote:
>>>>>>>>> 
>>>>>>>>> [FAILURE] oss-swift-package-linux-ubuntu-15_10 [#1930]
>>>>>>>>> 
>>>>>>>>> Build URL:	https://ci.swift.org/job/oss-swift-package-linux-ubuntu-15_10/1930/ <https://ci.swift.org/job/oss-swift-package-linux-ubuntu-15_10/1930/>
>>>>>>>>> Project:	oss-swift-package-linux-ubuntu-15_10
>>>>>>>>> Date of build:	Thu, 18 Aug 2016 20:26:52 -0700
>>>>>>>>> Build duration:	51 min
>>>>>>>>> Identified problems:
>>>>>>>>> 
>>>>>>>>> Regression test failed: This build failed because a regression test in the test suite FAILed. Below is a list of all errors:
>>>>>>>>> Indication 1 <https://ci.swift.org//job/oss-swift-package-linux-ubuntu-15_10/1930/consoleFull#465392916fca400bf-2f4a-462e-b517-e058d770b2d7>
>>>>>>>>> Changes
>>>>>>>>> 
>>>>>>>>> Commit 2ebab526bfd7a5ce6ef460bebee0399fe669612d by xi_ge:
>>>>>>>>> [FixCode] Apply
>>>>>>>>> coercion fixits on return statement and initialization
>>>>>>>>> 
>>>>>>>>> edit: test/FixCode/fixits-apply-objc.swift
>>>>>>>>> edit: test/FixCode/fixits-apply-objc.swift.result
>>>>>>>>> edit: lib/Sema/CSDiag.cpp
>>>>>>>>> edit: test/FixCode/fixits-apply.swift.result
>>>>>>>>> 
>>>>>>>>> Commit 6447a2d16edcfbceab940d58a321b3ef70525f1a by daniel_dunbar:
>>>>>>>>> [Basic] Update
>>>>>>>>> Thread shim to match changes on Linux side.
>>>>>>>>> 
>>>>>>>>> edit: Sources/Basic/Thread.swift
>>>>>>>> 
>>>>>>> 
>>>>>>> _______________________________________________
>>>>>>> swift-lldb-dev mailing list
>>>>>>> swift-lldb-dev at swift.org <mailto:swift-lldb-dev at swift.org>
>>>>>>> https://lists.swift.org/mailman/listinfo/swift-lldb-dev <https://lists.swift.org/mailman/listinfo/swift-lldb-dev>
>>>>>> 
>>>>>> _______________________________________________
>>>>>> swift-lldb-dev mailing list
>>>>>> swift-lldb-dev at swift.org <mailto:swift-lldb-dev at swift.org>
>>>>>> https://lists.swift.org/mailman/listinfo/swift-lldb-dev <https://lists.swift.org/mailman/listinfo/swift-lldb-dev>
>>>>> 
>>>>> _______________________________________________
>>>>> swift-lldb-dev mailing list
>>>>> swift-lldb-dev at swift.org <mailto:swift-lldb-dev at swift.org>
>>>>> https://lists.swift.org/mailman/listinfo/swift-lldb-dev <https://lists.swift.org/mailman/listinfo/swift-lldb-dev>
>>>> 
>>>> 
>>>> Thanks,
>>>> - Enrico
>>>> 📩 egranata@ <about:blank> <about:blank>.com <about:blank> ☎️ 27683
>>> 
>>> _______________________________________________
>>> swift-lldb-dev mailing list
>>> swift-lldb-dev at swift.org <mailto:swift-lldb-dev at swift.org>
>>> https://lists.swift.org/mailman/listinfo/swift-lldb-dev <https://lists.swift.org/mailman/listinfo/swift-lldb-dev>
>> _______________________________________________
>> swift-lldb-dev mailing list
>> swift-lldb-dev at swift.org <mailto:swift-lldb-dev at swift.org>
>> https://lists.swift.org/mailman/listinfo/swift-lldb-dev <https://lists.swift.org/mailman/listinfo/swift-lldb-dev>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-dev/attachments/20160819/7a63cd2c/attachment.html>

More information about the swift-dev mailing list