[swift-dev] Testing fails in GYBUnicodeDataUtils.py

Ryan Lovelett swift-dev at ryan.lovelett.me
Mon Jan 4 16:44:56 CST 2016 

I would suggest setting the environment variables and re-running but
that is just my opinion.

I went through many of the same issues on Arch as well. Arch uses Python
2.7.11 and Python 3.5.1; with 3.5.1 being default.

I can _maybe_ give you a clue on the lldb issue. Check out SR-14 [1]
which links to the upstream LLDB Bug 25744 [2]. That report seems to
indicate there is a known bug compiling on Gentoo. They've even created
a specific bug Bug 25866 [3] which Bug 25744 depends on to track fixing
it in Gentoo.

Unfortunately, it seems that the experimental patch submitted there
(which I use to build on Arch) seems to only work on Arch. However,
maybe your just the person to fix that bug! Anyways food for thought.
Happy compiling. 🍻

[1] https://bugs.swift.org/browse/SR-14
[2] https://llvm.org/bugs/show_bug.cgi?id=25744
[3] https://llvm.org/bugs/show_bug.cgi?id=25866

On Mon, Jan 4, 2016, at 05:29 PM, Tom Gall wrote:
> Hi Ryan,
> 
> In my case I'm on Python 2.7.  Your comment is interesting as I was
> just tracking down why python-config --libs and python-config
> --includes doesn't seem to be used to determine what is the system
> python install. I was just starting to trace through the build tool to
> figure out how build-script works it's initial magic.
> 
> In my case I have both python 3.4 and 2.7 installed but python 2.7 is
> the system default.  This ends up causing some interesting brand of
> hurt, when trying to build swift's lldb. (Test Case
> 'TestNSTimer.test_timerTickOnce' is freezing so was looking to debug
> that)
> 
> To answer some of your questions:
> 
> tgall at mars ~/swift $ locale
> 
> LANG=en_US
> LC_CTYPE=C
> LC_NUMERIC="en_US"
> LC_TIME="en_US"
> LC_COLLATE="en_US"
> LC_MONETARY="en_US"
> LC_MESSAGES="en_US"
> LC_PAPER="en_US"
> LC_NAME="en_US"
> LC_ADDRESS="en_US"
> LC_TELEPHONE="en_US"
> LC_MEASUREMENT="en_US"
> LC_IDENTIFICATION="en_US"
> LC_ALL=
> 
> >>> import sys
> >>> x=sys.getfilesystemencoding()
> >>> print x
> ANSI_X3.4-1968
> 
> That explains some things :-)
> 
> On Mon, Jan 4, 2016 at 4:12 PM, Ryan Lovelett via swift-dev
> <swift-dev at swift.org> wrote:
> > On Mon, Jan 4, 2016, at 03:40 PM, Tom Gall via swift-dev wrote:
> >> Building with: ./swift/utils/build-script -R -t --foundation
> >>
> >> on Linux (gentoo amd64) fails with
> >>
> >> + /usr/bin/cmake --build
> >> /home/tgall/swift/build/Ninja-ReleaseAssert/swift-linux-x86_64 -- -j4
> >> SwiftUnitTests
> >>
> >> [6/29] Generating UnicodeGraphemeBreakTest.cpp from
> >> UnicodeGraphemeBreakTest.cpp.gyb with ptr size = 8
> >>
> >> FAILED: cd /home/tgall/swift/swift/unittests/Basic && /usr/bin/cmake
> >> -E make_directory
> >> /home/tgall/swift/build/Ninja-ReleaseAssert/swift-linux-x86_64/unittests/Basic/8
> >> && /home/tgall/swift/swift/utils/gyb --test
> >> -DunicodeGraphemeBreakPropertyFile=/home/tgall/swift/swift/utils/UnicodeData/GraphemeBreakProperty.txt
> >> -DunicodeGraphemeBreakTestFile=/home/tgall/swift/swift/utils/UnicodeData/GraphemeBreakTest.txt
> >> -DCMAKE_SIZEOF_VOID_P=8 -o
> >> /home/tgall/swift/build/Ninja-ReleaseAssert/swift-linux-x86_64/unittests/Basic/8/UnicodeGraphemeBreakTest.cpp.tmp
> >> UnicodeGraphemeBreakTest.cpp.gyb && /usr/bin/cmake -E
> >> copy_if_different
> >> /home/tgall/swift/build/Ninja-ReleaseAssert/swift-linux-x86_64/unittests/Basic/8/UnicodeGraphemeBreakTest.cpp.tmp
> >> /home/tgall/swift/build/Ninja-ReleaseAssert/swift-linux-x86_64/unittests/Basic/8/UnicodeGraphemeBreakTest.cpp
> >> && /usr/bin/cmake -E remove
> >> /home/tgall/swift/build/Ninja-ReleaseAssert/swift-linux-x86_64/unittests/Basic/8/UnicodeGraphemeBreakTest.cpp.tmp
> >>
> >> Traceback (most recent call last):
> >>
> >>   File "/home/tgall/swift/swift/utils/gyb", line 3, in <module>
> >>     gyb.main()
> >>   File "/home/tgall/swift/swift/utils/gyb.py", line 1071, in main
> >>     args.target.write(executeTemplate(ast, args.line_directive,
> >>     **bindings))
> >>   File "/home/tgall/swift/swift/utils/gyb.py", line 974, in
> >>   executeTemplate
> >>     ast.execute(executionContext)
> >>   File "/home/tgall/swift/swift/utils/gyb.py", line 591, in execute
> >>     x.execute(context)
> >>   File "/home/tgall/swift/swift/utils/gyb.py", line 667, in execute
> >>     result = eval(self.code, context.localBindings)
> >>   File
> >>   "/home/tgall/swift/swift/unittests/Basic/UnicodeGraphemeBreakTest.cpp.gyb",
> >> line 23, in <module>
> >>     get_grapheme_cluster_break_tests_as_UTF8(unicodeGraphemeBreakTestFile)
> >>   File "/home/tgall/swift/swift/utils/GYBUnicodeDataUtils.py", line
> >> 553, in get_grapheme_cluster_break_tests_as_UTF8
> >>     for line in f:
> >>   File "/usr/lib64/python2.7/codecs.py", line 687, in next
> >>     return self.reader.next()
> >>   File "/usr/lib64/python2.7/codecs.py", line 618, in next
> >>     line = self.readline()
> >>   File "/usr/lib64/python2.7/codecs.py", line 533, in readline
> >>     data = self.read(readsize, firstline=True)
> >>   File "/usr/lib64/python2.7/codecs.py", line 480, in read
> >>     newchars, decodedbytes = self.decode(data, self.errors)
> >> UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position
> >> 0: ordinal not in range(128)
> >> [6/29] Building CXX object
> >> unittests/Parse/CMakeFiles/SwiftParseTests.dir/LexerTests.cpp.o
> >> ninja: build stopped: subcommand failed.
> >>
> >> Ah yes ... the joys of python stack dumps...  anyway, tracing this a bit:
> >>
> >> in swift/utils/GYBUnicodeDataUtils.py there is:
> >>
> >> with codecs.open(grapheme_break_test_file_name,
> >> encoding=sys.getfilesystemencoding(), errors='strict') as f:
> >>
> >
> > I wrote that code and patch (see:
> > https://github.com/apple/swift/commit/7dbb4127f55022bca7b191d448652b5decf8626e).
> > The change was in service of adding Python 3 support to GYB. So first of
> > all let me say: I'm sorry. 😏
> >
> > Open up your python interpreter and figure out what your filesystem is
> > reporting its encoding to be (e.g., `sys.getfilesystemencoding()`). On
> > OS X and my copy of Arch linux it reports `'utf-8'` which is why it
> > doesn't have an issue. Worst case scenario we can just force it to be
> > `with codecs.open(grapheme_break_test_file_name, encoding='utf-8',
> > errors='strict') as f:` but I went with the filesystem encoding because
> > hopefully it is always UTF-8.
> >
> >> It appears to be our offending bit of python code. Now my unicode &
> >> python foo isn't the strongest, but if I change what is passed as
> >> encoding to : encoding='utf-8', the swift testcases seem to run quite
> >> a bit better and end up reporting :
> >>
> >> Testing Time: 65.82s
> >>   Expected Passes    : 1748
> >>   Expected Failures  : 83
> >>   Unsupported Tests  : 585
> >> -- check-swift-linux-x86_64 finished --
> >> --- Finished tests for swift ---
> >>
> >> Question is, is that little fix the 'right thing' (TM) ?  If so happy
> >> to submit this as my first 'lame' patch.
> >>
> >> Thanks
> >>
> >> --
> >> Regards,
> >> Tom
> >>
> >> "Where's the kaboom!? There was supposed to be an earth-shattering
> >> kaboom!" Marvin Martian
> >> Director, Linaro Mobile Group
> >> Tech Lead, GPGPU
> >> Linaro.org │ Open source software for ARM SoCs
> >> irc: tgall_foo | skype : tom_gall
> >> _______________________________________________
> >> swift-dev mailing list
> >> swift-dev at swift.org
> >> https://lists.swift.org/mailman/listinfo/swift-dev
> > _______________________________________________
> > swift-dev mailing list
> > swift-dev at swift.org
> > https://lists.swift.org/mailman/listinfo/swift-dev
> 
> 
> 
> -- 
> Regards,
> Tom
> 
> "Where's the kaboom!? There was supposed to be an earth-shattering
> kaboom!" Marvin Martian
> Director, Linaro Mobile Group
> Tech Lead, GPGPU
> Linaro.org │ Open source software for ARM SoCs
> irc: tgall_foo | skype : tom_gall

More information about the swift-dev mailing list