[swift-dev] Testing fails in GYBUnicodeDataUtils.py

Ryan Lovelett swift-dev at ryan.lovelett.me
Mon Jan 4 17:54:21 CST 2016 

On Mon, Jan 4, 2016, at 06:09 PM, Tom Gall wrote:
> On Mon, Jan 4, 2016 at 4:44 PM, Ryan Lovelett
> <swift-dev at ryan.lovelett.me> wrote:
> > I would suggest setting the environment variables and re-running but
> > that is just my opinion.
> >
> > I went through many of the same issues on Arch as well. Arch uses Python
> > 2.7.11 and Python 3.5.1; with 3.5.1 being default.
> 
> Well point I was suggesting is that with the use of python-config the
> version of python, include and libs location is easily determined.[4]
> Looking at CMakeCache.txt in
> build/Ninja-ReleaseAssert/swift-linux-x86_64, it's pretty obvious the
> build environment isn't using it but seems to respect the PYTHON env
> variable. I find this curious.

To me that sounds exactly what those bugs I talked about are trying to
solve; the system selecting the wrong Python from the environment.

https://bugs.swift.org/browse/SR-14?focusedCommentId=10432&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-10432

Of course, you are right I digress.

> 
> > I can _maybe_ give you a clue on the lldb issue. Check out SR-14 [1]
> > which links to the upstream LLDB Bug 25744 [2]. That report seems to
> > indicate there is a known bug compiling on Gentoo. They've even created
> > a specific bug Bug 25866 [3] which Bug 25744 depends on to track fixing
> > it in Gentoo.
> 
> Fixing CMakeCache.txt by hand to use the system's python2.7 default
> lldb builds successfully on gentoo, no changes necessary.
> 
> Still a good data point would be to switch my default python over to
> 3.x and see what kind of trouble i can find. Easy enough to do.
> 
> > Unfortunately, it seems that the experimental patch submitted there
> > (which I use to build on Arch) seems to only work on Arch. However,
> > maybe your just the person to fix that bug! Anyways food for thought.
> > Happy compiling. 🍻
> 
> > [1] https://bugs.swift.org/browse/SR-14
> > [2] https://llvm.org/bugs/show_bug.cgi?id=25744
> > [3] https://llvm.org/bugs/show_bug.cgi?id=25866
> 
> [4] https://www.python.org/dev/peps/pep-0394/
> 
> Back to the original issue. Do you want to require unicode locales?

tl;dr If you hard coded it I certainly would not object and I doubt it
would break anything. If you think its necessary than I say: go for it!

In my opinion you've already illustrated that your locale setup is in
need of tweaking and that Python is behaving as designed. Therefore, to
me, hard-coding the encoding to be UTF-8 is a work around for an
improperly configured system. But I'm not in charge of anything.

Like Jordan said those specific files will always be UTF-8 (because they
are testing unicode encoding in Swift) so it should not break anything
to force the encoding as UTF-8.

> 
> Cheers!
> 
> > On Mon, Jan 4, 2016, at 05:29 PM, Tom Gall wrote:
> >> Hi Ryan,
> >>
> >> In my case I'm on Python 2.7.  Your comment is interesting as I was
> >> just tracking down why python-config --libs and python-config
> >> --includes doesn't seem to be used to determine what is the system
> >> python install. I was just starting to trace through the build tool to
> >> figure out how build-script works it's initial magic.
> >>
> >> In my case I have both python 3.4 and 2.7 installed but python 2.7 is
> >> the system default.  This ends up causing some interesting brand of
> >> hurt, when trying to build swift's lldb. (Test Case
> >> 'TestNSTimer.test_timerTickOnce' is freezing so was looking to debug
> >> that)
> >>
> >> To answer some of your questions:
> >>
> >> tgall at mars ~/swift $ locale
> >>
> >> LANG=en_US
> >> LC_CTYPE=C
> >> LC_NUMERIC="en_US"
> >> LC_TIME="en_US"
> >> LC_COLLATE="en_US"
> >> LC_MONETARY="en_US"
> >> LC_MESSAGES="en_US"
> >> LC_PAPER="en_US"
> >> LC_NAME="en_US"
> >> LC_ADDRESS="en_US"
> >> LC_TELEPHONE="en_US"
> >> LC_MEASUREMENT="en_US"
> >> LC_IDENTIFICATION="en_US"
> >> LC_ALL=
> >>
> >> >>> import sys
> >> >>> x=sys.getfilesystemencoding()
> >> >>> print x
> >> ANSI_X3.4-1968
> >>
> >> That explains some things :-)
> >>
> >> On Mon, Jan 4, 2016 at 4:12 PM, Ryan Lovelett via swift-dev
> >> <swift-dev at swift.org> wrote:
> >> > On Mon, Jan 4, 2016, at 03:40 PM, Tom Gall via swift-dev wrote:
> >> >> Building with: ./swift/utils/build-script -R -t --foundation
> >> >>
> >> >> on Linux (gentoo amd64) fails with
> >> >>
> >> >> + /usr/bin/cmake --build
> >> >> /home/tgall/swift/build/Ninja-ReleaseAssert/swift-linux-x86_64 -- -j4
> >> >> SwiftUnitTests
> >> >>
> >> >> [6/29] Generating UnicodeGraphemeBreakTest.cpp from
> >> >> UnicodeGraphemeBreakTest.cpp.gyb with ptr size = 8
> >> >>
> >> >> FAILED: cd /home/tgall/swift/swift/unittests/Basic && /usr/bin/cmake
> >> >> -E make_directory
> >> >> /home/tgall/swift/build/Ninja-ReleaseAssert/swift-linux-x86_64/unittests/Basic/8
> >> >> && /home/tgall/swift/swift/utils/gyb --test
> >> >> -DunicodeGraphemeBreakPropertyFile=/home/tgall/swift/swift/utils/UnicodeData/GraphemeBreakProperty.txt
> >> >> -DunicodeGraphemeBreakTestFile=/home/tgall/swift/swift/utils/UnicodeData/GraphemeBreakTest.txt
> >> >> -DCMAKE_SIZEOF_VOID_P=8 -o
> >> >> /home/tgall/swift/build/Ninja-ReleaseAssert/swift-linux-x86_64/unittests/Basic/8/UnicodeGraphemeBreakTest.cpp.tmp
> >> >> UnicodeGraphemeBreakTest.cpp.gyb && /usr/bin/cmake -E
> >> >> copy_if_different
> >> >> /home/tgall/swift/build/Ninja-ReleaseAssert/swift-linux-x86_64/unittests/Basic/8/UnicodeGraphemeBreakTest.cpp.tmp
> >> >> /home/tgall/swift/build/Ninja-ReleaseAssert/swift-linux-x86_64/unittests/Basic/8/UnicodeGraphemeBreakTest.cpp
> >> >> && /usr/bin/cmake -E remove
> >> >> /home/tgall/swift/build/Ninja-ReleaseAssert/swift-linux-x86_64/unittests/Basic/8/UnicodeGraphemeBreakTest.cpp.tmp
> >> >>
> >> >> Traceback (most recent call last):
> >> >>
> >> >>   File "/home/tgall/swift/swift/utils/gyb", line 3, in <module>
> >> >>     gyb.main()
> >> >>   File "/home/tgall/swift/swift/utils/gyb.py", line 1071, in main
> >> >>     args.target.write(executeTemplate(ast, args.line_directive,
> >> >>     **bindings))
> >> >>   File "/home/tgall/swift/swift/utils/gyb.py", line 974, in
> >> >>   executeTemplate
> >> >>     ast.execute(executionContext)
> >> >>   File "/home/tgall/swift/swift/utils/gyb.py", line 591, in execute
> >> >>     x.execute(context)
> >> >>   File "/home/tgall/swift/swift/utils/gyb.py", line 667, in execute
> >> >>     result = eval(self.code, context.localBindings)
> >> >>   File
> >> >>   "/home/tgall/swift/swift/unittests/Basic/UnicodeGraphemeBreakTest.cpp.gyb",
> >> >> line 23, in <module>
> >> >>     get_grapheme_cluster_break_tests_as_UTF8(unicodeGraphemeBreakTestFile)
> >> >>   File "/home/tgall/swift/swift/utils/GYBUnicodeDataUtils.py", line
> >> >> 553, in get_grapheme_cluster_break_tests_as_UTF8
> >> >>     for line in f:
> >> >>   File "/usr/lib64/python2.7/codecs.py", line 687, in next
> >> >>     return self.reader.next()
> >> >>   File "/usr/lib64/python2.7/codecs.py", line 618, in next
> >> >>     line = self.readline()
> >> >>   File "/usr/lib64/python2.7/codecs.py", line 533, in readline
> >> >>     data = self.read(readsize, firstline=True)
> >> >>   File "/usr/lib64/python2.7/codecs.py", line 480, in read
> >> >>     newchars, decodedbytes = self.decode(data, self.errors)
> >> >> UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position
> >> >> 0: ordinal not in range(128)
> >> >> [6/29] Building CXX object
> >> >> unittests/Parse/CMakeFiles/SwiftParseTests.dir/LexerTests.cpp.o
> >> >> ninja: build stopped: subcommand failed.
> >> >>
> >> >> Ah yes ... the joys of python stack dumps...  anyway, tracing this a bit:
> >> >>
> >> >> in swift/utils/GYBUnicodeDataUtils.py there is:
> >> >>
> >> >> with codecs.open(grapheme_break_test_file_name,
> >> >> encoding=sys.getfilesystemencoding(), errors='strict') as f:
> >> >>
> >> >
> >> > I wrote that code and patch (see:
> >> > https://github.com/apple/swift/commit/7dbb4127f55022bca7b191d448652b5decf8626e).
> >> > The change was in service of adding Python 3 support to GYB. So first of
> >> > all let me say: I'm sorry. 😏
> >> >
> >> > Open up your python interpreter and figure out what your filesystem is
> >> > reporting its encoding to be (e.g., `sys.getfilesystemencoding()`). On
> >> > OS X and my copy of Arch linux it reports `'utf-8'` which is why it
> >> > doesn't have an issue. Worst case scenario we can just force it to be
> >> > `with codecs.open(grapheme_break_test_file_name, encoding='utf-8',
> >> > errors='strict') as f:` but I went with the filesystem encoding because
> >> > hopefully it is always UTF-8.
> >> >
> >> >> It appears to be our offending bit of python code. Now my unicode &
> >> >> python foo isn't the strongest, but if I change what is passed as
> >> >> encoding to : encoding='utf-8', the swift testcases seem to run quite
> >> >> a bit better and end up reporting :
> >> >>
> >> >> Testing Time: 65.82s
> >> >>   Expected Passes    : 1748
> >> >>   Expected Failures  : 83
> >> >>   Unsupported Tests  : 585
> >> >> -- check-swift-linux-x86_64 finished --
> >> >> --- Finished tests for swift ---
> >> >>
> >> >> Question is, is that little fix the 'right thing' (TM) ?  If so happy
> >> >> to submit this as my first 'lame' patch.
> >> >>
> >> >> Thanks
> >> >>
> >> >> --
> >> >> Regards,
> >> >> Tom
> >> >>
> >> >> "Where's the kaboom!? There was supposed to be an earth-shattering
> >> >> kaboom!" Marvin Martian
> >> >> Director, Linaro Mobile Group
> >> >> Tech Lead, GPGPU
> >> >> Linaro.org │ Open source software for ARM SoCs
> >> >> irc: tgall_foo | skype : tom_gall
> >> >> _______________________________________________
> >> >> swift-dev mailing list
> >> >> swift-dev at swift.org
> >> >> https://lists.swift.org/mailman/listinfo/swift-dev
> >> > _______________________________________________
> >> > swift-dev mailing list
> >> > swift-dev at swift.org
> >> > https://lists.swift.org/mailman/listinfo/swift-dev
> >>
> >>
> >>
> >> --
> >> Regards,
> >> Tom
> >>
> >> "Where's the kaboom!? There was supposed to be an earth-shattering
> >> kaboom!" Marvin Martian
> >> Director, Linaro Mobile Group
> >> Tech Lead, GPGPU
> >> Linaro.org │ Open source software for ARM SoCs
> >> irc: tgall_foo | skype : tom_gall
> 
> 
> 
> -- 
> Regards,
> Tom
> 
> "Where's the kaboom!? There was supposed to be an earth-shattering
> kaboom!" Marvin Martian
> Director, Linaro Mobile Group
> Tech Lead, GPGPU
> Linaro.org │ Open source software for ARM SoCs
> irc: tgall_foo | skype : tom_gall

More information about the swift-dev mailing list