[swift-dev] utf-8 issues in gyb and line-directive

[Lukas Stabe]

This is a separate issue. Here are the backtraces for the two issues:

This is the issue in gyb (note that this occurs here because file is opened lazily. The real issue is a few lines before, where the argparse.FileType is instantiated)

   FAILED: cd /build/swiftc/src/swift/stdlib/public/core && /usr/bin/cmake -E make_directory /build/swiftc/src/build/buildbot_linux/swift-linux-x86_64/stdlib/public/core/8 && /build/swiftc/src/swift/utils/gyb --test -DunicodeGraphemeBreakPropertyFile=/build/swiftc/src/swift/utils/UnicodeData/GraphemeBreakProperty.txt -DunicodeGraphemeBreakTestFile=/build/swiftc/src/swift/utils/UnicodeData/GraphemeBreakTest.txt -DCMAKE_SIZEOF_VOID_P=8 -o /build/swiftc/src/build/buildbot_linux/swift-linux-x86_64/stdlib/public/core/8/Arrays.swift.tmp Arrays.swift.gyb && /usr/bin/cmake -E copy_if_different /build/swiftc/src/build/buildbot_linux/swift-linux-x86_64/stdlib/public/core/8/Arrays.swift.tmp /build/swiftc/src/build/buildbot_linux/swift-linux-x86_64/stdlib/public/core/8/Arrays.swift && /usr/bin/cmake -E remove /build/swiftc/src/build/buildbot_linux/swift-linux-x86_64/stdlib/public/core/8/Arrays.swift.tmp
   Traceback (most recent call last):
     File "/build/swiftc/src/swift/utils/gyb", line 3, in <module>
       gyb.main()
     File "/build/swiftc/src/swift/utils/gyb.py", line 1064, in main
       ast = parseTemplate(args.file.name, args.file.read())
     File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
       return codecs.ascii_decode(input, self.errors)[0]
   UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 13116: ordinal not in range(128)

After fixing this (in a way incompatible with OSX Python), I get this error in line-directive:

   FAILED: cd /build/swiftc/src/build/buildbot_linux/swift-linux-x86_64/stdlib/public/core && /usr/bin/cmake -E make_directory /build/swiftc/src/build/buildbot_linux/swift-linux-x86_64/stdlib/public/core/linux/x86_64 && /usr/bin/cmake -E make_directory /build/swiftc/src/build/buildbot_linux/swift-linux-x86_64/./lib/swift/linux/x86_64 && /build/swiftc/src/swift/utils/line-directive /build/swiftc/src/swift/stdlib/public/core/Algorithm.swift [… more files omitted for brevity …] /build/swiftc/src/swift/stdlib/public/core/VarArgs.swift /build/swiftc/src/swift/stdlib/public/core/Zip.swift /build/swiftc/src/swift/stdlib/public/core/Prespecialized.swift
   Traceback (most recent call last):
     File "/build/swiftc/src/swift/utils/line-directive", line 104, in <module>
       run()
     File "/build/swiftc/src/swift/utils/line-directive", line 92, in run
       file, line_num = map_line(m.group(1), int(m.group(2)))
     File "/build/swiftc/src/swift/utils/line-directive", line 60, in map_line
       map = fline_map(filename)
     File "/build/swiftc/src/swift/utils/line-directive", line 54, in fline_map
       map = _make_line_map(filename)
     File "/build/swiftc/src/swift/utils/line-directive", line 43, in _make_line_map
       for i, l in enumerate(input.readlines()):
     File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
       return codecs.ascii_decode(input, self.errors)[0]
   UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 6851: ordinal not in range(128)

— Lukas

> On 12 Jan 2016, at 20:31, Ryan Lovelett via swift-dev <swift-dev at swift.org> wrote:
> 
> I'm thought this was resolved in master already. Are you running the
> latest version of master?
> 
> See
> https://github.com/apple/swift/commit/97b98b193d1490685a0aaba78cccb76e2caba4c1
> 
> On Tue, Jan 12, 2016, at 01:12 PM, Lukas Stabe via swift-dev wrote:
>> I’ve tried to compile Swift in a clean Arch Linux chroot. In this
>> environment, `sys.getdefaultencoding()` does not return `utf-8`, which
>> causes gyb and line-directive to fail with non-ascii characters in input
>> files.
>> 
>> One example of such a file is Arrays.swift.gyb, which contains a
>> ≤-character
>> [here](https://github.com/apple/swift/blob/master/stdlib/public/core/Arrays.swift.gyb#L358).
>> 
>> In gyb, `argparse.FileType` is used to open input files, and
>> line-directive uses plain old `open(…)`. Sadly, both of those don’t
>> provide functionality to specify the file encoding in Python 2.7.10,
>> which is what OSX ships with.
>> 
>> I’m not that experienced in Python, so I don’t know what the pythonic way
>> to solve this would be, so if anyone with more Python experience would
>> help solve this, that would be great.
>> 
>> — Lukas
>> _______________________________________________
>> swift-dev mailing list
>> swift-dev at swift.org
>> https://lists.swift.org/mailman/listinfo/swift-dev
>> Email had 1 attachment:
>> + signature.asc
>>  1k (application/pgp-signature)
> _______________________________________________
> swift-dev mailing list
> swift-dev at swift.org
> https://lists.swift.org/mailman/listinfo/swift-dev

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 841 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <https://lists.swift.org/pipermail/swift-dev/attachments/20160112/13444168/attachment.sig>