[swift-dev] Breaking change in lexing operators next to comments

[John Calsbeek]

If you treat comments as though they are not present, you can no longer reason locally about whitespace on either side of an operator. Straw example:

foo/* insert an
excerpt from War
and Peace here */!

I need to scan to the other side of the comment to determine if ! is preceded by whitespace.

There is already a list of situations in which some token is treated as whitespace for the purpose of operators in The Swift Programming Language:

For the purposes of these rules, the characters (, [, and { before an operator, the characters ), ], and } after an operator, and the characters ,, ;, and : are also considered whitespace.

There is one caveat to the rules above. If the ! or ? predefined operator has no whitespace on the left, it is treated as a postfix operator, regardless of whether it has whitespace on the right. To use the ? as the optional-chaining operator, it must not have whitespace on the left. To use it in the ternary conditional (? :) operator, it must have whitespace around both sides.

Given that, it seems more natural to me to define comments as “treated-as-whitespace” in the same way.

“Treated as not present” is also not quite the right way to word the opposite case, since comments would still separate tokens. Say you had an automated tool that deletes comments (perhaps unlikely, but let’s roll with it). “Treated as not present” says you should completely delete the comment, but that doesn’t actually work since it could still cause two separate tokens to be glued together. “Treated as whitespace” just means that you have to replace the comment with at least one character of whitespace.

-John

> On Dec 14, 2015, at 9:42 PM, Chris Lattner via swift-dev <swift-dev at swift.org> wrote:
> 
>> 
>> On Dec 14, 2015, at 8:15 PM, Jesse Rusak via swift-dev <swift-dev at swift.org <mailto:swift-dev at swift.org>> wrote:
>> 
>> Hi all,
>> 
>> I’m investigating this bug: https://bugs.swift.org/browse/SR-186 <https://bugs.swift.org/browse/SR-186>
>> 
>> Which appears to be a result of the fact that the logic that determines if an operator is prefix/postfix/binary does not treat comments as whitespace. So, for example:
>> 
>> /* comment */!foo
>> 
>> does not lex as expected because the “!" thinks it has an something on both sides and so is treated as a binary operator. 
>> 
>> Fixing this (by treating comments as whitespace here) will break existing code which relies on the current behavior, such as:
>> 
>> foo/* comment */!
>> 
>> which currently treats the “!” as a postfix operator but will change to binary. I expect these cases would be rare (maybe in some generated code?), but the results might be pretty confusing. 
>> 
>> Any objections to fixing this or other thoughts? 
> 
> There are two defensible models here:
> 
> 1) comments should be treated as whitespace.
> 2) comments should be treated as if they were not present.
> 
> The later model seems more ideal to me (because you can put whitespace on either side of the comment after all), but I don’t have a strong opinion about that.  What do others think?
> 
> -Chris
> 
>  _______________________________________________
> swift-dev mailing list
> swift-dev at swift.org <mailto:swift-dev at swift.org>
> https://lists.swift.org/mailman/listinfo/swift-dev <https://lists.swift.org/mailman/listinfo/swift-dev>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-dev/attachments/20151214/7752c85e/attachment.html>