[swift-dev] undefining macros after including Tokens.def

[Micah Hainline]

I'm still wrapping my head around this, but we're doing some heavy
macro programming in swift/Parse/Tokens.def. We define macros such as
KEYWORD and then include the file, which allows different things to
happen based on what macros we've defined beforehand.

Because it's using macros, subsequent includes of Tokens.def will have
those same old macros defined unless they are subsequently undefined.
That is actually happening in the bottom of the Tokens.def now, but
apparently that wasn't always the case and there is some leftover code
floating around that tries to deal with it. Sometimes (see
SyntaxModel.cpp lines 98-100) we then #undef the macro afterward, in
effect manually cleaning up after ourselves. In Lexer.cpp line 546 we
define KEYWORD just to redefine it to be empty on line 602 presumably
to avoid the previous definition stepping on our toes in the next
couple of lines. Of course, the include already cleaned that up.

I think we should go through and try to be more consistent with this.
Tokens.def should have a comment block explaining exactly which macros
will be checked and just specifying they'll all be undefined
afterward, and all include usages should get rid of empty defines and
undefines. Does that make sense?