33 lines
1.4 KiB
Markdown
33 lines
1.4 KiB
Markdown
[lex.token]
|
||
|
||
# 5 Lexical conventions [[lex]](./#lex)
|
||
|
||
## 5.10 Tokens [lex.token]
|
||
|
||
[token:](#nt:token "5.10 Tokens [lex.token]")
|
||
[*identifier*](lex.name#nt:identifier "5.11 Identifiers [lex.name]")
|
||
[*keyword*](lex.key#nt:keyword "5.12 Keywords [lex.key]")
|
||
[*literal*](lex.literal.kinds#nt:literal "5.13.1 Kinds of literals [lex.literal.kinds]")
|
||
[*operator-or-punctuator*](lex.operators#nt:operator-or-punctuator "5.8 Operators and punctuators [lex.operators]")
|
||
|
||
[1](#1)
|
||
|
||
[#](http://github.com/Eelis/draft/tree/9adde4bc1c62ec234483e63ea3b70a59724c745a/source/lex.tex#L873)
|
||
|
||
There are five kinds of tokens: identifiers, keywords, literals,[12](#footnote-12 "Literals include strings and character and numeric literals.") operators, and other separators[.](#1.sentence-1)
|
||
|
||
Comments and the characters U+0020 space, U+0009 character tabulation,U+000b line tabulation, U+000c form feed, and new-line
|
||
(collectively, âwhitespaceâ), as described below, are ignored except
|
||
as they serve to separate tokens[.](#1.sentence-2)
|
||
|
||
[*Note [1](#note-1)*:
|
||
|
||
Whitespace can separate otherwise adjacent identifiers, keywords, numeric
|
||
literals, and alternative tokens containing alphabetic characters[.](#1.sentence-3)
|
||
|
||
â *end note*]
|
||
|
||
[12)](#footnote-12)[12)](#footnoteref-12)
|
||
|
||
Literals include strings and character and numeric literals[.](#footnote-12.sentence-1)
|