[lex.phases]

# 5 Lexical conventions [[lex]](./#lex)

## 5.2 Phases of translation [lex.phases]

[1](#1)

[#](http://github.com/Eelis/draft/tree/9adde4bc1c62ec234483e63ea3b70a59724c745a/source/lex.tex#L49)

The precedence among the syntax rules of translation is specified by the
following phases[.](#1.sentence-1)[7](#footnote-7 "Implementations behave as if these separate phases occur, although in practice different phases can be folded together.")

| [1.](#1.1) | An implementation shall support input files that are a sequence of UTF-8 code units (UTF-8 files)[.](#1.1.sentence-1)<br>It may also support an implementation-defined set of other kinds of input files, and, if so, the kind of an input file is determined in an implementation-defined manner that includes a means of designating input files as UTF-8 files, independent of their content[.](#1.1.sentence-2)<br>[*Note [1](#note-1)*:<br>In other words, recognizing the U+feff byte order mark is not sufficient[.](#1.1.sentence-3) â *end note*]<br> If an input file is determined to be a UTF-8 file, then it shall be a well-formed UTF-8 code unit sequence and it is decoded to produce a sequence of Unicode[8](#footnote-8 "Unicode®&nbsp;is a registered trademark of Unicode, Inc. This information is given for the convenience of users of this document and does not constitute an endorsement by ISO or IEC of this product.") scalar values[.](#1.1.sentence-4)<br>A sequence of translation character set elements ([[lex.charset]](lex.charset "5.3.1 Character sets")) is then formed by mapping each Unicode scalar value to the corresponding translation character set element[.](#1.1.sentence-5)<br>In the resulting sequence, each pair of characters in the input sequence consisting of U+000d carriage return followed by U+000a line feed, as well as each U+000d carriage return not immediately followed by a U+000a line feed, is replaced by a single new-line character[.](#1.1.sentence-6)<br>For any other kind of input file supported by the implementation, characters are mapped, in an implementation-defined manner, to a sequence of translation character set elements, representing end-of-line indicators as new-line characters[.](#1.1.sentence-7) |
| --- | --- |
| [2.](#1.2) | If the first translation character is U+feff byte order mark, it is deleted[.](#1.2.sentence-1)<br>Each sequence of a backslash character (\) immediately followed by zero or more whitespace characters other than new-line followed by a new-line character is deleted, splicing physical source lines to form [*logical source lines*](#def:source_line,logical "5.2 Phases of translation [lex.phases]")[.](#1.2.sentence-2)<br>Only the last backslash on any physical source line shall be eligible for being part of such a splice[.](#1.2.sentence-3)<br>[*Note [2](#note-2)*:<br>Line splicing can form a [*universal-character-name*](lex.universal.char#nt:universal-character-name "5.3.2 Universal character names [lex.universal.char]") ([[lex.charset]](lex.charset "5.3.1 Character sets"))[.](#1.2.sentence-4) â *end note*]<br> A source file that is not empty and that (after splicing) does not end in a new-line character shall be processed as if an additional new-line character were appended to the file[.](#1.2.sentence-5) |
| [3.](#1.3) | The source file is decomposed into preprocessing tokens ([[lex.pptoken]](lex.pptoken "5.5 Preprocessing tokens")) and sequences of whitespace characters (including comments)[.](#1.3.sentence-1)<br>A source file shall not end in a partial preprocessing token or in a partial comment[.](#1.3.sentence-2)[9](#footnote-9 "A partial preprocessing token would arise from a source file ending in the first portion of a multi-character token that requires a terminating sequence of characters, such as a header-name that is missing the closing &quot; or >. A partial comment would arise from a source file ending with an unclosed /* comment.")<br> Each comment ([[lex.comment]](lex.comment "5.4 Comments")) is replaced by one U+0020 space character[.](#1.3.sentence-3)<br>New-line characters are retained[.](#1.3.sentence-4)<br>Whether each nonempty sequence of whitespace characters other than new-line is retained or replaced by one U+0020 space character is unspecified[.](#1.3.sentence-5)<br>As characters from the source file are consumed to form the next preprocessing token (i.e., not being consumed as part of a comment or other forms of whitespace), except when matching a [*c-char-sequence*](lex.ccon#nt:c-char-sequence "5.13.3 Character literals [lex.ccon]"), [*s-char-sequence*](lex.string#nt:s-char-sequence "5.13.5 String literals [lex.string]"), [*r-char-sequence*](lex.string#nt:r-char-sequence "5.13.5 String literals [lex.string]"), [*h-char-sequence*](lex.header#nt:h-char-sequence "5.6 Header names [lex.header]"), or [*q-char-sequence*](lex.header#nt:q-char-sequence "5.6 Header names [lex.header]"), [*universal-character-name*](lex.universal.char#nt:universal-character-name "5.3.2 Universal character names [lex.universal.char]")*s* are recognized ([[lex.universal.char]](lex.universal.char "5.3.2 Universal character names")) and replaced by the designated element of the translation character set ([[lex.charset]](lex.charset "5.3.1 Character sets"))[.](#1.3.sentence-6)<br>The process of dividing a source file's characters into preprocessing tokens is context-dependent[.](#1.3.sentence-7)<br>[*Example [1](#example-1)*:<br>See the handling of < within a #include preprocessing directive ([[lex.header]](lex.header "5.6 Header names"), [[cpp.include]](cpp.include "15.3 Source file inclusion"))[.](#1.3.sentence-8) â *end example*] |
| [4.](#1.4) | The source file is analyzed as a [*preprocessing-file*](cpp.pre#nt:preprocessing-file "15.1 Preamble [cpp.pre]") ([[cpp.pre]](cpp.pre "15.1 Preamble"))[.](#1.4.sentence-1)<br>Preprocessing directives ([[cpp]](cpp "15 Preprocessing directives")) are executed, macro invocations are expanded ([[cpp.replace]](cpp.replace "15.7 Macro replacement")), and _Pragma unary operator expressions are executed ([[cpp.pragma.op]](cpp.pragma.op "15.13 Pragma operator"))[.](#1.4.sentence-2)<br>A #include preprocessing directive ([[cpp.include]](cpp.include "15.3 Source file inclusion")) causes the named header or source file to be processed from phase 1 through phase 4, recursively[.](#1.4.sentence-3)<br>All preprocessing directives are then deleted[.](#1.4.sentence-4) |
| [5.](#1.5) | For a sequence of two or more adjacent [*string-literal*](lex.string#nt:string-literal "5.13.5 String literals [lex.string]") preprocessing tokens, a common [*encoding-prefix*](lex.ccon#nt:encoding-prefix "5.13.3 Character literals [lex.ccon]") is determined as specified in [[lex.string]](lex.string "5.13.5 String literals")[.](#1.5.sentence-1)<br>Each such [*string-literal*](lex.string#nt:string-literal "5.13.5 String literals [lex.string]") preprocessing token is then considered to have that common [*encoding-prefix*](lex.ccon#nt:encoding-prefix "5.13.3 Character literals [lex.ccon]")[.](#1.5.sentence-2) |
| [6.](#1.6) | Adjacent [*string-literal*](lex.string#nt:string-literal "5.13.5 String literals [lex.string]") preprocessing tokens are concatenated ([[lex.string]](lex.string "5.13.5 String literals"))[.](#1.6.sentence-1) |
| [7.](#1.7) | Each preprocessing token is converted into a token ([[lex.token]](lex.token "5.10 Tokens"))[.](#1.7.sentence-1)<br>Whitespace characters separating tokens are no longer significant[.](#1.7.sentence-2)<br>The resulting tokens constitute a [*translation unit*](#def:translation_unit "5.2 Phases of translation [lex.phases]") and are syntactically and semantically analyzed as a [*translation-unit*](basic.link#nt:translation-unit "6.7 Program and linkage [basic.link]") ([[basic.link]](basic.link "6.7 Program and linkage")) and translated[.](#1.7.sentence-3)<br>[*Note [3](#note-3)*:<br>The process of analyzing and translating the tokens can occasionally result in one token being replaced by a sequence of other tokens ([[temp.names]](temp.names "13.3 Names of template specializations"))[.](#1.7.sentence-4) â *end note*]<br> It is implementation-defined whether the sources for module units and header units on which the current translation unit has an interface dependency ([[module.unit]](module.unit "10.1 Module units and purviews"), [[module.import]](module.import "10.3 Import declaration")) are required to be available[.](#1.7.sentence-5)<br>[*Note [4](#note-4)*:<br>Source files, translation units and translated translation units need not necessarily be stored as files, nor need there be any one-to-one correspondence between these entities and any external representation[.](#1.7.sentence-6)<br>The description is conceptual only, and does not specify any particular implementation[.](#1.7.sentence-7) â *end note*]<br>[*Note [5](#note-5)*:<br>Previously translated translation units can be preserved individually or in libraries[.](#1.7.sentence-8)<br>The separate translation units of a program communicate ([[basic.link]](basic.link "6.7 Program and linkage")) by (for example) calls to functions whose names have external or module linkage, manipulation of variables whose names have external or module linkage, or manipulation of data files[.](#1.7.sentence-9) â *end note*]<br>While the tokens constituting translation units are being analyzed and translated, required instantiations are performed[.](#1.7.sentence-10)<br>[*Note [6](#note-6)*:<br>This can include instantiations which have been explicitly requested ([[temp.explicit]](temp.explicit "13.9.3 Explicit instantiation"))[.](#1.7.sentence-11) â *end note*]<br>The contexts from which instantiations may be performed are determined by their respective points of instantiation ([[temp.point]](temp.point "13.8.4.1 Point of instantiation"))[.](#1.7.sentence-12)<br>[*Note [7](#note-7)*:<br>Other requirements in this document can further constrain the context from which an instantiation can be performed[.](#1.7.sentence-13)<br>For example, a constexpr function template specialization might have a point of instantiation at the end of a translation unit, but its use in certain constant expressions could require that it be instantiated at an earlier point ([[temp.inst]](temp.inst "13.9.2 Implicit instantiation"))[.](#1.7.sentence-14) â *end note*]<br>Each instantiation results in new program constructs[.](#1.7.sentence-15)<br>The program is ill-formed if any instantiation fails[.](#1.7.sentence-16)<br>During the analysis and translation of tokens, certain expressions are evaluated ([[expr.const]](expr.const "7.7 Constant expressions"))[.](#1.7.sentence-17)<br>Constructs appearing at a program point P are analyzed in a context where each side effect of evaluating an expression E as a full-expression is complete if and only if <br>   [(1.7.1)](#1.7.1)E is the expression corresponding to a [*consteval-block-declaration*](dcl.pre#nt:consteval-block-declaration "9.1 Preamble [dcl.pre]") ([[dcl.pre]](dcl.pre "9.1 Preamble")), and<br>[(1.7.2)](#1.7.2)either that [*consteval-block-declaration*](dcl.pre#nt:consteval-block-declaration "9.1 Preamble [dcl.pre]") or the template definition from which it is instantiated is reachable from ([[module.reach]](module.reach "10.7 Reachability")) <br><br>[(1.7.2.1)](#1.7.2.1)P, or<br>[(1.7.2.2)](#1.7.2.2)the point immediately following the [*class-specifier*](class.pre#nt:class-specifier "11.1 Preamble [class.pre]") of the outermost class for which P is in a complete-class context ([[class.mem.general]](class.mem.general "11.4.1 General"))[.](#1.7.sentence-18) <br>[*Example [2](#example-2)*: class S {class Incomplete;   class Inner {void fn() {/* p1 */ Incomplete i; // OK}}; /* p2 */consteval {  define_aggregate(^^Incomplete, {});  }}; /* p3 */<br> Constructs at p1 are analyzed in a context where the side effect of the call to define_aggregate is evaluated because <br>   [(1.7.3)](#1.7.3)E is the expression corresponding to a consteval block, and<br>[(1.7.4)](#1.7.4)p1 is in a complete-class context of S and the consteval block is reachable from p3[.](#1.7.sentence-19) â *end example*] |
| [8.](#1.8) | Translated translation units are combined, and all external entity references are resolved[.](#1.8.sentence-1)<br>Library components are linked to satisfy external references to entities not defined in the current translation[.](#1.8.sentence-2)<br>All such translator output is collected into a program image which contains information needed for execution in its execution environment[.](#1.8.sentence-3) |

[7)](#footnote-7)[7)](#footnoteref-7)

Implementations behave as if these separate phases
occur, although in practice different phases can be folded together[.](#footnote-7.sentence-1)

[8)](#footnote-8)[8)](#footnoteref-8)

Unicode® is a registered trademark of Unicode, Inc[.](#footnote-8.sentence-1)

This information is given for the convenience of users of this document and
does not constitute an endorsement by ISO or IEC of this product[.](#footnote-8.sentence-2)

[9)](#footnote-9)[9)](#footnoteref-9)

A partial preprocessing
token would arise from a source file
ending in the first portion of a multi-character token that requires a
terminating sequence of characters, such as a [*header-name*](lex.header#nt:header-name "5.6 Header names [lex.header]") that is missing the closing " or >[.](#footnote-9.sentence-1)

A partial comment
would arise from a source file ending with an unclosed /* comment[.](#footnote-9.sentence-2)
