Theory Lexer_Command
theory Lexer_Command
imports TIGER_Pattern TIGER_Rules
keywords "lexer" :: thy_decl
and "modes" "start" "exclusive" "final" "after" "newline" "rules"
and "skip" "error" "push" "pop" "⇒"
begin
section ‹The lexer command›
text ‹The command @{command lexer} defines a lexer spec from a flex-like source
\<^cite>‹"paxson2026-flex-manual"›: the token type, the modes, named classes and regexes
after ‹where› (lex's definitions \<^cite>‹‹Sect.~6› in "lesk1975-lex"›) and the rules.
The spec is not evaluated. Each entry ‹c› of ‹where› becomes a constant ‹L.c›, and a
template ‹c(x, …)› is substituted where it is used.
The modes are names, which become a datatype ‹L_mode›, or a datatype whose values the
command enumerates. A rule is active in the modes its tags match (HOL patterns over the
mode type, ‹_› for any part) or, untagged, in every mode that is not exclusive (flex's
‹%x› \<^cite>‹‹Sect.~10› in "paxson2026-flex-manual"›). A scope ‹<p> (r | …)› tags each rule in it. Each mode has its active rules in
the order of the source, so priority is the file order (flex \<^cite>‹‹Sect.~7› in
"paxson2026-flex-manual"›). We write a scope with parentheses where flex has braces
(‹{› starts a proof block in Isar).
An action is ‹skip› (trivia), ‹error›, a token, a function from the lexeme to a token,
or a complete action of the type of \<^const>‹act›. A guard ‹when P› and a mode
operation ‹begin m›, ‹push m›, ‹pop› or a stack function follow. The command builds
the action with \<^const>‹mk_act›, whose hook is the function of ‹after› (the identity
by default).
The command defines ‹L.spec› by cases on the mode, with the equations ‹L.spec.simps›,
the start mode ‹L.start›, the final modes ‹L.final› (by default those that are not
exclusive) and ‹L.is_newline› from the class of ‹newline› (by default ‹[\n]›).
The command also builds the lexer ‹L_lex› of the spec. It evaluates the tables of
‹L› once per distinct rule list into constants ‹L_lex.tables_k›, whose code
equations ‹L_lex.tables_k_code› are the evaluated tables, so modes with the same
active rules share them. ‹L_lex.tables› is the array of these tables and
‹L_lex.index› maps each mode to its entry. ‹L_lex› is \<^const>‹cp_lexer› on
them from the stack of ‹L.start› and the first position. ‹L_lex.ok› holds of an
input that ‹L_lex› consumes up to a final mode, and ‹L_lex.correct› is
@{thm [source] cp_lexer_iff_lexes_in_range} for ‹L.spec› (from
‹L_lex.index_tables›, which says that the index of each mode holds its tables).
The option ‹(sketch)› stops after the spec.
The command reports each table with the modes that share it, its sizes and the time of
its evaluation, and the time of all tables. It warns at a rule that wins in no state of
the tables of a mode, naming the modes. Such a rule cannot be matched there (flex's
warning of that name \<^cite>‹‹Sect.~23› in "paxson2026-flex-manual"›), as a rule of higher priority matches each of its lexemes. The
option ‹(no_check)› drops the warning.
The option ‹(char)› builds the lexer on bytes, \<^const>‹char_lexer› on the tables
of \<^const>‹char_spec› ‹L.spec›, with ‹L_lex.correct› from
@{thm [source] char_lexer_iff_lexes}. A class of ‹L› that reaches above ‹0xFF› is
an error there (the byte instance of JFlex's ‹%8bit› \<^cite>‹‹``Input Character sets› in "klein2023-jflex-manual"›), unless it reaches the end of the
codepoints: such a class is a complement such as ‹_› or ‹[^…]›, and on bytes it is
its part below ‹0x100›.›
ML_file ‹lexer_command.ML›
ML_file ‹lexer_tables.ML›
ML_file ‹lexer_syntax.ML›
end