Theory TIGER_Lexer.Lexer_Command

(* Author: Basil Fürer *)

theory Lexer_Command
  imports TIGER_Pattern TIGER_Rules
  keywords "lexer" :: thy_decl
    and "modes" "start" "exclusive" "final" "after" "newline" "rules"
    and "skip" "error" "push" "pop" "⇒"
begin

section ‹The lexer command›

text ‹The command @{command lexer} defines a lexer spec from a flex-like source
  cite‹"paxson2026-flex-manual"›: the token type, the modes, named classes and regexes
  after ‹where› (lex's definitions cite‹‹Sect.~6› in "lesk1975-lex"›) and the rules.
  The spec is not evaluated. Each entry ‹c› of ‹where› becomes a constant ‹L.c›, and a
  template ‹c(x, …)› is substituted where it is used.

  The modes are names, which become a datatype ‹L_mode›, or a datatype whose values the
  command enumerates. A rule is active in the modes its tags match (HOL patterns over the
  mode type, ‹_› for any part) or, untagged, in every mode that is not exclusive (flex's
  ‹%x› cite‹‹Sect.~10› in "paxson2026-flex-manual"›). A scope ‹<p> (r | …)› tags each rule in it. Each mode has its active rules in
  the order of the source, so priority is the file order (flex cite‹‹Sect.~7› in
  "paxson2026-flex-manual"›). We write a scope with parentheses where flex has braces
  (‹{› starts a proof block in Isar).

  An action is ‹skip› (trivia), ‹error›, a token, a function from the lexeme to a token,
  or a complete action of the type of const‹act›. A guard ‹when P› and a mode
  operation ‹begin m›, ‹push m›, ‹pop› or a stack function follow. The command builds
  the action with const‹mk_act›, whose hook is the function of ‹after› (the identity
  by default).

  The command defines ‹L.spec› by cases on the mode, with the equations ‹L.spec.simps›,
  the start mode ‹L.start›, the final modes ‹L.final› (by default those that are not
  exclusive) and ‹L.is_newline› from the class of ‹newline› (by default ‹[\n]›).

  The command also builds the lexer ‹L_lex› of the spec. It evaluates the tables of
  ‹L› once per distinct rule list into constants ‹L_lex.tables_k›, whose code
  equations ‹L_lex.tables_k_code› are the evaluated tables, so modes with the same
  active rules share them. ‹L_lex.tables› is the array of these tables and
  ‹L_lex.index› maps each mode to its entry. ‹L_lex› is const‹cp_lexer› on
  them from the stack of ‹L.start› and the first position. ‹L_lex.ok› holds of an
  input that ‹L_lex› consumes up to a final mode, and ‹L_lex.correct› is
  @{thm [source] cp_lexer_iff_lexes_in_range} for ‹L.spec› (from
  ‹L_lex.index_tables›, which says that the index of each mode holds its tables).
  The option ‹(sketch)› stops after the spec.

  The command reports each table with the modes that share it, its sizes and the time of
  its evaluation, and the time of all tables. It warns at a rule that wins in no state of
  the tables of a mode, naming the modes. Such a rule cannot be matched there (flex's
  warning of that name cite‹‹Sect.~23› in "paxson2026-flex-manual"›), as a rule of higher priority matches each of its lexemes. The
  option ‹(no_check)› drops the warning.

  The option ‹(char)› builds the lexer on bytes, const‹char_lexer› on the tables
  of const‹char_spec› ‹L.spec›, with ‹L_lex.correct› from
  @{thm [source] char_lexer_iff_lexes}. A class of ‹L› that reaches above ‹0xFF› is
  an error there (the byte instance of JFlex's ‹%8bit› cite‹‹``Input Character sets› in "klein2023-jflex-manual"›), unless it reaches the end of the
  codepoints: such a class is a complement such as ‹_› or ‹[^…]›, and on bytes it is
  its part below ‹0x100›.›

ML_file ‹lexer_command.ML›
ML_file ‹lexer_tables.ML›
ML_file ‹lexer_syntax.ML›

end