Theory Lexer_Check

(* Author: Basil Fürer *)

theory Lexer_Check
  imports Lexer_Command
  keywords "lexer_check" :: thy_decl and "lex" :: diag
begin

section ‹Edit-time tools: a test command and a lexing diagnostic›

text ‹⬚‹lexer_check ‹t1› … ‹tn›› (also separated by ⬚‹and›) evaluates
  each closed boolean term ‹ti› in the ML compiler, as ‹by eval› would, and fails at
  the first that is not ‹True›. For an equation it shows the value of the left-hand
  side. It proves nothing: it replaces ⬚‹lemma … by eval› where the lemma is not
  used afterwards.

  The first check that mentions a constant compiles the constant's whole program into an
  ML structure of the theory and registers its datatypes for the evaluation target, as
  ⬚‹code_reflect› does. A check takes the compiled constants out of its term as
  arguments, generates code only for what is left, and applies it to the compiled values.
  So the table literals of a lexer are compiled once per theory chain, not once per check.
  A constant with sort constraints is not compiled (its code takes dictionaries). The
  command is the test command ‹parser_check› of the LR parser generator ‹grammar›
  (‹tests/parser_check.ML›), renamed.

  ⬚‹lex L_lex ‹text›› lexes the text with a lexer of @{command lexer} (on
  codepoints, or on the UTF-8 bytes of the text for a lexer on bytes) and prints a line per
  token: its kind, line and column (from 0), lexeme and trailing trivia. Then it prints the
  residue and the final stack, with whether its top is final in ‹L›. Each lexeme of the
  text shows its kind on hover. A lexer compiled by an earlier @{command lexer_check} is
  not compiled again.›

text ‹What ⬚‹lex› prints of a result: the length of the leading trivia, per token
  its kind, line, column, offset, length and length of trivia, the length of the residue,
  the final stack and whether its top satisfies ‹fin›:›

definition lex_summary ::
  ‹('k, 'ch) lexed × 'ch list × 'm list ⇒ ('m ⇒ bool) ⇒
    nat × ('k × nat × nat × nat × nat × nat) list × nat × 'm list × bool›
where
  ‹lex_summary x fin = (case x of (l, r, st) ⇒
    (length (header l),
     map (λt. (tok t, s_line (span t), s_col (span t), s_off (span t), s_len (span t),
       length (trivia t))) (toks l),
     length r, st, st ≠ [] ∧ fin (hd st)))›

ML_file ‹lexer_check.ML›

end