Theory TIGER_Lexer.Lexer_Check
theory Lexer_Check
imports Lexer_Command
keywords "lexer_check" :: thy_decl and "lex" :: diag
begin
section ‹Edit-time tools: a test command and a lexing diagnostic›
text ‹⬚‹lexer_check ‹t⇩1› … ‹t⇩n›› (also separated by ⬚‹and›) evaluates
each closed boolean term ‹t⇩i› in the ML compiler, as ‹by eval› would, and fails at
the first that is not ‹True›. For an equation it shows the value of the left-hand
side. It proves nothing: it replaces ⬚‹lemma … by eval› where the lemma is not
used afterwards.
The first check that mentions a constant compiles the constant's whole program into an
ML structure of the theory and registers its datatypes for the evaluation target, as
⬚‹code_reflect› does. A check takes the compiled constants out of its term as
arguments, generates code only for what is left, and applies it to the compiled values.
So the table literals of a lexer are compiled once per theory chain, not once per check.
A constant with sort constraints is not compiled (its code takes dictionaries). The
command is the test command ‹parser_check› of the LR parser generator ‹grammar›
(‹tests/parser_check.ML›), renamed.
⬚‹lex L_lex ‹text›› lexes the text with a lexer of @{command lexer} (on
codepoints, or on the UTF-8 bytes of the text for a lexer on bytes) and prints a line per
token: its kind, line and column (from 0), lexeme and trailing trivia. Then it prints the
residue and the final stack, with whether its top is final in ‹L›. Each lexeme of the
text shows its kind on hover. A lexer compiled by an earlier @{command lexer_check} is
not compiled again.›
text ‹What ⬚‹lex› prints of a result: the length of the leading trivia, per token
its kind, line, column, offset, length and length of trivia, the length of the residue,
the final stack and whether its top satisfies ‹fin›:›
definition lex_summary ::
‹('k, 'ch) lexed × 'ch list × 'm list ⇒ ('m ⇒ bool) ⇒
nat × ('k × nat × nat × nat × nat × nat) list × nat × 'm list × bool›
where
‹lex_summary x fin = (case x of (l, r, st) ⇒
(length (header l),
map (λt. (tok t, s_line (span t), s_col (span t), s_off (span t), s_len (span t),
length (trivia t))) (toks l),
length r, st, st ≠ [] ∧ fin (hd st)))›
ML_file ‹lexer_check.ML›
end