3.7.2 Compiler Testomatic
| (require cpsc411/test-suite/utils) | package: cpsc411-lib |
The compiler testomatic framework keeps a map from language interpreters to test programs. Each pass is associated with an source interpreter and a target interpreter. The pass is tested by running the source program in the source interpreter against the output program in the target interpreter. Any output program that is deemed valid is retained for testing later passes.
The framework also keeps an auxiliary map from interpreters to validators, and will run validators before attempting to interpret output programs.
procedure
(test-compiler-pass pass src-interp trg-interp trg-validator [ src-equiv]) → void? pass : ('a -> 'b) src-interp : ('a -> 'c) trg-interp : ('b -> 'd) trg-validator : ((or/c any/c 'b) -> boolean?) src-equiv : ('c 'd -> boolean?) = equal?
Takes a compiler pass from some language 'a to some language 'b, an interpeter for each language, and a validator that recognizes program in the language 'b. Run pass on each test source program registered with the framework, and compares the results in the respective interpreters. The results are compared using the optional src-equiv procedure, which defaults to equal?. Note that all test suites documented in the next section implicitly register tests with the framework.
test-compiler-pass executes a sequence of test-case?, and so should be run inside a test-suite, or it will have no effect.
> (define (uniquify p) (define (uniquify-tail tail) (match tail [`(let ([x ,v]) ,t) `(let ([x.1 ,v]) ,(uniquify-tail t))] ['x 'x.1] [_ tail])) (match p [`(module ,t) `(module ,(uniquify-tail t))])) > (test-compiler-pass uniquify interp-values-lang-v4 interp-values-lang-v4 values-unique-lang-v4?)
> (run-tests (test-suite "" (test-compiler-pass uniquify interp-values-lang-v4 interp-values-unique-lang-v4 values-unique-lang-v4?))) 0 success(es) 0 failure(s) 0 error(s) 0 test(s) run
0
> (register-test-programs! interp-values-lang-v4 '(("" (module 5)) ("" (module (let ([x 5]) x)))))
> (run-tests (test-suite "" (test-compiler-pass uniquify interp-values-lang-v4 interp-values-unique-lang-v4 values-unique-lang-v4?))) 2 success(es) 0 failure(s) 0 error(s) 2 test(s) run
0
> (run-tests (test-suite "" (test-compiler-pass values interp-values-lang-v4 interp-values-unique-lang-v4 values-unique-lang-v4?)))
--------------------
> values suite
FAILURE
test-program: (module (let ((x 5)) x))
expected: 5
src-interp: interp-values-lang-v4
trg-interp: interp-values-unique-lang-v4
output-program: (module (let ((x 5)) x))
test-type: "Checking that output is syntactically correct"
name: check-true
location:
/builds/packages/cpsc411-pub/cpsc411-lib/cpsc411/test-suite/utils.rkt:408:14
params: '(#f)
--------------------
1 success(es) 1 failure(s) 0 error(s) 2 test(s) run
1
> (require cpsc411/test-suite/public/v4)
> (run-tests (test-suite "" (test-compiler-pass uniquify interp-values-lang-v4 interp-values-unique-lang-v4 values-unique-lang-v4?)) 'quiet) 18
> (define (specify-implementation x) (match x [`(module ,x) `(module ,(* x 8))])) > (require cpsc411/langs/v7)
> (register-test-programs! interp-exprs-unsafe-data-lang-v7 '(("" (module 5))))
> (run-tests (test-suite "" (test-compiler-pass specify-implementation interp-exprs-unsafe-data-lang-v7 interp-exprs-bits-lang-v7 exprs-bits-lang-v7? (lambda (sv tv) (equal? sv (ptr->v tv))))) 'quiet) 0
procedure
(compiler-testomatic passls interpls) → test-suite?
passls : (listof ('a -> 'b)) interpls : (listof (or/c #f ('c -> any/c)))
The lists are expected to have the same length, and the final pass in the list must produce a string that can be assembled and executed using execute with an empty current-pass-list, i.e., it must be a whole assembly program.
Each interpreter in the interpls is expected to indicate the source language interpreter for the respective pass in the passls. That is, for a passin the passls of type ('a -> 'b), then corresponding interpreter must have the type ('a -> any/c).
Furthermore, the next interpreter in the list must exist and be the interpreter for the target of pass, i.e, must have type ('b -> any/c), or be #f (unless the pass is the final element of the passls, in which case the target interpreter execute is used).
If an interpreter is set to #f, that indicates the language is not interpretable in isolation. Instead, the pass is composed with the previous pass until a target language interpreter can be found.
> (register-test-programs! interp-paren-x64-v2 '(("" (begin (set! rax 120)))))
> (register-test-programs! interp-paren-x64-fvars-v2 '(("" (begin (set! rax 120))) ("" (begin (set! fv1 121) (set! rax fv1)))))
> (run-tests (compiler-testomatic ; Compiles all programs to 120 (list (lambda (x) "mov rax, 120") wrap-x64-run-time wrap-x64-boilerplate) (list interp-paren-x64-v2 #f #f))) 1 success(es) 0 failure(s) 0 error(s) 1 test(s) run
0
> (run-tests (compiler-testomatic ; Compiles all programs to 120 (list ; Compiles programs to themselves; 1 test fails, since it ; contains fvars, thus the output isn't valid. (lambda (x) x) ; Compiles programs to 120; 0 tests fails, since all ; registered test programs in this language produce 120, ; and no earlier passes produces new valid test programs. (lambda (x) "mov rax, 120") wrap-x64-run-time wrap-x64-boilerplate) (list interp-paren-x64-fvars-v2 interp-paren-x64-v2 #f #f)))
--------------------
> anonymous suite
FAILURE
test-program: (begin (set! fv1 121) (set! rax fv1))
expected: 121
src-interp: interp-paren-x64-fvars-v2
trg-interp: interp-paren-x64-v2
output-program: (begin (set! fv1 121) (set! rax fv1))
test-type: "Checking that output is syntactically correct"
name: check-true
location:
/builds/packages/cpsc411-pub/cpsc411-lib/cpsc411/test-suite/utils.rkt:408:14
params: '(#f)
--------------------
2 success(es) 1 failure(s) 0 error(s) 3 test(s) run
1
procedure
(register-test-programs! src-interp test-progs) → void? src-interp : (any/c -> any/c) test-progs : (listof (list string? any/c))
The test programs are registered for later use in the testomatic framework.