A complement to The Oracle for compilers class. Given some test cases, this program automatically queries the oracle and tests your compiler!
More spefically this program:
- Runs your *.cshanty test inputs thru the oracle & collects its response to give a “true output”; writes *.truth to test dir.
- Runs all your test inputs through your compiled compiler & collects outputs to give an “actual output”; writes *.actual to test dir.
- For each (*.truth, *.actual) pair, essentially runs a more nuanced version of diff to check if your compiler is producing correct output; provides score and details.
python3 diviner.py <diviner version> <path to cshantyc> <path to dir containing *.cshantyc test cases> <-l, optional>
python3 diviner.py --help
python3 diviner.py d5 -h
python3 diviner d5 cshantyc my_tests/
python3 diviner d5 cshantyc my_tests/ --lazy-query
python3 diviner d5 cshantyc my_tests/ -l
python3 diviner d6 --help
python3 diviner d6 -h
my_tests/ might contain test1.cshanty, test2.cshanty, etc.
pip3 install bs4
pip3 install tqdm
…the rest should be standard.
You can also (alternatively) run
pip3 install -r requirements.txtfor convenience
While we are on the topic of make directives,
make cleanremoves all compiled
- For a given *.cshanty, if this program already detects a *.truth it will still requery the oracle for that test. Setting
--lazy-query=1in cmd line args will instead skip it to avoid excess time and requests.
- Currently, only type analysis version for P5 is developed (diviner version = d5). However, feel free to fork and add new version
versions/. If you inherit from
core/diviner.py/DivinerBaseit should be very trivial to do other versions.