TestBank

TestBank is hosted at https://testbank.roualdes.us/. To generate an instance of exercise 0Y7x, run

curl https://testbank.roualdes.us/0Y7x

Rerun the above command. Notice that there is a random component to the exercise. This is the crux of TestBank: exercise templates with random components generated from code run in a Jupyter backend. Right now, only Python3 and R are available.

Examples

The directory examples contains two primary examples. First, an approximate exam from my MATH 314 course at Chico State. The second a working example of how one could leverage TestBank with check50. Each subfolder has its own README.

The files exampels/add.r and examples/add.json are intended to demonstrate the file structure necessary for adding an exercise to TestBank's databases.

Data

For now, let's simplify things and store exercises in .json files. I'll use lowdb to interact with the JSON files. There will be one file for exercises and one file for tags. The last bit will be how each exercise is returned to the end user.

Exercise schema

{
    id: "a unique ID, 4 characters long; [0-9a-zA-Z]{4}", # generated by TestBank
    language: python | r,
    exercise: "code to produce an exercise",
    tags: ["tag1", ..., "tagT"] # planned but not implemented,
}

Tags

If a user searches the database with a complex query such as tags: tag1 AND (tag2 OR tag3), then the following secondary database should make for more efficient searching.

{
    tag1: [id1, ..., idA],
    tag2: [id1, ..., idB],
    ...
    tagT: [id1, ..., idT]
}

After collecting the queried tags' arrays, simple set operations should enable direct replacement of AND with && and OR with ||.

Output schema

Each exercise will be returned from the TestBank server as a JSON object with one of the two following structures. If an exercise is requested, the user will receive a JSON object with the following schema

{
    id: "a unique ID",
    seed: 1234,
    context: "the context of this exercise",
    questions: ["partA", ..., "partZ"],
    random: {X: x, μ: m, σ: s} # Associative Array specific to language
    # eg R => list(), Python => dict() or {}
}

The order of the following elements of an exercise's output schema is not important.

• id string. A string of 4 characters, [0-9a-zA-Z]{4}, that uniquely
  identifies each exercise. TestBank will generate these autmatically
  upon insertion of an exercise into its database.
• seed int. A seed for the (pseudo) random number generation which is
  constrained to be in [1, min(R int, or Numpy np.uint32)] = [1, 2147483647]. If no seed is specified with each request for an
  exercise (or its solution), TestBank will randomly chose a seed.
• context string. A string the sets up the exercise's context. If no
  follow up questions are involved in an exercise, the context can
  contain the question/prompt.
• questions array of strings. The questions array holds parts,
  say A, B, C, D, and E, of an exercise as strings. While the
  questions key is required, the value may be an empty array..
• random Object, aka associative array. The random object holds named
  elements of the randomly generated components of an exercise. While
  the random key is required, the value may be an empty associative
  array

If a solution is requested, the user will receive a JSON object with the following schema

{
  id: "a unique ID",
  seed: 1234,
  solutions: 0.314 || "0.314" || ["partA", ..., "partZ"],
  random: {X: x, μ: m, σ: s} # Associative Array specific to language
  # eg R => list(), Python => dict() or {}
}

Again, the order is not important. solutions is the only new key.

• solutions number, string, or array of strings. If questions is an
  array, then solutions should be an array containing answers to
  each part of the exercise's questions. If questions is an empty
  array, then solutions can be a number or a string, appropriately
  matching the prompt of context.

Writing exercises

To write an exercise, you must at least use the following structure, inclusive of the custom Mustache tags for the exercise's ID:

id = '#< ID >#'
... code to produce Output schema

These custom Mustache tags have two benefits. First, they help prevent my text editor from getting confused while developing TestBank exercises, at least within some fairly standard data science programming languages, R, Python, Julia. Second, they simplify writing LaTeX within Python strings, since Python's string formatting insists on double curly braces (otherwise used in Mustache) to get single curly braces (used in LaTeX) into a string.

Examples exist in the examples directory.

Extra Mustache tags

If you want to provide a seed and/or solutions, and keep the solution hidden from the (not yet developed) website, use Mustache tags that allow TestBank to selectively ignore the exercise and/or the solution.

... code necessary for both exercise and solution
id = '#< ID >#'
seeed = #< SEED >#

#< #exercise >#
... code to produce exercise Output schema
#< /exercise >#

#< #solution >#
... code to produce solution Output schema
#< /solution >#

It is possible to ignore the seed template entirely. Simply don't put the seed template within the exercise's code.

The solution template has two goals. First, to enable a request for a specific exercise to return only the solution. Second, when a website that permits searching through TestBank's database is developed, the solution template is intended to allow for hidden solutions. Thus any exercise could be searched, found, and read without displaying the solution.

Examples exist in the examples directory.

Check an exercise before entering it into the database

The goal is to make entering exercises into the database as automatic as possible, while simultaneously addressing security of the TestBank sever, stability of the kernel, and response time.

For instance, if you wanted to add the problem examples/add.r (with associated meta data file at examples/add.json) follow these steps.

1. Eyeball code in add.r for malicious content.
2. From this project's root directory, run

test/exercise.sh examples/add.json

If the tests above pass, insert the exercise to TestBank's database with the upsert command from TestBank's CLI.

The directory test contains more details on tests.

TestBank command line interface (CLI)

TestBank's GitHub repository comes with a command line interface, cli.js, which attempts to help with testing exercises and entering exercises into the database. The node commands below must be run from the root directory of this repository.

Testing

To test whether or not an exercise will run (after the Mustache tags are entered), consider the file examples/add.r. At the command line, with littler installed/aliased as lr run

$ lr -e "$(node cli.js test examples/add.json exercise)"

If the above command runs just fine, then there's a hope that your code will run on the TestBank kernel after being entered into the TestBank database.

To test a Python solution, run

$ node cli.js test examples/ex01.json solution | python3

To ensure that the output is a correctly formatted JSON object, run

$ node cli.js test examples/ex01.json solution | python3 | python3 -m json.tool

To ensure that the output, and correctly formatted, JSON object appropriately follows the Output schema run

$ node cli.js test examples/ex01.json solution | python3 | python3 -m
json.tool | bash test/schema.sh

The script exercise.sh in directory test will run all of these steps for you, and for both the exercise and solution section of the code, by running

test/schema.sh examples/add.json

Inserting an exercise into TestBanks's database

To insert an exercise into TestBank's database, use the command line interface as

$ node cli.js insert ex.json

Where ex.json is JSON file for an exercise which provides some meta information about the exercise to be entered. See examples in the examples directory for more information.

TODO

[] Add to README something about requesting solutions based on evenness of seed.

[] Enable more complete/robust testing of the database and TestBank backend.

[] Develop a database searchable landing/home page for TestBank.

[] Add HTML, online homework system-esque, example. Maybe bootswatch's theme flatly?

[] Authorization? I'm thinking anybody can see questions and the code that generates them. Anybody can request a solution. But it should necessitate some form of authorization to see the code the produces the solutions. If seeds are used appropriately, then just knowing the answer is 0.532 for a specific seed won't help much. And if no randomization is appropriate for a question, then no likely no solution code is appropriate.

[] Figure out versions of dependencies.

[] Insert (possible) FAQs as exercises in the database. Should double as helping to explain how to use TestBank and show off some of the features, like showing/hiding solutions and ignoring the SEED.

[] GZIP databases.

[] Restart Jupyter kernel's when/if they crash.

Development

To work on TestBank, the following dependencies are necessary. There is an Ansible playbook named ansible.yml that can set up an Ubuntu 18 machine, if you prefer.

• Node.js
  • all of the Node.js packages within the file package.json, which
    can be installed with npm install from within this directory.
• Python3
• R

The following packages within each language's ecosystem

• For Python use pip to obtain
  • jupyterlab
  • numpy
  • scipy
  • pandas
• For R use the function install.packages() to obtain
  • IRkernel
    • After installation, in R (not RStudio), within a Terminal that
      has access to the Python envirnoment containing jupyterlab, run

      library(IRkernel)
      IRkernel::installspec()
      

  • dplyr
  • tidyverse
  • jsonlite
  • pander

And for tests one needs,

• littler
  • This can not be installed as an alias, since some scripts
    necessitate lr being on the path.
• bash
• jq

This is my best attempt at a complete list, but I'm still iffy about versions of everything. Let me know what I've missed.

PRs welcome.

License

License: Open source, BSD (3-clause).