Websheets Manual

This manual describes how to write new exercises. It doesn't cover installation — for that, see the README or the git repository.

Format Overview

If you look in the exercises directory, you will see a bunch of .py files. Each one of them defines a single Websheet exercise. Here is the simplest example — it is TwoPlusTwo.py:
source_code = r"""
public static void main(String[] args) {
\[
   System.out.print(2+2);
]\
}
"""

description = r"""
Fill in the program so it <i>prints</i> out the integer obtained by adding 2 and 2.
"""

tests = r"""
testMain();
"""
This defines a Python module, using triple-quotes so that the strings can span multiple lines, and using the r""" raw-string prefix so that you don't need to escape backslashes. However, that is all the Python you need to know.

See this definition in action by visiting the TwoPlusTwo websheet. Try making some correct and incorrect submissions.

So at its core, a websheet is a list of field names, each taking on a string value. Reviewing the above, you must define the fields

There are a few other optional fields beyond these three. Here are the sections of this manual, which describe how to write each component in more detail:

Of course, looking at existing examples is the most thorough way to understand what's going on, once you're familiar with the main concepts.

source_code: The reference solution and blank locations

When writing a websheet, normally you will start by writing the source_code as a correct solution to the problem. (Note that you can either write in public class ExerciseName like HelloWorld.py or let it be filled in implicitly, like TwoPlusTwo above.)

Once you fill in a working program, you need to decide on what parts you want to blank out for the student. For the purposes of websheets, every blank has to fall in one of the following two categories:

For example, the Distance websheet has both kinds of blanks. Try typing some stuff in the blanks and see how they resize to accomodate unlimited student input.

Both kinds of blanks are indicated by using \[ to denote the start of the blank, and ]\ to denote the end of the blank. The interior region will be left blank and editable for the user.

The rules for how these delimiters must appear in source_code are:

For example, look at how Distance.py defines both kinds of blanks.

(In the multi-line case, additional whitespace on the lines containing the delimiters is currently ignored, although it could be meaningful in a later version, say where Python is used and we want to force indentation of a region.)

Hidden code, fake code, default contents for blanks

The simple \[...]\ delimiters are sufficient to describe a large variety of exercises. But to allow the exercise writer more flexibility, websheets allow a few additional kinds of regions.

You can use \hide[...]\ to enter code that will be hidden from the student's UI. It will be included in both the reference solution and the student's combined solution that gets executed upon submission. For examples, see

You can use \fake[...]\ to display read-only text in the user interface that looks like it is part of the program, but that won't actually be included. For examples, see

You can use \show: to have a blank initially populated with some text. (Think of providing a buggy program fragment for a student to fix.) To do this, change \[ blanked-out text ]\ to \[ blanked-out text \show: default text here ]\. For example, see

Cheatsheet: What elements appear where?

in UI?in reference solution?in executed/tested combined student solution?
plain text yes (read-only) yes yes
blanked-out \[...]\ yes (editable, initially blank) yes yes (edited student text spliced in)
\show: of blanked-out yes (editable) no yes (edited student text spliced in)
\fake[ ... ]\ yes (read-only)nono
\hide[ ... ]\ no yes yes

description: The exercise description, in HTML

You can use as few or as many HTML elements as you like in describing the exercise. The simplest descriptions are just text, using <p> to separate paragraphs, and sometimes using <tt> and <pre> to use a monospace font.

LaTeX. All websheets have MathJaX processing turned on. This means that you can include LaTeX using $inline format$ or $$big centered display format$$. Look at for examples of both.

Links. Sometimes you'll want to add a link to another websheet inside of the description. One way to do this nicely is to write a link with a URL like

<a href="?group=SquareSwap">SquareSwap</a>
E.g. see .

Diagrams. You can use any software and any formats you like, but the example exercises using diagrams (e.g. , ) were created using IPE (which uses PDFs as an editable format) and rasterized to PNG using render.sh.

tests: What the grader should test

The tests contains a snippet of Java code that makes calls to our testing API. These API calls cause the automatic grader to dynamically execute both the reference code and student code in separate namespaces, capturing their outputs and return values, and comparing the results for correctness.

Because tests will be spliced in to a Java program, any valid Java code can be used. For example, instead of writing

testMain("AA");
testMain("BB");
…
testMain("ZZ");
you could write
for (char ch='A'; ch<='Z'; ch++) testMain(""+ch+ch);
You could even define local or anonymous classes if you want (although no current examples do this).

Basics: testing static methods

The testing methods testMain() for testing main, test() for instance methods, testConstructor() for constructors, and testOn() for instance methods are used to execute student code and execute reference code and compare the results. It handles things like comparing return values, capturing and comparing printed output, catching and comparing exceptions, and looking for changes to arguments automatically.
testMain(String... args)
call main on these arguments.
E.g. see
test(String methodName, Object... args)
call a static method named methodName on this list of arguments
E.g. see
note: if there's exactly one argument, which is an array, Java requires you use
test(methodName, (Object) /* your array here */)
due to inherent ambiguity in varargs. E.g. see . The ambiguity doesn't exist if there are multiple arguments of which some are arrays, e.g. see .

Basics: testing instance methods

For testing instance methods, you'll generally call a constructor defined by the student, and then call an instance method. The typical approach involves 3 steps, like this sample from :
saveAs = "attendance"; // (1)
testConstructor(); // (2) i.e. attendance = new Clicker();
testOn("attendance", "curr"); // (3) i.e. attendance.curr(); 
This means,
saveAs = String  (default null)
save the value returned by the next test as a named object. (implementation note: we maintain two parallel namespaces for student and reference solutions.)
Note: you can save anything, not just constructor calls. E.g. in , we save a method's return value:
saveAs = "scaled";
testOn("x", "times", 10.0);
testConstructor(Object... args)
call the constructor with this list of arguments
testOn(String instanceName, String methodName, Object... args)
on the named instance, call the named instance method with this list of arguments
Here is one more tool used in conjunction with saveAs.
var(String)
once an object is saved, you may want to pass it an argument to a method, rather than just calling its instance methods. For example in , after constructing two objects x and y, we call x.plus(y) like so:
testOn("x", "plus", var("y"));
basically, the var() wrapper is needed to distinguish the saved variable y from the string literal "y"

Exceptions and standard input

expectException = boolean  (default false)
indicates that the student code should throw an exception on the following test. (it will catch whatever exception the reference code throws, and expect the student code to throw the same class of exception)
E.g., .
stdin = String  (default null)
for the next test, set the contents of standard input to the following String
E.g., .
stdinURL = String  (default null)
same as stdin, but fetch this webpage and use its contents. Useful when there is a long data set posted online, and you prefer not to copy a long text into the websheet definition
E.g., .
note: use simple string literals only. e.g. stdURL = "http://goo.gl/" + "robots.txt"; is bad, it will confuse the prefetcher that grabs the data before starting the sandbox.

Further methods

remark(String s)
Print out a remark to the user explaining what the tests are doing. Useful when you still want them to see the automatically-generated description of the test, but you also want to explain more context to the student. s should be valid HTML.
c.f. title = which replaces the automatically-generated test description.
E.g., .
E.g., .
store(Object o)
Copy an object to the user and reference namespace. Used for checking shallow copying in .
construct(String pckg, String clss, String type_prms, Object... args)
Use this to create a new instance of some other class. pckg should be a fully qualified package or null to use student/reference code; clss should be a simple class name; type_prms should be "" if you are not constructing a generic, or a list of type arguments for a generic if you are (this is just shown to the student in the description of the test since generic erasure happens at runtime); and args are the consturctor arguments. It is best seen by these two examples:
In we call
construct("stdlibpack", "Queue", "<Integer>")
to mimic a call to new Queue<Integer>() (the package stdlibpack has the COS 126 "standard" classes, and we want the Queue from that package)
In we call
construct(null, "Person", "", "Tony")
to mimic a call to new Person("Tony")Person is in the dependencies of that websheet. Leaving pckg as null means that the student's code uses the Person from the student package while the reference uses the one from the reference package.
testNamedMain(String methodToCall, String fakeClassName, String... args)
Call some hidden internal method, but pretend you are calling main(args) on some other testing class. Used in at testNamedMain("exampleClientMain", "ExampleClient"); to run the hidden method exampleClientMain but to tell the student we are running java ExampleClient.
Note: store(), construct() and testNamedMain() will reset all options to their defaults, like the basic test…() methods.

Further options

title = String  (default null)
replace the automatically-generated description of the next test with the given HTML. Useful if you are using a lot of fake or hidden code and the automatically-generated description the student would see in testing would make no sense.
c.f. remark() which adds extra commentary before the next test
E.g. .
E.g. .
quietOnPass = boolean  (default false)
Don't show the next test unless the student fails it. Especially useful if you have a lot of tests.
E.g. .
E.g. .
maxOutputBytes = int  (default 10000)
Change the maximum number of bytes the student code can print per test before it is terminated.
E.g. .
E.g. .
dontRunReference = boolean  (default false)
Don't run the reference solution. Instead, rely on either template code or hidden code that will determine if the student passes, and which will throw an exception if they failed, but not if they passed.
E.g. .
E.g. .
Note that hidden code can make thrown exceptions look a bit nicer by using websheets.Grader.FailException like FTPLimiter.
cloneForStudent = boolean  (default true)
Before passing arguments to the student, clone them. So if the student mutates the arguments, it doesn't affect the grader.
This is turned off to check shallow copying in .
cloneForReference = boolean  (default true)
Same as above, but for reference code. No current example.
realTolerance = double  (default 1E-4)
Relative error accepted for outputs. (Except if reference answer is 0, this is the absolute error accepted.) No current example.
ignoreRealFormatting = boolean  (default true)
If false, ignore realTolerance, also don't accept 5.00 as the same as 5.0. No current example.
ignoreTrailingSpaces = boolean  (default true)
Ignore trailing spaces at the end of each line. No current example.

Misc

defaultOptions
To set an option permanently (as opposed to just for one test), use syntax like
defaultOptions.quietOnPass = true;
E.g, see .
randgen
This is a Random instance just made available for convenience. E.g., see which calls randgen.nextInt(100) to generate a random number from 0 to 99.

Optional websheet fields

Here are a couple of other fields that are available for use in websheets
classname
when you want the visible and executed class name not to be the same as the name of the overall exercise (the Python module name, referred to internally as the slug).
E.g. .
show_class_decl
set to False if you don't want the user to see the class declaration.
E.g. .
epilogue
show a message or explanation after problem is completed.
E.g. .
dependencies
require student to complete some other websheet before this one; the student's most recent correct solution to the dependency will be made available when compiling executing this one.
E.g. .
Note: if instead you want to give student extra reference code without requiring them to actually complete the dependency, you can use \hide instead:
E.g. .
imports
a list of classes to import (will be made visible in UI and added to reference solution)
E.g. .

Adding new exercises to the system

  1. Create a new .py file in the above format in the exercises directory. (E.g., NewProblem.py)
  2. Go visit index.php?group=NewProblem in your browser. Or, refresh index.php and select NewProblem from the list of exercises.