ENGR 101 | University of Michigan

Guide to the Autograder

What is this guide for?

This guide is intended to document how to use the Autograder. The guide will describe how to submit files to the Autograder, how to interpret the feedback from the Autograder, and possible errors and pitfalls.

Submitting Your Files

The required file and file format for each project is clearly specified in each project specification. The file(s) need to be named exactly the same as the Project specification. If any of your files are not named correctly or you haven’t uploaded all the files that are expected, the Autograder will warn you with a dialog box:

If you choose to submit despite the warning, the Autograder will delete any files that are not named correctly to protect the Autograder from malicious files, but it will still count as a submission.

You can always make a submission to the Autograder with only some of your files uploaded. Some of our projects have multiple files that are required for the complete project to work correctly, but you can submit files as you go to verify that your helper functions work correctly before moving on to your driver program.

Interpreting Feedback from the Autograder

The Autograder will run your code against a variety of test cases and give you feedback on whether your code has the intended behavior and/or produces the correct output. The Autograder looks primarily for:

1. Exit status. This just means that the program ran without any errors.
2. Output. This can include return values from functions, information printed to a file, or images created. The Autograder compares your output to the expected output and checks to see if they match.

Here is an example of what you will see if your code passes all the test cases:

The light blue color means that your code behaved correctly for all the parts of the test case. If you click on the title of a test case, you’ll see detailed feedback:

The feedback for a test case that passed is perhaps not all that interesting to you; it just verifies that the program works correctly for this case.

Here is an example of what you will see if your code does not pass all the parts of a test case:

The red color indicates a part of the test case in which your code did not behave as expected. If you click on the title of the part that failed, you will see the feedback for that part of the test case:

We’ve tried to make this feedback useful, but you should also refer to the Project FAQs for additional guidance on what to look for when your code doesn’t pass a test case.

If you come across something strange that says program fails at Line number XX, and your program doesn’t even contain a Line number XX, it’s because the autograder is not talking about the line number on your program but on the test program on our end. Ignore the Line number and try to read the rest of the message and use that to debug your code.

Watch for Whitespace Differences!

The Autograder does a literal comparison of what your code creates and what the expected output is, whether that is text or an image. One particular nuance with text is when and where are newlines placed (a newline is when a line of text ends and you go to a new line).

Here is example of feedback for a test case that failed:

The Autograder says that the expected output doesn’t match the student output (meaning, your output), but the two sets of text look exactly the same… don’t they? Almost! Click the “Show Whitespace” option:

And then the output comparison will update to show spaces, tabs, and newlines:

AHA! Now we can see the difference: The student version is missing a newline character at the very end of what their program created. They can now go back and revise their program to print out this missing whitespace.

Sometimes the Autograder Runs Slow

The autograder is sometimes (rarely) very strangely slow. If this problem arises try again after 15 minutes. Sometimes if a lot of students are trying to submit at the same time the queue may be very long. This will happen very rarely because ENGR 101 has a relatively simple grading scheme and it shouldn’t take too long for the autograder to provide feedback for the students. Please do not freak out if it is slow.

Who to contact when Autograder is going wrong?

It’s unlikely that the Autograder will have issues, but it does happen on rare occassions. If you feel like you are following all the steps to correctly submit your code and it’s just not working correctly, please post on Piazza so we can figure out what is going on!

Updates

This document will be updated often, so please check here if you have a strange error.