Autolab: Open Source Collaboration & Feedback

What is Autolab?

Autolab is an Apache-2.0 licensed course management service, written in Ruby on Rails, that lets instructors offer autograded programming assignments to students over the web, and it is built for university courses, teaching staff, and institutions that need automated grading of student code at scale.

What it is

Autolab is a course management service, initially developed by a team of students at Carnegie Mellon University, that enables instructors to offer autograded programming assignments to their students over the Web. It was released as an open-source system in Fall 2014, having been in use at CMU since 2010. The application is written in Ruby and runs on Rails 6, and it sits in the education and autograding ecosystem, carrying topics such as autograding, education, feedback, rails, scoreboard, students, and teaching.

The concrete problem it solves is the manual effort of grading programming work. Autolab rests on two key ideas: autograding, that is, programs evaluating other programs, and scoreboards. A student submits code, an autograder program evaluates that submission, and the result flows into the course record. Around that core, Autolab provides the services instructors expect from a course management system, including gradebooks, rosters, handins and handouts, lab writeups, code annotation, manual grading, late penalties, grace days, cheat checking, meetings, partners, and bulk emails. In other words, it replaces the loose collection of separate grading scripts, spreadsheets, and generic course tooling that a programming course would otherwise assemble by hand.

Key capabilities

Autograding, in which programs evaluate other programs, as the central grading mechanism.
Scoreboards, alongside gradebooks, for surfacing submission results.
Handins and handouts, lab writeups, and code annotation for assignment delivery.
Manual grading, late penalties, and grace days layered on top of the automated results.
Cheat checking to detect copied or suspicious submissions.
Roster management, partner assignment, meetings, and bulk emails for course administration.
A Rails 6 codebase with a master-rails-5 branch, where Rails 5 support has been discontinued and new features are not backported.

Who uses it and how

Carnegie Mellon University runs Autolab at scale: about 5,000 students each semester, across courses in Pittsburgh, Silicon Valley, Qatar, and Rwanda, since 2010.
Other schools and organizations adopt it directly, since the README invites anyone to use Autolab at their school or organization.
Small teaching operations fit too: the core developers provide servers for free on a case-by-case basis, especially for NGOs and small high-school classrooms.
Evaluators and prospective instructors can try a hosted demo site before committing to an install.
Contributors work through the Contributing to Autolab Guide and the project Slack community, while documentation changes are built with mkdocs and mkdocs-material.

Getting started

Installation and deployment follow the project documentation at https://docs.autolabproject.com, and the application itself is run as a Rails app with Bundler, for example bundle exec rails spec for the test suite and RAILS_ENV=test bundle exec rails autolab:setup_test_env to prepare the test environment. A demo instance is available at https://nightly.autolabproject.com/ for trying the system before installing it.

How it compares

The facts provided name no comparable tools and no list of paid products that Autolab replaces, so it stands alone in this registry on that axis, and any comparison would have to rest on the Apache License 2.0 terms and the self-hosted Ruby on Rails deployment rather than on stated alternatives.

When to use it — and when not to

A self-hoster must operate a Rails application, including a configured database.yml, a test database created and migrated through autolab:setup_test_env, a tmp/ directory, and a separate mkdocs toolchain if the documentation is to be rebuilt locally. The project is a poor fit for anyone wanting a fully hosted, zero-operations service, or for a team unwilling to run and maintain a Ruby on Rails stack; the repository also carries 189 open issues, and the README gives no inline package or container install command, deferring installation entirely to external documentation. Rails 5 users are out of scope, since that support is discontinued.

project readme (upstream, from github) — read inline

Autolab is a course management service, initially developed by a team of students at Carnegie Mellon University, that enables instructors to offer autograded programming assignments to their students over the Web. The two key ideas in Autolab are autograding, that is, programs evaluating other programs, and scoreboards.

Autolab also provides other services that instructors expect in a course management system, including gradebooks, rosters, handins/handouts, lab writeups, code annotation, manual grading, late penalties, grace days, cheat checking, meetings, partners, and bulk emails.

Since 2010, Autolab has had a transformative impact on education at CMU. Each semester, it is used by about 5,000 CMU students in courses in Pittsburgh, Silicon Valley, Qatar, and Rwanda. In Fall, 2014, we are releasing Autolab as an open-source system, where it will be available to schools all over the world, and hopefully have the same impact it's had at CMU.

GitHub last commit

Subscribe to our mailing list to receive announcements about major releases and updates to the Autolab Project.

Try It Out

We have a demo site running at https://nightly.autolabproject.com/. See the docs for more information on how to log in and suggestions on things to try.

Installation

We released new documentation! Check it out here.

Testing

Setting up Tests

Add a test database in database.yml
Create and migrate the database.
```
RAILS_ENV=test bundle exec rails autolab:setup_test_env
```
Do not forget to use RAILS_ENV=test bundle exec in front of every rake/rails command.
Create necessary directories.
```
mkdir tmp/
```

Running Tests

After setting up the test environment, simply run spec by:

bundle exec rails spec

You may need to run RAILS_ENV=test bundle exec rails autolab:setup_test_env when re-running tests as some tests may create models in the database.

You can also run individual spec files by running:

rake spec SPEC=./spec/<path_to_spec>/<spec_file>.rb

Rails 5 Support

Autolab is now running on Rails 6. The Rails 5 branch can be found on master-rails-5. We will not be backporting any new features from master to master-rails-5, and we have discontinued Rails 5 support.

Updating Docs

To install mkdocs, run

pip install --user mkdocs

We rely on the mkdocs-material theme, which can be installed with

pip install --user mkdocs-material

To run and preview this locally, run:

python3 -m mkdocs serve

Once your updated documentation is in master, Jenkins will automatically run a job to update the docs. You can trigger a manual update with

python3 -m mkdocs gh-deploy

This will build the site using the branch you are currently in (hopefully master), place the built HTML files into the gh-pages branch, and push them to GitHub. GitHub will then automatically deploy the new content in gh-pages.

Contributing

We encourage you to contribute to Autolab! Please check out the Contributing to Autolab Guide for guidelines about how to proceed. You can reach out to us on Slack as well.

License

Autolab is released under the Apache License 2.0.

Using Autolab

Please feel free to use Autolab at your school/organization. If you run into any problems, you can reach the core developers at [email protected] and we would be happy to help. On a case-by-case basis, we also provide servers for free. (Especially if you are an NGO or small high-school classroom)

Changelog

v3.0.0 (2024/07/24) UI-based deployment configuration, Simple File Manager, Export / Import Course, Tango, Grading Improvements

Configuration of authentication, Github, SMTP moved to UI
Simple File Manager to manage files in courses through a UI
Import / Export Course data in one go, with improved assessment importing
Improvements to grading UI/UX such as an improved Gradebook, Code Diff Viewer, Starred Problems
Improvements to autograding with direct Docker file uploading and the option to selectively disable network access

v2.12.0 (2024/01/20) Attachment categories and visual cues

Ruby upgraded to 3.2.2
Rails upgraded to 6.1.7.6
Instructors can now specify a category and "release at" date for attachments
Assessment start / end dates are now shown on course homepages

v2.11.0 (2023/05/21) LTI Settings UI, extensions metrics, and simultaneous extension creation

Introduced UI to manage LTI integration settings
Added extension metrics for instructors to monitor students by number of extensions granted
Instructors can now create extensions for multiple students at once

v2.10.0 (2023/01/13) LTI Integration, Generalized Feedback, and Streaming Output

Autolab now supports roster syncing with courses on Canvas and other LTI (Learning Tools Interoperability) services. For full instructions on setup, see the documentation.
Streaming partial output and new feedback interface
Generalized annotations

For older releases, please check out the releases page.

Frequently asked questions

Is Autolab free to use?

Autolab is open source under the Apache-2.0 licence. There is no licence fee and no seat count — you can self-host it or, where the project offers one, pay a vendor for a managed version instead.

What does Autolab do?

Course management service that enables auto-graded programming assignments.

What is Autolab written in?

Autolab is primarily written in Ruby. Its source is publicly available at https://github.com/autolab/Autolab, and it has 808 GitHub stars.