Skip to content

A system to support (semi-)automated assessment of textual exercises

License

Notifications You must be signed in to change notification settings

pal03377/Athena

 
 

Repository files navigation

Athene: A system to support (semi-)automated assessment of textual exercises

This system implements an approach for (semi-)automated assessment of textual exercises and can be integrated into learning management systems (LMS). A reference integration exists for the LMS Artemis.

Athene implements the approach described in our paper Toward the Automatic Assessment of Text Exercises:

Jan Philip Bernius and Bernd Bruegge. 2019. Toward the Automatic Assessment of Text Exercises. In 2nd Workshop on Innovative Software Engineering Education (ISEE). Stuttgart, Germany, 19–22. [pdf]

Architecture

The Athene system is built using a microservice architecture, as depicted in Figure 1. Four components comprise the Athene system:

UML Component Diagram

Figure 1: Athene Microservice Architectuer depicted using a UML Component diagram.

  1. Load Balancer: Provides Service in the form of a HTTP REST API to submit an Athene job. This component will manage the work distribution to other components.
  2. Segmentation: Component for segmenting Student Answers based on Topic Modeling.
  3. Language Embedding: Component for computing Language Embeddings using ELMo and uploading training material for ELMo.
  4. Clustering: Component for clustering of ELMo Vectors using HDBSCAN.

The load balancer orchestrates jobs and calls all components using their HTTP APIs. After the job is completed, it calls back to LMS to submit the results.

Basic Usage

Using the docker-compose.yml file included in the root directory of the repository is the easiest way to start Athene. The execution of

docker-compose up -d

will automatically build and start the Load-balancer, Segmentation, the Embedding and the Clustering component (The -d parameter will run containers in the background). By default, a traefik-container will manage API-Endpoints and expose them on port 80 (default HTTP-port). The following API-routes are available after start:

Traefik provides a dashboard to monitor the status of underlying components. This dashboard is available on http://localhost:8081/dashboard by default.

For testing and development purposes, a single component can be re-built using e.g.

docker-compose build segmentation

or be started separately (in foreground mdoe) using e.g.

docker-compose up segmentation

Stopping Athene can be achieved by stopping the running containers using

docker-compose down

For further information have a look at the Compose file reference and the Compose command-line reference.

Configuration

For configuration of the Athene system you can make use of the .env-file in the repository. All variables in there are used in the docker-compose.yml to bring up the Athene system.

Contributing

We welcome contributions in any form! Assistance with documentation and tests is always welcome. Please submit a pull request.

Citing

To reference the automatic assessment approach developed in this system please cite our paper in ISEE 2019 proceedings.

@inproceedings{BerniusB19,
  title     = {Toward the Automatic Assessment of Text Exercises},
  author    = {Jan Philip Bernius and Bernd Bruegge},
  booktitle = {2nd Workshop on Innovative Software Engineering Education (ISEE)},
  address   = {Stuttgart, Germany},
  year      = {2019},
  pages     = {19--22},
  url       = {http://ceur-ws.org/Vol-2308/isee2019paper04.pdf}
}

About

A system to support (semi-)automated assessment of textual exercises

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Python 94.9%
  • Dockerfile 1.7%
  • Kotlin 1.4%
  • Shell 1.2%
  • Other 0.8%