Skip to content

johnhany97/grader

BranchesTags

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

42 Commits

assets

assets

 
 

examples

examples

 
 

grader

grader

 
 

processors

processors

 
 

test

test

 
 

vendor

vendor

 
 

.gitignore

.gitignore

 
 

README.md

README.md

 
 

go.mod

go.mod

 
 

go.sum

go.sum

 
 

main.go

main.go

 
 

Repository files navigation

Automatic Programming Assignments Grader

Usage of main.go:
  -schema string
        Marking scheme to follow when grading the assignment (required)

Testing & Benchmarking

To run the test suite, simply execute the following command:

go test -v -cover ./...

To run the benchmarks, simply execute the following command:

go test -v -run="none" -bench=. -benchmem ./... -benchtime="5s"

Installation

First off, you'll need to Install Go and make sure enviornment variables are set up properly. Following that, you'll need to Install Docker. To be able to run Java Style checking test tasks, you'll need to install Checkstyle as well.

Running the following two commands install two Go-based projects that run the grading.

go get -u github.com/docker-exec/dexec
go get -u github.com/johnhany97/grader

Note: We use a modified version of dexec which supports timing out executions. Changes should've been merged, however, ensure you have version 1.0.9-SNAPSHOT of dexec installed before continuing.

That customized version could also be found at https://github.com/johnhany97/dexec

Supported languages

Following is a list of supported languages with the required docker image.

  • C: dexec/lang-c
  • Clojure: dexec/lang-clojure
  • CoffeeScript: dexec/lang-coffee
  • C++: dexec/lang-cpp
  • C#: dexec/lang-csharp
  • D: dexec/lang-d
  • Erlang: dexec/lang-erlang
  • F#: dexec/lang-fsharp
  • Go: dexec/lang-go
  • Groovy: dexec/lang-groovy
  • Haskell: dexec/lang-haskell
  • Java: dexec/lang-java AND johnhany97/grader-junit
  • Lisp: dexec/lang-lisp
  • Lua: dexec/lang-lua
  • JavaScript: dexec/lang-node
  • Nim: dexec/lang-nim
  • Objective C: dexec/lang-objc
  • OCaml: dexec/lang-ocaml
  • Perl 6: dexec/lang-perl6
  • Perl: dexec/lang-perl
  • PHP: dexec/lang-php
  • Python: dexec/lang-python
  • R: dexec/lang-r
  • Racket: dexec/lang-racket
  • Ruby: dexec/lang-ruby
  • Rust: dexec/lang-rust
  • Scala: dexec/lang-scala
  • Bash: dexec/lang-bash

Example usage

The following commands are to be ran from the project's root directory.

grader -schema="examples/inputOutput/javaSchema.json"

grader -schema="examples/python/pySchema.json"

grader -schema="examples/inputOutput/goSchema.json"

Schema Properties

  • file - Name of the file being graded, should contain it's extension - required
  • language - Name of the programming language in which the file being graded is written - optional
  • className - Class name of the file provided, usually is the name without the extension - required by junit and pyunit tests
  • folder - Folder within which the file provided is located - required
  • tests - Array of test tasks - required (even if just empty array)
  • outfile - Whether to store results in a file or just output it in the terminal - default = ''

Test Tasks

Currently supported are 4 different types of test tasks. They should confirm to the following format:

{
  "type": "TYPE_OF_TEST", // io - junit - pyunit - output
 
  // io --- start
  
  "input": [ " " ], // Array of strings where each string is a line in the input. They are joined by \n
  "expectedOutput": [ " " ], // Array of strings where each string is a line in the expected output. They are joined by \n
  
  // io --- end
 
  // junit/pyunit --- start
  
  "unitTest": "", // The test to be injected in the shell file, after being escaped
  
  // junit/pyunit --- end
  
  // output --- start
  
  "expectedOutput": [ " " ], // Array of strings where each string is a line in the expected output. They are joined by \n
  
  // output --- end
  
}

Example Schema

{
  "file": "Solution.java",
  "language": "java",
  "className": "Solution",
  "folder": "examples/inputOutput/",
  "tests": [
    {
      "type": "javaStyle"
    },
    {
      "type": "io",
      "description": "Testing normal boundaries",
      "input": [
        "3",
        "1",
        "2",
        "3"
      ],
      "expectedOutput": [
        "1",
        "2",
        "3"
      ]
    },
    {
      "type": "io",
      "input": [
        "2",
        "2",
        "5"
      ],
      "expectedOutput": [
        "2",
        "5"
      ]
    },
    {
      "type": "junit",
      "unitTest": "@Test\n  public void adderWorksWithZero() {\n    Solution s = new Solution();\n    int actual = s.adder(0, 3);\n    assertEquals(3, actual);\n  }"
    }
  ],
  "outfile": ""
}

Output: (Order may vary as tasks are finished by the job worker)

[  
   {  
      "test":{  
         "type":"javaStyle",
         "description":"",
         "input":null,
         "expectedOutput":null,
         "unitTest":""
      },
      "description":"Results generated by Checkstyle. A static analyzer built for analyzing Java Programs.",
      "stdOut":"Starting audit...\n[WARN] /Users/john/projects/dissertation/grader/examples/inputOutput/Solution.java:1: Using the '.*' form of import should be avoided - java.util.*. [AvoidStarImport]\n[WARN] /Users/john/projects/dissertation/grader/examples/inputOutput/Solution.java:4:3: Missing a Javadoc comment. [JavadocMethod]\n[WARN] /Users/john/projects/dissertation/grader/examples/inputOutput/Solution.java:16: Comment has incorrect indentation level 0, expected is 2, indentation should be the same level as line 18. [CommentsIndentation]\n[WARN] /Users/john/projects/dissertation/grader/examples/inputOutput/Solution.java:16: Line is longer than 100 characters (found 211). [LineLength]\nAudit done.",
      "stdErr":"",
      "successful":false,
      "similarity":0,
      "timedOut":false
   },
   {  
      "test":{  
         "type":"io",
         "description":"Testing normal boundaries",
         "input":[  
            "3",
            "1",
            "2",
            "3"
         ],
         "expectedOutput":[  
            "1",
            "2",
            "3"
         ],
         "unitTest":""
      },
      "description":"Expected:\n1\n2\n3\nGot:\n1\n2\n3\nTest passed: true",
      "stdOut":"1\n2\n3",
      "stdErr":"",
      "successful":true,
      "similarity":1,
      "timedOut":false
   },
   {  
      "test":{  
         "type":"io",
         "description":"",
         "input":[  
            "2",
            "2",
            "5"
         ],
         "expectedOutput":[  
            "2",
            "5"
         ],
         "unitTest":""
      },
      "description":"Expected:\n2\n5\nGot:\n2\n5\nTest passed: true",
      "stdOut":"2\n5",
      "stdErr":"",
      "successful":true,
      "similarity":1,
      "timedOut":false
   },
   {  
      "test":{  
         "type":"junit",
         "description":"",
         "input":null,
         "expectedOutput":null,
         "unitTest":"@Test\n  public void adderWorksWithZero() {\n    Solution s = new Solution();\n    int actual = s.adder(0, 3);\n    assertEquals(3, actual);\n  }"
      },
      "description":"",
      "stdOut":"JUnit version 4.10\r\n.\r\nTime: 0.009\r\n\r\nOK (1 test)",
      "stdErr":"",
      "successful":true,
      "similarity":0,
      "timedOut":false
   }
]

Test and Benchmark runs

Last Test run:

=== RUN   TestExecute
--- PASS: TestExecute (8.65s)
=== RUN   TestExecuteWithInput
--- PASS: TestExecuteWithInput (8.89s)
=== RUN   TestExecuteJUnitTests
--- PASS: TestExecuteJUnitTests (4.23s)
=== RUN   TestExecutePyUnitTests
--- PASS: TestExecutePyUnitTests (2.38s)
=== RUN   TestExecuteJavaStyle
--- PASS: TestExecuteJavaStyle (0.66s)
PASS
coverage: 77.9% of statements
ok      github.com/johnhany97/grader/processors 24.823s coverage: 77.9% of statements
=== RUN   TestRunTestInputOutputTestHandler
--- PASS: TestRunTestInputOutputTestHandler (28.10s)
    input_output_test.go:98: Successfully graded the solution for .java language
    input_output_test.go:98: Successfully graded the solution for .py language
    input_output_test.go:98: Successfully graded the solution for .cpp language
    input_output_test.go:98: Successfully graded the solution for .cs language
    input_output_test.go:98: Successfully graded the solution for .java language
    input_output_test.go:98: Successfully graded the solution for .py language
    input_output_test.go:98: Successfully graded the solution for .cpp language
    input_output_test.go:98: Successfully graded the solution for .cs language
    input_output_test.go:98: Successfully graded the solution for .java language
    input_output_test.go:98: Successfully graded the solution for .py language
    input_output_test.go:98: Successfully graded the solution for .cpp language
    input_output_test.go:98: Successfully graded the solution for .cs language
=== RUN   TestNewResultInputOutputTestHandler
--- PASS: TestNewResultInputOutputTestHandler (0.00s)
    input_output_test.go:187: Successfully obtained the expected result
    input_output_test.go:187: Successfully obtained the expected result
=== RUN   TestHandleErrInputOutputTestHandler
--- PASS: TestHandleErrInputOutputTestHandler (0.00s)
=== RUN   TestRunTestJavaStyleTestHandler
--- PASS: TestRunTestJavaStyleTestHandler (0.62s)
    java_style_test.go:49: Successfully style checked the solution for .java language
=== RUN   TestNewResultJavaStyleTestHandler
--- PASS: TestNewResultJavaStyleTestHandler (0.00s)
    java_style_test.go:119: Successfully obtained the expected result
    java_style_test.go:119: Successfully obtained the expected result
=== RUN   TestHandleErrJavaStyleTestHandler
--- PASS: TestHandleErrJavaStyleTestHandler (0.00s)
=== RUN   TestRunTestJUnitTestHandler
--- PASS: TestRunTestJUnitTestHandler (8.23s)
    junit_test.go:65: Successfully graded the solution for .java language
    junit_test.go:65: Successfully graded the solution for .java language
=== RUN   TestNewResultJUnitTestHandler
--- PASS: TestNewResultJUnitTestHandler (0.00s)
    junit_test.go:138: Successfully obtained the expected result
    junit_test.go:138: Successfully obtained the expected result
=== RUN   TestHandleErrJUnitTestHandler
--- PASS: TestHandleErrJUnitTestHandler (0.00s)
=== RUN   TestRunTestOutputTestHandler
--- PASS: TestRunTestOutputTestHandler (8.67s)
    output_test.go:66: Successfully graded the solution for .java language
    output_test.go:66: Successfully graded the solution for .py language
    output_test.go:66: Successfully graded the solution for .cpp language
    output_test.go:66: Successfully graded the solution for .cs language
=== RUN   TestNewResultOutputTestHandler
--- PASS: TestNewResultOutputTestHandler (0.00s)
    output_test.go:145: Successfully obtained the expected result
    output_test.go:145: Successfully obtained the expected result
=== RUN   TestHandleErrOutputTestHandler
--- PASS: TestHandleErrOutputTestHandler (0.00s)
=== RUN   TestRunTestPyUnitTestHandler
--- PASS: TestRunTestPyUnitTestHandler (3.36s)
    pyunit_test.go:54: Successfully graded the solution for .py language
    pyunit_test.go:54: Successfully graded the solution for .py language
=== RUN   TestNewResultPyUnitTestHandler
--- PASS: TestNewResultPyUnitTestHandler (0.00s)
    pyunit_test.go:109: Successfully obtained the expected result
    pyunit_test.go:109: Successfully obtained the expected result
=== RUN   TestHandleErrPyUnitTestHandler
--- PASS: TestHandleErrPyUnitTestHandler (0.00s)
PASS
coverage: 93.3% of statements
ok      github.com/johnhany97/grader/test       48.986s coverage: 93.3% of statements

Last Benchmark Run:

goos: darwin
goarch: amd64
pkg: github.com/johnhany97/grader/processors
BenchmarkExecute-12                            2        2687419688 ns/op           18880 B/op        177 allocs/op
BenchmarkExecuteWithInput-12                   2        2746329358 ns/op           19224 B/op        183 allocs/op
BenchmarkExecuteJUnitTests-12                  2        4290938234 ns/op           12016 B/op        100 allocs/op
BenchmarkExecutePyUnitTests-12                 5        1649888741 ns/op           18478 B/op        182 allocs/op
BenchmarkExecuteJavaStyle-12                  10         622639507 ns/op           13876 B/op        115 allocs/op
PASS
ok      github.com/johnhany97/grader/processors 53.897s
goos: darwin
goarch: amd64
pkg: github.com/johnhany97/grader/test
BenchmarkRunTestInputOutputTestHandler-12              2        2855917014 ns/op           20500 B/op        197 allocs/op
BenchmarkNewResultInputOutputTestHandler-12     20000000               430 ns/op              96 B/op          7 allocs/op
BenchmarkHandleErrInputOutputTestHandler-12     200000000               45.9 ns/op             0 B/op          0 allocs/op
BenchmarkRunTestJavaStyleTestHandler-12               10         664239069 ns/op           14096 B/op        117 allocs/op
BenchmarkNewResultJavaStyleTestHandler-12       200000000               44.0 ns/op             0 B/op          0 allocs/op
BenchmarkHandleErrJavaStyleTestHandler-12       200000000               46.9 ns/op             0 B/op          0 allocs/op
BenchmarkRunTestJUnitTestHandler-12                    2        4260429266 ns/op           13772 B/op        105 allocs/op
BenchmarkNewResultJUnitTestHandler-12           100000000               57.2 ns/op             0 B/op          0 allocs/op
BenchmarkHandleErrJUnitTestHandler-12           200000000               37.6 ns/op             0 B/op          0 allocs/op
BenchmarkRunTestOutputTestHandler-12                   2        2676039517 ns/op           20284 B/op        188 allocs/op
BenchmarkNewResultOutputTestHandler-12          20000000               399 ns/op              92 B/op          6 allocs/op
BenchmarkHandleErrOutputTestHandler-12          200000000               45.6 ns/op             0 B/op          0 allocs/op
BenchmarkNewResultPyUnitTestHandler-12          100000000               56.4 ns/op             0 B/op          0 allocs/op
BenchmarkHandleErrPyUnitTestHandler-12          200000000               36.7 ns/op             0 B/op          0 allocs/op
PASS
ok      github.com/johnhany97/grader/test       141.407s

About

Automatic Programming Assignments Grader in Go

Topics

go programming job-queue grader assignments

Resources

Readme
Activity

Stars

1 star

Watchers

2 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

No packages published

Languages