Module	Done	Todo	Skip
🟢 audio	28	0	0
🟢 data	12	0	0
🟢 event	4	0	2
🟢 filesystem	29	0	2
🟢 font	7	0	0
🟡 graphics	102	2	1
🟢 image	5	0	0
🟢 math	20	0	0
🟡 physics	22	4	0
🟢 sound	4	0	0
🟢 system	6	0	2
🟢 thread	5	0	0
🟢 timer	6	0	0
🟢 video	2	0	0
🟢 window	34	0	2

The following modules are not covered as we can't really emulate input nicely:
joystick, keyboard, mouse, and touch

Running Tests

The testsuite aims to keep things as simple as possible, and just runs all the tests inside Löve to match how they'd be used by developers in-engine. To run the tests, download the repo and then run the main.lua as you would a Löve game, i.e:

WINDOWS: & 'c:\Program Files\LOVE\love.exe' PATH_TO_TESTING_FOLDER --console
MACOS: /Applications/love.app/Contents/MacOS/love PATH_TO_TESTING_FOLDER
LINUX: ./love.AppImage PATH_TO_TESTING_FOLDER

By default all tests will be run for all modules.
If you want to specify a module/s you can use:
--runSpecificModules filesystem,audio
If you want to specify only 1 specific method only you can use:
--runSpecificMethod filesystem write

All results will be printed in the console per method as PASS, FAIL, or SKIP with total assertions met on a module level and overall level.

When finished, the following files will be generated in the /output directory with a summary of the test results:

an XML file in the style of JUnit XML
a HTML file that shows the report + any visual test results
a Markdown file you can use with this github action

An example of all types of output can be found in the /examples
The visual results of any graphic tests can be found in /output/actual

Architecture

Each method and object has it's own test method written in /tests under the matching module name.

When you run the tests, a single TestSuite object is created which handles the progress + totals for all the tests.
Each module has a TestModule object created, and each test method has a TestMethod object created which keeps track of assertions for that method. You can currently do the following assertions:

assertNotNil(value)
assertEquals(expected, actual, label)
assertNotEquals(expected, actual, label)
assertRange(actual, min, max, label)
assertMatch({option1, option2, option3 ...}, actual, label)
assertGreaterEqual(expected, actual, label)
assertLessEqual(expected, actual, label)
assertObject(table)
assertPixels(imgdata, pixeltable, label)
assertCoords(expected, actual, label)

Example test method:

-- love.filesystem.read test method
-- all methods should be put under love.test.MODULE.METHOD, matching the API
love.test.filesystem.read = function(test)
  -- setup any data needed then run any asserts using the passed test object
  local content, size = love.filesystem.read('resources/test.txt')
  test:assertNotNil(content)
  test:assertEquals('helloworld', content, 'check content match')
  test:assertEquals(10, size, 'check size match')
  content, size = love.filesystem.read('resources/test.txt', 5)
  test:assertNotNil(content)
  test:assertEquals('hello', content, 'check content match')
  test:assertEquals(5, size, 'check size match')
  -- no need to return anything or cleanup, GCC is called after each method
end

Each test is run inside it's own coroutine - you can use test:waitFrames(frames) to pause the test for a small period if you need to check things that won't happen for a few seconds.

After each test method is ran, the assertions are totalled up, printed, and we move onto the next method! Once all methods in the suite are run a total pass/fail/skip is given for that module and we move onto the next module (if any)

For sanity-checking, if it's currently not covered or it's not possible to test the method we can set the test to be skipped with test:skipTest(reason) - this way we still see the method listed in the test output without it affected the pass/fail totals

Todo

Test classes that still need to be written:

graphics.Mesh
graphics.ParticleSystem
physics.Body
physics.Contact
physics.Shape (this will include physics.Fixture properties)

Runner Exceptions

The automated tests through Github work for the most part however there are a few exceptions that have to be accounted for due to limitations of the VMs and the graphics emulation used.

These exceptions are either skipped, or handled by using a 1px or 1/255rgba tolerance - when run locally on real hardware, these tests pass fine at the default 0 tolerance.

Test	OS	Exception	Reason
love.graphics.points	MacOS	1px tolerance	Points are offset by 1,1 when drawn
love.graphics.setWireframe	MacOS	1px tolerance	Wireframes are offset by 1,1 when drawn
love.graphica.arc	MacOS	Skipped	Arc curves are drawn slightly off at really low scale
love.graphics.setLineStyle	Linux	1rgba tolerance	Rough lines blend differently with the background rgba
love.audio.RecordingDevice	All	Skipped	Recording devices can't be emulated on runners

Future

add BMfont alts for font class tests (Rasterizer + GlyphData)
graphics.isCompressed() should have an example of all compressed files
ability to test loading different combinations of modules
performance tests?