automatic tests

[sorokin]

I have perhaps quite unusual feature request. I would like edb to have automatic tests.

I did some testing a while ago. I tested edb under valgrind, under address sanitizer and with debug mode of libstdc++. As the result I found a few bugs. I have fixes for some of them and I'm going send patches when they are ready.

My problem is that I believe that fixes without tests are fragile. Nothing prevent the same code to break in the future again during some refactoring. That's why I would like edb to get some automatic tests.

I can imagine two types of automatic tests in edb:

1. Tests that uses edb as a library. They don't do any UI. They just initialize debugger and call some functions of it. Like "start this program for debugging", "set break point to this address", "run and wait until break point is hit", and so on. The programs to run can be some minimal (a few kb in size) programs designed to check the specific features of the debugger. Which test framework to use for these tests doesn't matter much in my opinion and can be decided later.
2. Tests that test edb as whole. Qt has ability to simulate clicks and user actions in the application (https://doc.qt.io/qt-5/qttestlib-tutorial3-example.html). I have never used them, but at first glance they looks pretty straightforward. These tests are primary designed to test UI. In theory they can be used to test debugger logic as well, but if we have many of them I expect them to run much slower than tests 1.

I believe we can do either 1 or 2 or both. At least for my use case either option is OK.

I think I can spend some time implementing these tests. But before I start I would like to ask your opinion.

• What type of tests do you think should be implemented? 1 or 2 or both or perhaps you have some other ideas?
• Tests EDB scrolls by bytes, not instruction #1 require edb to be split into a library part and a UI part. Looking at edb source code it is not apparent to me which translation unit should go to which part. Do you think making this split is feasible. If not perhaps just tests 2 should be implemented.

At the moment I'm currently leaning toward implementing option 2.

---

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

[eteran]

I would love to have some tests be available. It's just not always easy to imagine how to implement them for an interactive GUI application such as edb...

I can definitely imagine a test that use the DebuggerCore plugin directly, uses it to manipulate a test binary, and looks at the results. That is certainly a "independent" unit which can/should be tested on its own, even without a GUI.

So as for the type of tests, I would say whatever is really doable and would be effective.

I am also curious what bugs you have found? Can you please report them so I am aware of them, even if you intend to fix them yourself?

[sorokin]

I can definitely imagine a test that use the DebuggerCore plugin directly, uses it to manipulate a test binary, and looks at the results. That is certainly a "independent" unit which can/should be tested on its own, even without a GUI.

In my case one bug is related to RunUntilRet which is implement in main binary, therefore testing DebuggerCore alone is not enough.

So as for the type of tests, I would say whatever is really doable and would be effective.

OK, then I would like to try the option 2. I tried to test a simple qt application, here what the test looks like:

    QTest::keyClicks(win.findChild<QLineEdit*>("lineEdit"), "foo");
    QTest::keyClicks(win.findChild<QLineEdit*>("lineEdit_2"), "bar");
    QTest::mouseClick(win.findChild<QPushButton*>("pushButton"), Qt::LeftButton);
    QCOMPARE(win.findChild<QLineEdit*>("lineEdit_3")->text(), QString("foobar"));

I'll try to make a prototype for edb, so we can have something to discuss.

I am also curious what bugs you have found? Can you please report them so I am aware of them, even if you intend to fix them yourself?

One problem is SIGABRT ("some debug event handlers weren't property removed") in DebugEventHandlers when edb is closed while RunUntilRet is running. Looks like RunUtilRet is not destroyed before DebugEventHandlers. I haven't investigated yet.

The others are valgrind reports of accessing uninitialized memory. Valgrind reports lots of issues, but most of them are manifestations of only few problems. I don't know how many distinct of them are there. I just decided to fix them one by one until valgrind output nothing. Also most of them are harmless (technically it's undefined behavior, but in practice they are harmless) as the uninitialized memory is accessed, something based on it is computed and then discarded. Nothing serious that's why I decided not to bother you until I have some patches.

I'll report what I found.