Extension to mpi programs #351

incardon · 2020-08-24T06:35:17Z

[] I have added an entry to docs/changelog.md

Summary of changes

This diff add support to work with mpi programs in Gdbgui.

The main changes regard adding the mpi_rank element to debug session a way to search the gdbsession by rank extend run_gdb_command with the process id that is default to -1. -1 in MPI based debug session -1 mean run the command on all sessions. in case of non-MPI is ingnored and -1 mean run on the single underline gdb session. An additional HTML page on the server to give to the client access to the MPI servers informations. Process information is attached to all gdb response answer in case of non MPI process information is -1. process information has been also attached to some function in Actions now process information of the message process, but because it is a defaulted to -1 so should not produce problem if there is code around calling the function in the old form.

inferior_program_paused has been changed to process processor information

change_process_on_focus: and refresh_state_for_gdb_pause has been added to process change of processor on focus and update of the GUI information

connect_to_gdbserver_mpi is added to handle the connection to MPI gdb server

Several global variables to initial_store_data has been added to store information for each processor

A processo status bar has been added when MPI programs has been debugged

gdbgui/src/js/Breakpoints.tsx has been fixed to remove breakpoint placed in templated code

All around there are change to handle process information, when needed

Test plan

2 tests has been added one for the JSX side that test server + Gdbgui TypeScript code using puppeteer and chrome headless, emulating click events and keyboard typing. The test go trough a full debug session.

The second test test is for the python server the test send command directly to the server to go through a debug session.
Tested by running

# command(s) to exercise these changes

The best is to look at the

.github/workflows/tests.yml

for the required machine setup.

The rest is:

launching an mpi program with the script ./gdbgui-mpi/launch_mpi_debugger 6 ./my_program -arg1 -arg2

where 6 is the number of process. Launch gdbgui, select connect to MPI gdbservers use *:6000 as input test for the server

incardon · 2020-08-24T06:37:24Z

I know is a big commit. I will first start to pull from the actual master. But would be nice to have a point of discussion in case there is interest to merge in the actual gdbgui.

Merge from base

incardon · 2020-08-24T19:21:16Z

ok I started to merge and I found the first problem when I try to use the server ...

File ".../gdbgui/gdbgui/server/sessionmanager.py", line 9, in
from pygdbmi.IoManager import IoManager
ModuleNotFoundError: No module named 'pygdbmi.IoManager'

Where do I get IoManager ? I do not see in the repo of pygdbmi

incardon · 2020-08-24T19:30:07Z

It is in add_reader branch ... of pygdbmi ... and is failing in your CI

Also I am not able to make this last refactor working:

The server die with:

File "/home/i-bird/Desktop/MOSAIC/OpenFPM_project/gdbgui/gdbgui/cli.py", line 254, in main
gdbgui.server.server.run_server(
AttributeError: module 'gdbgui.server' has no attribute 'server'

Not sure how it should works.

OK fixed with import gdbgui.server.server in line 25 of cli.py

cs01 · 2020-08-25T04:42:25Z

gdbgui/src/js/Actions.js

+      jQuery.ajax({
+        url: "mpi_processes_info",
+        success: function(data, status) {
+          data_lines = data.split(/\r?\n/);
+        },
+        async: false
+      });
+    }


I am going to eventually replace all jQuery.ajax calls with fetch.

It would be helpful if this was switched to use fetch, and the function turned into an async function. If you don't know what that means, don't worry about it. I will just fix it in a future update.

I will have a look. My programming skill in python and Javascript ( now TypeScript ) are basic, I am learning while going.

It’s fine to leave as is. I can migrate it along with the other calls at a later time.

cs01 · 2020-08-25T04:44:02Z

gdbgui/src/js/GdbApi.jsx

+   * @param state mpi state
+   * @return nothing
+   */
+  set_mpi_state: function(state) {


All files are now typescript, so arguments like state can be annotate with types now. I will be working to update existing code, but would like to see new code annotated from the start.

Yeah I saw merging with master ... I am already changing all of them doing it.

cs01 · 2020-08-25T04:47:42Z

Hi, thanks for the PR! Very impressive.

I just landed a whole bunch of changes to gdbgui recently. Unfortunately those changes have caused conflicts with this PR. Most of the code that was changed was on the backend. The frontend code was not affected very much other than the switch from .js/.jsx to .ts/.tsx. I just merged the pygdbmi add-reader branch into master and published a new version of pygdbmi (0.10.0.0) which is required by the latest version of gdbgui (0.14.0.0).

I looked at your screenshot in the related issue. The only thing I have to suggest is to move the processor selection to the right pane.

I am looking forward to seeing the merge conflicts resolved and giving it a try. I am curious -- is there a particular project or company these changes already being used in, or did you make this just in case you will need it in the future?

cs01 · 2020-08-25T04:49:29Z

gdbgui-mpi/__main__.py

@@ -0,0 +1,15 @@
+from gdbgui import backend


is this new file needed or can the existing cli.py be used?

Probably is removable, I will have a look

cs01 · 2020-08-25T04:50:26Z

gdbgui-mpi/compile.sh

@@ -0,0 +1,3 @@
+#! /bin/bash


I think the gdbgui-mpi folder might fit better in examples/mpi. There are already examples in the examples folder.

cs01 · 2020-08-25T04:58:24Z

gdbgui/src/js/ProcessorsButtons.jsx

+    store.connectComponentState(this, ["processors_states"]);
+  }
+  render() {
+    let btn_class = "btn btn-default btn-sm";


Bootstrap classes are being phased out in favor of tailwind css.

If you don’t know tailwind that’s is okay. I will update it all later.

cs01 · 2020-08-25T04:59:17Z

gdbgui/src/js/SourceCode.jsx

@@ -165,8 +169,10 @@ class SourceCode extends React.Component {
  ) {
    let row_class = ["srccode"];

-    if (is_gdb_paused_on_this_line) {
+    if (is_gdb_paused_on_this_line === 1) {


This is a boolean, it doesn't make sense to compare to a number. If the semantics around this have changed, the variable name and type should change.

cs01 · 2020-08-25T05:02:54Z

noxfile.py

@@ -13,7 +13,7 @@

 @nox.session(reuse_venv=True, python=python)
 def tests(session):
-    session.install(".", "pytest", "pytest-cov")
+    session.install(".", "pytest", "pytest-cov", "flask-socketio>4.2.0")


"flask-socketio should already be included (if necessary) with the . target

I will check again the problem of not having the > 4.2 it was that it was installing the version 2.9.3 (I think). And I needed flask-socketio bigger than 3.3 because of test_backend_mpi.py. I introduced a test that to emulate a debug session for the server and I had to connect the HTTP cookies of the flask client to the flask-io socket. This feature was introduced in flask-socketio 3.3

If the version is upgraded (and everything still works) that is fine, but it should be done in setup.py, which is the where Gdbgui defines its dependencies.

incardon · 2020-08-25T07:03:20Z

We are a research group specialized in Parallel simulations and I am one of the software developer in HPC. From long time I was searching for a small project that could the adapted to become an opensource parallel debugger for the HPC community. And this fullfill all the requirements. In few days was possible to create something working

cs01 · 2020-08-25T07:09:01Z

We are a research group specialized in Parallel simulations and I am one of the software developer in HPC. From long time I was searching for a small project that could the adapted to become an opensource parallel debugger for the HPC community. And this fullfill all the requirements. In few days was possible to create something working

Do you have a website I could check out?

incardon · 2020-08-25T07:12:48Z

http://mosaic.mpi-cbg.de/

While for the simulation library

http://openfpm.mpi-cbg.de

…on/gdbgui into extension_to_mpi_programs

Pietro Incardona added 10 commits August 1, 2020 12:15

Adding MPI support continuing

497a751

Adding multi process GUI elements

831c5de

Process on focus

20c9cf8

Adding MPI support

cf7307e

Cluster

936bc39

Adding mossing file

b3a0ffa

Nox testing passing

79e49ba

First pytest for mpi version

7bc9406

Refactoring tests for MPI

ffc4ae8

Testing JS with puppeters

c68ae89

incardon added 2 commits August 24, 2020 08:48

Merge pull request #1 from cs01/master

905b189

Merge from base

Fixing package.json

a867e74

cs01 reviewed Aug 25, 2020

View reviewed changes

Pietro Incardona added 4 commits August 27, 2020 16:48

MPI extension working with Pty

c7e1cef

Fixing tests

fc29ead

Fixing format to pass check

49498bb

Removing garbage

082cb87

i-bird and others added 30 commits January 26, 2021 23:26

Fixing tests

6c27678

Fixing tests

1b7369d

Fixing tests

c8048e7

Fixing pagination and sigint

973ac13

Fixing syntax

e0f6c9d

Fixing syntax

3e71552

Fixing syntax

37a6353

Fixing syntax

0c27d40

Fixing lint

39ee11f

Fixing lint

93831e6

Fixing test

8d15c97

Fixing SIGINT for MPI

6af6f3f

Set timeout for launching gdbgui server

38c1e75

Set timeout for launching gdbgui server

01059cd

Set timeout for launching gdbgui server

d261481

Fixing lint

3d6c4aa

Fixing test timeout

9e2a8ae

Adding saving breakpoints on cookies

9133063

Fixing breakpoints saved on cookies + activate tests

48c805a

Fixing format

e85a796

Fixing cookies breakpoints

e6fe204

Fixing formatting

509c596

Adding bigger timeout for connection

56ab743

Increasing even more timeout

722df00

Reducing number of processors for testing to 2

0b7bffb

Adding case where the first breakpoint is at line 8 rather than 10

d39f0e8

Fixing gdbgui

488f4fb

Merge branch 'extension_to_mpi_programs' of https://github.com/incard…

1252c83

…on/gdbgui into extension_to_mpi_programs

Fixing Pagination ... again

d8cded6

Fixing lint

fa184cc

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Extension to mpi programs #351

Extension to mpi programs #351

incardon commented Aug 24, 2020 •

edited

incardon commented Aug 24, 2020 •

edited

incardon commented Aug 24, 2020

incardon commented Aug 24, 2020 •

edited

cs01 Aug 25, 2020

incardon Aug 25, 2020

cs01 Aug 25, 2020

cs01 Aug 25, 2020 •

edited

incardon Aug 25, 2020

cs01 commented Aug 25, 2020 •

edited

cs01 Aug 25, 2020

incardon Aug 25, 2020 •

edited

cs01 Aug 25, 2020

cs01 Aug 25, 2020

cs01 Aug 25, 2020

cs01 Aug 25, 2020

cs01 Aug 25, 2020

incardon Aug 25, 2020 •

edited

cs01 Aug 25, 2020 •

edited

incardon commented Aug 25, 2020 •

edited

cs01 commented Aug 25, 2020

incardon commented Aug 25, 2020 •

edited

Extension to mpi programs #351

Are you sure you want to change the base?

Extension to mpi programs #351

Conversation

incardon commented Aug 24, 2020 • edited

Summary of changes

Test plan

incardon commented Aug 24, 2020 • edited

incardon commented Aug 24, 2020

incardon commented Aug 24, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cs01 Aug 25, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cs01 commented Aug 25, 2020 • edited

Choose a reason for hiding this comment

incardon Aug 25, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

incardon Aug 25, 2020 • edited

Choose a reason for hiding this comment

cs01 Aug 25, 2020 • edited

Choose a reason for hiding this comment

incardon commented Aug 25, 2020 • edited

cs01 commented Aug 25, 2020

incardon commented Aug 25, 2020 • edited

incardon commented Aug 24, 2020 •

edited

incardon commented Aug 24, 2020 •

edited

incardon commented Aug 24, 2020 •

edited

cs01 Aug 25, 2020 •

edited

cs01 commented Aug 25, 2020 •

edited

incardon Aug 25, 2020 •

edited

incardon Aug 25, 2020 •

edited

cs01 Aug 25, 2020 •

edited

incardon commented Aug 25, 2020 •

edited

incardon commented Aug 25, 2020 •

edited