User interface

[baileythegreen]

Scenario:

• I'm a user, fairly new to pyani (v3).
• I have run a few analyses that are now stored in the database.
• I now want to generate some plots or reports for some of those runs.
• I do not know any SQL.

Questions about the current UI:

• How do I know which run_ids are assigned to any particular run?
• Do I even know this, other than because the pyani plot information mentions a run_id as a required parameter?
• Is the expectation that I would run pyani report outdir --runs first, in order to learn this information?
  • If this is the case, it might be helpful to note that I have an example output file from this command that—aside from the time stamp—contains three identical rows:

# output of `--runs` option

run ID	name	method	date run	command-line
1	ANIm_2021-09-06T12:24:07.126560	ANIm	2021-09-06 12:24:07.126560	> /Users/baileythegreen/Software/miniconda3/bin/pyani anim -i scratch/input -o scratch/output --dbpath scratch/scratch.db --classes scratch/input/classes.txt --labels scratch/input/labels.txt -l scratch/scratch.log -v
2	ANIm_2021-09-06T12:25:17.674630	ANIm	2021-09-06 12:25:17.674630	/Users/baileythegreen/Software/miniconda3/bin/pyani anim -i scratch/input -o scratch/output --dbpath scratch/scratch.db --classes scratch/input/classes.txt --labels scratch/input/labels.txt -l scratch/scratch.log -v
3	ANIm_2021-09-06T12:59:58.194378	ANIm	2021-09-06 12:59:58.194378	/Users/baileythegreen/Software/miniconda3/bin/pyani anim -i scratch/input -o scratch/output --dbpath scratch/scratch.db --classes scratch/input/classes.txt --labels scratch/input/labels.txt -l scratch/scratch.log -v

The rows are:

• the second, for a successful version of a run;
• the third for the same run (where nothing was actually done, so any output would be identical to that of the second); and
• the first, for a failed version of a run that generated no output, and for which I get this error when I try to generate the run matrices:

! pyani report scratch --run_results 1 --dbpath scratch/scratch.db --run_matrices 1

Traceback (most recent call last):
  File "/Users/baileythegreen/Software/miniconda3/bin/pyani", line 11, in <module>
    load_entry_point('pyani', 'console_scripts', 'pyani')()
  File "/Users/baileythegreen/Software/pyani/pyani/scripts/pyani_script.py", line 126, in run_main
    returnval = args.func(args)
  File "/Users/baileythegreen/Software/pyani/pyani/scripts/subcommands/subcmd_report.py", line 277, in subcmd_report
    matrix = pd.read_json(matdata.data)
  File "/Users/baileythegreen/Software/miniconda3/lib/python3.8/site-packages/pandas/util/_decorators.py", line 214, in wrapper
    return func(*args, **kwargs)
  File "/Users/baileythegreen/Software/miniconda3/lib/python3.8/site-packages/pandas/io/json/_json.py", line 584, in read_json
    filepath_or_buffer, _, compression, should_close = get_filepath_or_buffer(
  File "/Users/baileythegreen/Software/miniconda3/lib/python3.8/site-packages/pandas/io/common.py", line 200, in get_filepath_or_buffer
    raise ValueError(msg)
ValueError: Invalid file path or buffer object type: <class 'NoneType'>

Ideas for improving the UI:

• Could the run_id (and relevant database file name) not be reported on stdout and put into the .log file, when one is written? It's fragile if the database file is deleted, of course; but it would allow for possibly easier association between a run and how one is expected to specify that run in order to retrieve results.
• Could a failed run (or one where no new comparisons were run) be annotated as such in the output of --runs? This could help clarify why two (or more) seemingly identical entries might be found.

[widdowquinn]

How do I know which run_ids are assigned to any particular run?

pyani report --runs (and eventually the Flask interface)

I have an example output file from this command that—aside from the time stamp—contains three identical rows

Can you share the output?

Could the run_id (and relevant database file name) not be reported on stdout and put into the .log file, when one is written?

Yes.

It's fragile if the database file is deleted, of course.

Yes, but that's true for all output of all tools - if you delete it you lose it. You could link the run ID from in log with the path to the output database, but that's also brittle if the database is moved. We can't ensure the relationship between the .log data and what's in a database at a particular filepath - we can only report what was true at the time of running. There is, I think, no solution to that.

Could a failed run (or one where no new comparisons were run) be annotated as such in the output of --runs

Yes. There's a status field in the runs table which is intended to be used for this.

[widdowquinn]

I think we should resist putting too much functionality into the tabular output, right now. Broadly-speaking, the direction I had in mind was to have the in-browser interface (via Flask) be the way that users interrogate the database in a detailed, interactive way.

On-disc tabular files:

• reorganise and reproduce data that is already present in the database (i.e. they are essentially redundant, and prone to sync errors after generation)
• consolidate and cement choices (e.g. columns reported) made by the developer, unless a means of customisation is provided.
• are very useful for onward calculations (e.g. in workflows)

The browser interface:

• reorganises but reflects, rather than reproduces, the data in the database
• can be dynamically changed to suit the user
• enables interactive plots, exploration and subsetting of data
• lacks interoperability with automated workflows, unless the API is exposed

Rather than try to predict what "A Typical User" would want, or attempt to provide for all cases, in the tabular output my idea was to enable users to get the data they wanted through the Flask interface. There is, obviously, a place for tabular output, and for that there are two competing design goals:

1. stable output that downstream developers/users can design against (e.g. tab/CSV of identity values)
2. flexible output that suits a user's purposes at that time (e.g. BLAST format 6 column choices)

These may need handled in different ways via the command-line.

Note though that v3 is still in (serious) Alpha, and the current output functionality isn't "final"- much of it is placeholder to establish how we get data out of the database and help settle on the ORM method/funtion calls we'll need.

[widdowquinn]

If we want to imagine our typical (non-genomeRxiv) user, I'd propose this:

• they have a new genome sequence or sequences that they want to place among a set of existing sequences
• they want to know where the likely species boundaries are, and on which side of these their own sequence(s) lie(s)
• they want clear graphical and quantitatice evidence for this that they can include in their manuscript to support their case against reviewer criticism

Some users (including me) want to be able to subset and explore the output dataset. The easier we make it, the more people will see the value, but I think our users are more "I have a question already, give me the answer" and the impending problem is likely to be that they have too many genomes to handle comfortably.

[That last problem is motivating for: (i) downloading type strains only; (ii) establishing ranges of equivalence for alternative distance measures that are differentially efficient; (iii) enabling efficient scheduling and maybe even extending to some kind of cloud support (e.g. Kubernetes to distribute jobs - as this will not be cheap for large runs, we need to be sure the engineering is sound).

[baileythegreen]

Regarding the on-disc discussion, for my particular goal here—making it easy to find out which run_id to give to another subcommand, creation of an on-disc file isn't strictly mandatory; although, if there are many runs, printing to stdout can be annoying. I also would not think such a summary needed (or would be likely) to support future calculations, as I don't imagine it including the right type of information.

I was thinking, in addition to the statistics, it would show the algorithm, and perhaps modifiable parameters that were set. Maybe a date.

As for the browser interface, that's fine for interactive work, but I think that is separate from the point of this discussion. It would be possible to see all of the information (and more) I have just listed in a dashboard, and to see plots interactively, but I don't think using the dashboard should be required if I only want to find out the run_id for the run I want to plot. (Running things like dashboards or Jekyll sites in the browser (even if they are only running locally) can be quite intensive for my laptop, so I personally only do that if I actually need to view a thing interactively.)

[widdowquinn]

TBF, the title of thius discussion is "User Interface" - I think contextualising which bits of the interface are likely to be used under what circumstances is helpful ;)

I was thinking, in addition to the statistics, it would show the algorithm, and perhaps modifiable parameters that were set. Maybe a date.

I think a date/time is important. It may be the most useful information depending on how the user records what they do. For example "that run I did last Thursday" as opposed to "the run I changed the colours for."

Running things like dashboards or Jekyll sites in the browser (even if they are only running locally) can be quite intensive for my laptop

It's not been particularly problematic for mine, and it should be considerably less intensive than running the comparisons…

[baileythegreen]

TBF, the title of thius discussion is "User Interface" - I think contextualising which bits of the interface are likely to be used under what circumstances is helpful ;)

It is; I just wanted to note that we were describing two different parts of it. And I think your post will be very handy as a reference when it comes time to actually write the browser interface.