LSMB Help Proposal

[neilt]

LSMB Help Proposal

This document proposes embedding all help and documentation into a running LSMB system for both users and developers using various parts of LLM (large language models), AI (artificial intelligence) and NLP (natural language processing).

When someone needs to know something, instead of trying to figure where to find the answer, this proposal puts all of the answers in a defined location using a focused ChatGPT like search and response process that does not access any external servers.

Why?

2. Eliminates wasted user time scanning or searching through the book trying to find help. While the book table of contents provides context, it is not very granular. The book index is granular, but does not provide context. In addition to the book, the wanted information may be on the website, in an FAQ, in the release notes, in an open or closed issue on github, in the code comments, in a github discussion, etc.
3. Easier to establish context. One problem with the book is that in order to document a fact, procedure, or explanation (hereinafter called help) and have it make sense, a context must be established. This takes additional time, which is often unavailable, and leads to delays in book updates. Sometimes this delay is infinite.
4. Help being provided directly to the user on the screen at the point it is needed is automatically in context.
5. This proposal will make help closer to the code and easier for developers to update help when the code changes.
6. Developers don't have to know LaTeX.
7. Documentation version is automatically matched to code version.
8. PDF files are easier to search, HTML files are not. By embedding search at the text level, in the source, some complexity is removed. Remember LSMB is primarily an HTML generator.
9. Using LLM, AI, and NLP is modern, popular, and will separate LSMB from the competition. This may also draw development resources to the project from people who want to establish their AI credentials.

Overview

The help subsystem will be comprised of both User and Developer help.

1. User help will always be on. The defaults system will allow the LSMB installer or user (need to decide) to install Developer help and to turn it on or off. Most installations will not need developer help.
2. On each view or screen, if there is User help available for that screen, potentially multiple help icons will be shown in the appropriate place(s) on the screen. When the user clicks on a help icon, LSMB opens a pop up window (toast) with the localized help that is specific to the appropriate view, field, or procedure.
3. The main menu will also contain an item labeled 'Help'. Selecting "Help" will show a screen with 2 components, an input search textfield and a multiline response textview. When the user types in the input textfield, LSMB will search for related help using various AI, NLP, LMM technologies and show what it finds in the multiline results textview.

Design

User help

Screen based help will be available via an icon or multiple icons on a screen. When the user click on the icon the help will be loaded from the database and shown in a pop-up. If the database help text contains image links, they will also be loaded.

The screen based help will be developed either specifically for the help system or moved from the book, and stored in the database help table. For an example help table, see below.

Where screen fields directly match database fields, the help can be loaded from the database column comments. If necessary, the database column comments can contain both user and developer comments. The database column comment can be annotated so the appropriate text is used.

Translations will be exported from the database, updated, then imported back into the database.

User help will be available for both login.pl and setup.pl

While some help may need to be loaded into the help table via a configuration file, it is anticipated that a view will, if necessary, add or update its own help in help table when it is initialized. That way help can updated when any changes are made to the view simply by including the new help in the view code.

Now one might wonder why use the help table at all? Good question. The answer is that the help table is needed for the ChatGPT like search functionality.

In my own code I am not yet using a LLM, so I maintain vectorized search strings separate from vectorized response text. Some more research or someone more knowledgable is needed here. If that is needed then the help table example below will change.

Developer help

Developer help will make available all text that is in perl files (perldoc), database comments, etc. via a ChatGPT like (question & answer) interface. Essentially any text that can be scanned in the code base.

The only difference between User help and Developer help is the depth of content.

All developer help will come from scanning developer generated files. No manual help writing will be used. As such, it is not expected to significantly increase the LSMB download size.

Technical Requirements

With any enhancement there is the need to evaluate added dependencies. The following dependencies are envisioned.

Screen based help may need the following:

1. Text::Markdown - Certainly HTML or LaTeX help could be used, but most non-UI developers are more comfortable with Markdown.

Searchable based help may need the following:

1. PostgreSQL pgvector extension. For storing and searching sentence embedding vectors.
2. Lingua::*. For processing and creating sentence vectors?
3. Text::Sentence. For processing and creating sentence vectors?
4. Algorithm::NaiveBayes. For processing and creating sentence vectors?

Questions

1. Is there a need to keep the book?
   • Does publishing a book in 2024 have any marketing value?
2. In the new UI can each field have a pop-up?
3. Will pop-ups be enhanced by showing images (screen views)?
4. Is there a need to scan Vue files for Developer documentation?
5. One thing the book is good at is the tutorial. Two options exist.
   • Leave the tutorial in the book. Probably the best short term solution.
   • Move the tutorial to a guided tutorial in LSMB. The better long term solution, but poses some technical problems, including developer resources.
6. What primary format should the help text be in the database? Some options:
   • Template format using Template::Toolkit for the conversion. Which while already used in LSMB is complicated and not straightforward for most developers.
   • Markdown format using Text::Markdown for the conversion. Which makes the source easier to understand.
   • Plain text. Easiest of all, but does not allow easy inclusion of images or text enhancements that aid in understanding.
   • HTML. Yuk, enough said.
7. Is there any unification that should be considered with the existing localization code?
8. Should the help key be an integer as shown below in the example help table with no meaning, should it be a string, with maybe a column for 'View Name' and a column for 'Field Name', or should it be something else.
9. Should the screen help be limited to pop-ups or should there also be the possibility of navigating away from the screen to full window help.

Consequences

1. The book will go away?
2. The release process will be more complicated, but most of the complications will be automated.
3. Comments will have to be added to and maintained for every table, column, trigger, etc. in the database.

Step to implementation

1. Make sure all SQL entities have comments.
   • tables have comments. Can be on-going.
   • table columns have comments. Can be on-going.
   • SQL functions have comments. Can be on-going.
   • SQL triggers have comments. Can be on-going.
2. Make sure all perl functions have correct perldoc. Can be on-going.
3. Create and release the help table and its update procedures. Does not need to be fully populated, but should be populated for one view that is using the new UI. Just get the initial infrastructure in place and released.
4. Then migrate more views as time permits.
5. Add code to scan and update the help table.
6. Start moving more external content to LSMB.
   • Incorporate FAQs.
   • Incorporate book into into the appropriate source.
   • Incorporate other web information.

Technical Notes

The following is only provided to enhance understanding of this proposal.

CREATE TABLE help
(
    help_id         integer   NOT NULL   GENERATED BY DEFAULT AS IDENTITY,
    "language"      text      NOT NULL   DEFAULT 'en-US',  -- IETF BCP 47 language tag
       
    help_format     text      NOT NULL,  -- text, markdown, html, template, etc.
    help_content    text      NOT NULL,  -- actual help content, needs to be translated
    context_labels  text[],              -- assists plain text search, needs to be translated

    help_vector     vector(512),         -- sentence embedding vector (pg_vector) for matching,
                                         -- needs recalculating
                                         -- after every change or translation
    
    source_program  text,                -- automated help creation program name
    source_key      text,                -- source key used to match source auto updates

    notes_to_tr     text,                -- notes to translators
    notes_from_tr   text,                -- notes from translators
    
    create_ts       timestamptz      NOT NULL DEFAULT transaction_timestamp(),
    update_ts       timestamptz      NOT NULL DEFAULT transaction_timestamp(),
    
    PRIMARY KEY (help_id, "language"),
    UNIQUE (source_program, source_key)
);

[neilt]

I've been researching using Perl for calculating sentence embedding vectors, but have not found anything that seems promising.

It looks like we might have to calculate the sentence embedding vectors outside of LSMB using Python. I know, yuck. The general process is described here.

If anyone knowns of any perl alternatives, please add them to the discussion.