need better integration of filepkg with packages

[masinter]

@rmkaplan writes

The readtable and the package in the DEFINE-FILE-INFO expression have to be installed before any sense can be made of a file. I discovered that you can’t even read the FILECREATED expression if you don’t deal with those. And it’s actually hard to deal with them, because the specified package may not be defined in that particular file but in some other file that you don’t know about.

With respect to the filecreated expression, FILECREATED itself is safely in the Interlisp package. It will show up as IL:FILECREATED if DEFINE-FILE-INFO says there is a different default package (e.g. “CL”). That’s OK, the problem is that the filecreated expression also has a list of changes—the names of functions, variables, etc. in the file that have been modified. And those are atoms either in the default package (no colon) or in some other package that may or not be known.

I ran into this when I was trying to extract the internal dates for the directory comparisons. I finagled it by temporarily disabling : in the specified readtable, so it wouldn’t break on unknown packages, and then I could just call READ to get the expression rather than writing my own special parser. (I thought of creating at least the default package on the fly, but wasn’t sure whether that would screw up the actual definition of that package if was loaded later.)

I think that this imperfect integration of packages with the filepkg is the source of other problems, for example, that FINDCALLERS and EDITCALLERS tend to break on files that reference not-yet-defined packages.

[rmkaplan]

The system function for reading the DEFINE-FILE-INFO and FILECREATED expression of a file is \PARSE-FILE-HEADER in MACHINEINDEPENDENT. It uses READ with the file's default readable and default package. And the problem is that it causes a package-not-found error if the changes-to expression includes symbols in packages that don't exist (perhaps even the file's default package if it is defined later in the file).

Some aberrant behavior may disappear if the error is suppressed in this particular context. What would happen if we simply skipped over (SKREAD?) the changes sub-expression of the FILECREATED, so that it shows up as NIL in the return? Are the changes used for anything other than human reading of the file header?

Or, the hack that I mentioned above for the directory comparison: turn off the package delim, so that some symbols would appear in the file's default package with an internal colon (SEDIT%:FOO). Symbols in system-known packages (IL, CL, XCL) could be corrected after the fact. But symbols in other packages would not be coded properly.

We might also think of creating the missing package on the fly. But then the system doesn't like packages to be defined twice, even if they have all the same attributes (e.g. the sequence (make-package "foo") (make-package "foo") causes an error). So an on-the-fly definition would cause a later actual definition to fail.

How bad would it be if we either skipped reading the changes, or in some cases we read them inaccurately? This wouldn't solve everything, but it would eliminate some of the top-level silly errors.

[masinter]

I'm OK with skipping the changes or even turning them into strings in the case when he DEFINE-FILE-INFO package is undefined. Basically, it's an error to try to load a file whose DEFINE-FILE-INFO package or readable is undefined.

[masinter]

some test cases (in internal/test directory) would help. This looks like it's nearly done.

[masinter]

what we have now works pretty well; I'm going to close this; open new issues with examples if there are cases where things aren't working properly.

[rmkaplan]

It doesn't work properly unless you define the package in the define-file-info of every file that makes use of it (and keep those definitions consistent). Or define the package in a common file and have the other consuming files have a loadfrom or load-package in their define-file-infos. And, however it is done, the package can't be defined in the coms of the file (as opposed to the define-file-info) unless there are no references to the package internal to that file.

The problem, as noted elsewhere, is that we don't have a notion of package-scope that is broader than the collection of files constituting a module that makes use of it, and an easy way of sharing that scope. Exports is a way of providing a cross-file compiling scope that is bigger than the doeval@compile/dontcopy commands of a single file. The package facility needs a similar conception to create a cross-file loading scope. E.g an easy/systematic/FILEPOS way for define-file-info to find and install just the packages located in other files, without loadfromming everything.

[nbriggs]

I notice that a number of multi-file systems have a fooDECLS file which they use to unify the definitions (e.g., LAFITEDECLS, SEDIT-DECLS) -- unfortunately, for the define-file-info package, it's too late by the time you notice you need to load the DECLS file to change the default for the rest of the reading of the surrounding file.

SEdit's decls file explains "This file is for declaring things which are shared among sedit files. This file is loaded at compile time by each sedit file, but the contents of this file are not copied into any of the compiled files. The RECORDS declarations are here, and they are INITRECORDed and SYSRECORDed in SEDIT-ACCESS. If you change a record, make sure sedit-access gets remade NEW. All GLOBALVARS declarations are done here, because they're needed at compile time, but the actual variable declarations are done in the appropriate file. Constants and Macros are declared here only if they are shared among files; otherwise they can remain in the appropriate file. "

[rmkaplan]

Yes, that’s the problem: if a package is referenced in a file, it has to be installed in the current environment by the time that file's define-file-info finishes, even if the package is defined later in that file or in another file. The package affects the loading itself, at a primitive level. Macros and other declarations may cause problems later on in a load, if things are being evaluated, but at least at that point you are in the game.

…

On Sep 19, 2021, at 1:31 PM, Nick Briggs ***@***.***> wrote: I notice that a number of multi-file systems have a fooDECLS file which they use to unify the definitions (e.g., LAFITEDECLS, SEDIT-DECLS) -- unfortunately, for the define-file-info package, it's too late by the time you notice you need to load the DECLS file to change the default for the rest of the reading of the surrounding file.

[masinter]

look at clos/DEFSYS

[masinter]

i think it is necessary (unfortunately) to limit the style of common lisp programs to ones that meet some constraints we need to decide on.

When I look at this issue, it seems like to make progress we'll need better examples of what the nature of the problem is.

==> discussion

[rmkaplan]

Maybe in this sequence or elsewhere, I think that package definitions on files (PACKAGES filecommand) should put recognizable comment-strings around the package definition, similar to what the EPORTS command does. Then DEFINE-FILE-INFO can be extended to scan and install the necessary packages from the local file (default) or from a reference to another file that that has been specified as containing the package. This is in addition to just naming the package (and hoping that it exists) or putting in a package-definition expression in the define-file-info.

Then, when the define-file-info is executed in LOAD or LOADFROM of file A, it can pre-load the package even if defined later in A, or it can (quietly) get the package from some other file B. Kind of like the extension we did to load non-existent image object definitions by doing WHEREIS on their get-function.