READ-FROM-STRING-WITH-SOURCE-POSITIONS

[brpocock]

For second-generation source maps (ref: #87 ), the longer-term solution will require tighter integration with the Reader. Since READ returns whole objects, it could be rewritten to have a lower-level function JSCL:Read-With-Source-Positions (and JSCL:Read-From-String-With-Source-Positions) that returns the FILE-POSITION (resp. string index) associated with the start and end of (recursively) reading each object.

CL:READ can then be simply a wrapper that discards the extra values. Or, both can be (tail-call friendly) front-ends for a back-end function that takes a parameter as to whether or not to return the additional value. Or we could maybe just return the extra value all the time (is that allowed per ANSI?)

This could also be quite useful in implementing a nice Listener later, as it should also enable reading partial expressions … any incomplete strings, lists, &c. that trail off in the partially-read expression would be closed out, and the end-positions of them all set to NIL, indicating that they're unfinished. Thus, reading

   (LIST A B "C" "D

… the String "D" would have a start-position, but no end-position, as well as the containing List.

Type:

 (function (&optional (or stream boolean) t t t t) (values t list))

 (defun read-with-source-positions (&optional (stream *standard-input*)
                                                           (eof-error-p t) (eof-value nil)
                                                           (recursive-p nil) (incomplete-error-p t))
        …
        (values object-read alist-of-positions))

… where alist-of-positions references object-read and sub-objects used to create it, ergō

 "(:a :b \"string\" 99)"
 ↓
  (values #1=(:a :b #2="string" 99)
              ( ( #1# 0 18 )
                ( :a 1 2 )
                ( :b 4 5 )
                ( #2# 7 14 )
                ( 99 16 17 ) ))