notes.txt

Ideas / TODOs
-------------
experimental optimization branch
	checks and cleanup tasks
		new compiler:trace(str) method, only prints if self.tracing == true
		Make sure [ and ] work, since compile looks for target on stack
		back up compiling? on cstack in eval, and restore at end
	inline functions
		compiles to an immediate that compiles the firth code in the body inline
		implies immediate
		explicit?
			: ;inline postpone interpret compilebuf inlinefunc bindfunc immediate ;immed
		automatic?
			fewer than x words in def?
			fewer than y lines of generate lua in compilebuf?
	pre-cache used lua library functions as locals and close over them
		string
		io
		stringio
		table
	stack cache?
		use closed-over locals for quick-access to top stack elements?
		what kind of bookkeeping and calculations are needed to make this work?
		does the overhead dominate the possible savings?
		where do these values live? the compiler? the prims file?
	compile time optimization module
		likely to be complicated
			load as a separate, optional module
			patch itself into the compiler machinery
		calculate stack deltas
			: foo bar baz ;
			foo.delta = bar.delta + baz.delta
			capture stack usage even for immediates that compile executable code
				+.consumes = 2
				+.writes = 1
		build a simple tree to represent input
			use knowledge of ops and stack usage to group expressions
			transform and optimize tree is possible
			generate lua code from optimized tree
		stack elision
			use locals wherever possible
			commit stack changes lazily
			pre-calculate results when literals are arguments?
refactor input stream tokenization
	newlines(str)
		returns an array table of offsets into str where newlines live
		used in interpret() to set self.linenum in the processing loop
		e.g. { 0, 35, 71, 123, 151, 200, ... }
	nexttoken()
		rather than resizing input string, use token begin/end pointers
		makes extracting substrings faster and more memory efficient
		makes >ts faster and more memory efficient
		so we get: stringio.nexttoken(string, delim, searchstart) -> token, tokstart, tokend
		if only delims in string (or empty) return "" and start == end
			FIXES: will error if an input line has more than one trailing whitespace.
stack data type
	3>[], 4>[], n>[] ( n*x s n -- s )
	[3]>, [4], [n]> ( s n -- s n*x )
	[3]@, [4]@, [n]@ ( s n -- s n*x )
	: [ ( -- s ) // TS: i*x ']'
		immediate
		allocates new stack
		executes each following word
			pulls tokens in a loop
			executes or num converts, pushes onto stack object
		] breaks the loop and pushes the resulting stack object onto sys stack
	e.g.
		[] val: st
		1 2 3 st 3>[] [3]> .S clear // prints [1 2 3]
loops
	for: i/j/k
		implement with for
		pop TOS to local i
		e.g.
			users @ for: user genpassword user setpass end
	ranges
		return an iterator closure
		1 10 range		// -> 1 2 3 4 5 6 7 8 9 10
		1 10 range: 2	// -> 1 3 5 7 9
	each, map, reduce, etc.
		similar to foreach style loop, but applies an xt to each item
	reference forth loops
		end start DO xxx LOOP			// for i=start,end-1 ...
		end start DO xxx step +LOOP		// for i=start,end-1,step ...
		BEGIN xxx cond UNTIL			// do ... while(!cond)
		BEGIN xxx cond WHILE yyy REPEAT	// while(cond) yyy OR do xxx while(cond)
		LEAVE							// break
conversions
	>string
	>number
change parse routines to allow for doubled delimiters
	// e.g. alternative string syntax, for quoting options
	: '' ( -- s ) " ''" nextword parse ;immed
execution, lambdas, functional programming
	compiler:pushfunc()
		in contrast to execfunc()
		pushes self.target and restores
		throws, if not target or target.name ~= "ANON" or some such?
	lambdas
		has no name, or name is ANON or something
		leaves an xt on tos, using pushfunc()
		when implemented, closes over locals?
		possible names/syntax
			lambda: 5 > ;	// rewrite ; to push based on target.name?
			:( 5 > );		// ); will always do pushfunc()
			:NONAME 5 > ;
		e.g.
			:( 1- 2* ); data each
compiletime macros / constexprs?
	precalculate the value as it's compiled
	most useful for showing the calculation in a func, but speeding it up at runtime
	lispy macros
		(+ 1 12 foo ) // compiles to push(13+compiletimevalueof(foo))
	forthy macros
		[ 1 12 foo +] // compiles to the same
better error debug messagges (xpcall and debug lib)
	if "debug mode", compile in a local var that counts firth source line
recurse
	start all defs with -> " local function word()"
	compile as append " word()"
	end all defs with -> " end \n return word"
local variables
	given name mapped to new temp variable
		in nested situations, should we be pushing the locals map?
	local ( name -- )
		new primitive word, analogue to variable
		must only be used from an immediate because it compiles a new local decl
		local function newlocal()
			local name = stack:pop()
			compiler:newlocal(name, nil)
		end
		function compiler:newlocal(name, initialval)
			self.target.locals[name] = compiler:newtmp(initialval)
		end
	local? ( name -- b )
		new immediate prim word
		compiler:push(target.locals[name] ~= nil)
	setl ( x name -- )
		new prim, analogue to !
		must only be used from an immediate because it alters compile state
		local function setlocal()
			local name = stack:pop()
			local val = stack:pop()
			compiler:setlocal(name, val)
		end
		function compiler:setlocal(tmpname, value)
			self:append(tmpname..'='..tostring(value))
		end
	local: ( x -- ) // TS: name
		new primitive word, analogue to var:
		must be immediate because it alters compile state, reads input token
		: local: ( ) nextword dup local setl ;immed
	locals: ( -- ) // TS: names...
		allocates a list of local variables
		probably most useful at the top of a word def
		every space-delimited token following on the line is a new var name?
		: locals: ( ) char \n parse char %s split each local end ;immed
	setl: ( x -- ) // TS: name
		new prim, no analogue with global variables
		unlike !, this must be immediate to read tokens
		updates (but does not define) a local variable
		: setl: ( ) nextword setl ;immed
	=: (x -- ) // TS: name
		defines or updates a local, like go?
		immediate; checks at compile time which action to compile?
		: =: ( )
			nextword dup local? if
				setl
			else
				dup local setl
			end
		;immed
	e.g...
		// uses a local to keep the stack cleaner
		0 =: width
		100 loops
			nextfield dup printcontent widefield? if
				50 setl: width	// will print a wide break
			else
				25 setl: width	// will print a narrower break
			end
			// ...doing who knows what with the stack...
			width loops char . .raw end CR 
		end
	process for compiling a token becomes
		1. check target.locals[token]
		2. if not found, check dictionary
		3. if not found, attempt numeric conversion
		4. if fails, lookup error
	print user's locals in stack traces?
		at top level, or first after dumptrace
		reverse lookup on func 
		pull locals map from entry
access lua tables form firth land
	compiler
		new prim word
		local function pushcompiler() stack:push(compiler) end
	dictionary
		: dictionary ( -- dict ) compiler " dictionary" @@ ;
		or, for efficiency...
		local function pushdict() stack:push(dictionary) end
refactor scratch/last scheme
	should set us up for nested functions/lambdas
multiline comments and strings
	pending parse callback if parsing unsatisfied at end of line?
		char ) parse => if didn't actually find it, run parse again on next line
	in a loop, read and throw away lines from input device until close comment?
		need to rework compiler input scheme
			lines(file) returns an iterator function; useful?
		version of parse that returns whether the delimiter was found
			we can keep looping if not found
			regular parse uses this and drops the result?
param pattern matching
	core compiler/language feature?
	implemented through metaprogramming?
	// assumes something like : concat ( s1 s2 -- s1s2 ) ... ;
	: strreps	( s n -- n*s )
				(( _ 1 )) drop
				(( _ _ )) over swap loops over concat end swap drop
	;
runtime documentation
	call lists
		if foo is deleted or redefined, remove from bar.calledby
	attach stack diagram comments to words as a docstring
	attach preceding comments to words, in a specific form?
		append string to EOL, to internal buffer
		: checks this buffer, and attaches to dict entry, if it exists
interactive emacs mode
	repl in minibuffer
		use standard M-: key to jump to repl
		stay in minibuffer, in repl mode, between executions
		after minibuffer used for anything else (help, search) revert to repl
		output to secondary buffer
	code in main buffer
		executes code on save
		i/o redirected to minibuffer/secondary buffer
	syntax highlighting
		as words are defined, they are automatically added to font lock pattern list
		tokens can set self format and next-parsed format
		parse finds begin and end span for currently-set formatting
		found word can alway override its own formatting when "executed"
		variables to set
			wordstyle // default for any word
			numberstyle // default for numbers
			selfstyle // how to style this word, overrides whatever the default would have been
			nextstyle // how to style following word, overrides default and any self-styling
			wordstyle @ gray var: compilerstyle // wordstyle but gray; useful for styling compiler words
	code translation protocol
		main mechanism for running firth code from emacs
		translation layer for elisp <-> firth code
		collects firth code from buffers, sends via pipe to firth
		firth runs code, and sends back elisp to run
			could be as simple as display result
			could add font-lock rules for newly-defined words
create a vocabulary for writing executable grammars for parsing DSLs
whittle the lua codebase down to
	generic io and string handling
	access table fields
	compile buffers to lua functions
	call lua functions
FirthOS
	tiny os that will boot directly into interactive emacs firth mode
	stripped-down linux kernel
		terminal drivers and maybe framebuffer
		audio of any sort?
		simplest/easiest FS support
		single-user mode
		needs virtual memory and multitasking
	minimal set of support utils and libs
		probably things like ls, rm, mkdir, and such
		libc
		ncurses
		readline?
	stripped-down emacs
		remove modes, routines, and elisp libraries that aren't applicable
	luajit
	firth