Releases: Technologicat/mcpyrate
Version 3.6.0
3.6.0 (28 January 2022) New Year's edition:
Added:
- Python 3.10 support.
- Add block macro
mcpyrate.metatools.expand_first
. This can be used to force, within thewith expand_first[macro0, ...]:
block, the given macros to expand before others. Macros can be specified either by name (will be looked up in the current expander's bindings) or by hygienic capture. See examples in unit tests. - Add function
mcpyrate.utils.get_lineno
to conveniently extract alineno
from an AST-node-ish thing, no matter if that thing is an actual AST node, a list of AST nodes (i.e. statement suite), or an AST marker containing either of those, possibly recursively. - Facilitate programmatic inspection of the whole public API of
mcpyrate
. See the recipes in troubleshooting.- This is an interim solution while we decide whether to start supporting Sphinx at some point, so that we could auto-generate proper API docs from the docstrings (which are carefully maintained, and already contain all the necessary content).
Fixed:
- Fix #29, with thanks to @set-soft and @brathis for reporting.
mcpyrate
should now support Python 3.10. - Dialect subsystem fixes.
- Fix #30, thus extending the fix of #28 (in the previous release) into the dialect subsystem, too.
__future__
imports are accounted for both the dialect template and in user code that invokes the template.- This is implemented in the utility function
mcpyrate.splicing.splice_dialect
, so if your dialect definition uses that function in its AST transformer, now your dialect should not choke when the template and/or the user code have__future__
imports.
- Fix #31; the dialect machinery now has the infrastructure to pass in the source location info of the dialect-import statement.
- This allows dialects to mark any lines coming from the dialect template as effectively coming from the line that contains the dialect-import. If you import one dialect per line, this makes it easy to see which lines of the expanded code were injected by which dialect, for debugging purposes. (Recall that you can use the
StepExpansion
dialect frommcpyrate.debug
to see the line numbers before and after dialect expansion.) - During dialect expansion,
DialectExpander
automatically makes this info available inself.lineno
andself.col_offset
of your dialect definition (i.e. in the instance of your subclass ofDialect
, which has the transformer methods). In your AST transformer, you can pass these tomcpyrate.splicing.splice_dialect
. - See updated example dialects in
unpythonic.dialects
.
- This allows dialects to mark any lines coming from the dialect template as effectively coming from the line that contains the dialect-import. If you import one dialect per line, this makes it easy to see which lines of the expanded code were injected by which dialect, for debugging purposes. (Recall that you can use the
- Fix handling of rare case where the dialect template consists of a single statement that is not wrapped in a list.
- Fix #30, thus extending the fix of #28 (in the previous release) into the dialect subsystem, too.
- Docstring of
mcpyrate.utils.NestingLevelTracker
now has usage examples.
Version 3.5.3
Fixed:
- #28: Using
__future__
imports when multiphase compilation is enabled causesSyntaxError
.
Version 3.5.2
3.5.2 (22 June 2021) - Midsummer's eve edition:
Changed:
-
Small improvements to unparser:
- No space after unary
+
,-
or~
. - Future-proofing: yell if an unsupported constant value type is encountered.
- No space after unary
-
Add a new troubleshooting item on another Heisenbug that can occur when buggy macros are used inside a
with step_expansion
.
Version 3.5.1
3.5.1 (26 May 2021) - Detailed logbook edition:
Changed:
- Documentation improved. Particularly, AST markers are now documented (in the main user manual).
Version 3.5.0
3.5.0 (9 May 2021)
New:
-
Add
temporary_module
, a context manager that usescreate_module
, and automatically removes the temporary module fromsys.modules
when the context exits. -
Add a global postprocessor hook facility. Hooks are called, in the order registered, by
global_postprocess
when the macro expansion of a module is otherwise done. This e.g. allows a macro library to use its ownASTMarker
subclasses for internal communication between macros, and delete (only) its own markers when done. Seeadd_postprocessor
andremove_postprocessor
inmcpyrate.core
.
Fixed:
-
Run-time part of
n[]
: upon a parse error, make it clearer in the error message that what was being compiled was an invocation ofn[]
, not the whole source file. (Because these expressions are often one-liners, usuallylineno
will be1
, which otherwise looks confusing.) -
Fix error message in run-time typecheck of
a
(ast-unquote). Now it mentions correctly what was expected. -
Now
ASTMarker
may contain a statement suite (list
of AST nodes) as itsbody
.- The debug mode of
mcpyrate.unparse
now renders such bodies correctly. mcpyrate.markers.delete_markers
now deletes such markers correctly, splicing in thelist
of AST nodes where the marker was.
- The debug mode of
Version 3.4.1
3.4.1 (4 May 2021):
Changed:
- Update docs: as of
unpythonic
0.15, it runs onmcpyrate
, and provides fully functional example dialects based on a whole-module AST transformation. - The colorizer now injects some styles to
Style
that are missing fromcolorama
0.4.4, particularlyITALIC
.
Fixed:
- Now we pass a filename to
ast.parse
everywhere. This allows e.g.SyntaxError
during macro-import scanning (in the macro-import dependency graph analyzer), and possible internal errors in the interactive consoles, to report the filename correctly.
Version 3.4.0
3.4.0 (2 May 2021) - Quasiquotes ahoy edition:
New:
- The unparser now recognizes hygienic captures and destructures them in debug mode. This makes the result much more readable when you unparse an AST that uses a lot of hygienic unquotes.
-
To see it in action, use
mcpyrate.debug.step_expansion
macro onunpythonic.syntax.tests.test_lazify
. See particularly the HasThon test; both the autocurry and the lazifier produce many hygienic captures.Without this helpful destructuring, the macro-expanded code is completely unreadable, but with this, it only exhibits mild symptoms of parenthesitis. For example, this snippet:
filename=callsite_filename()
becomes, after autocurry and lazification,
filename=$h[Lazy]((lambda: $h[maybe_force_args]($h[force]($h[currycall]), $h[Lazy]((lambda: $h[force]($h[callsite_filename]))))))
Here each
$h[...]
is a hygienic capture. That's seven captures for this very simple input! Compare this notation to the actual AST representation of, e.g.,$h[Lazy]
:__import__('mcpyrate.quotes', globals(), None, (), 0).quotes.lookup_value(('Lazy', b'\x80\x04\x95 \x00\x00\x00\x00\x00\x00\x00\x8c\x13unpythonic.lazyutil\x94\x8c\x04Lazy\x94\x93\x94.'))
-
Fixed:
-
The importer now reports the source location if destructuring a macro invocation candidate fails.
- Some internal functions, including
mcpyrate.expander.destructure_candidate
, now take a mandatoryfilename
kwarg for this purpose.
- Some internal functions, including
-
Fix detection of globally bound macro invocations (hygienic macro captures) in the helper method
mcpyrate.expander.ismacrocall
. -
Fix syntax analysis for detecting
expr
macro invocations inmcpyrate.expander.destructure_candidate
. Version 3.3.0 (and only that version) errored out on the AST forf()[...]
even iff
was not bound as a macro.
Version 3.3.0
3.3.0 (29 April 2021) - Captain Debughook edition:
New:
-
Debug hook added to
mcpyrate.core.BaseMacroExpander
to see what the macro expander is doing. Thestep_expansion
macro now uses it (which see for usage), but you can also hook your own functions to it. -
Public function
mcpyrate.quotes.is_captured_value
for advanced macrology. This allows your own macros to detect expansions ofq[h[somename]]
in the AST, and grabsomename
(original name, no name mangling) as well as the corresponding value. (There is alsois_captured_macro
, but the use cases of that are much more limited.) Detailed explanation in docstrings for now. Usage examples in the tests for thequotes
module. -
mcpyrate.walkers.ASTTransformer
andASTVisitor
now have a methodgeneric_withstate
, to temporarily replace the state when visiting the direct children of the given node. (This is a closer equivalent formacropy
'sset_ctx
, sometimes useful for writing custom walkers.) -
Improve documentation on creating magic variables: add another major strategy, and explain both strategies in more detail.
Changed:
-
step_expansion
andstepr
now accept the string"detailed"
as a macro argument (in addition to the earlier"dump"
that selects the AST dump renderer).When
"detailed"
is given, they will report every macro expansion using the debug hook. This facilitates debugging of macros that expand inside-out (using explicit recursion). The definition of step remains the same: thestep
counter is incremented whenever the debug stepper gets control back. Just as previously, inside-out expansion therefore occurs within one step, but now you can see the subtree of each inner macro invocation just before and after that macro expands.In block mode
with step_expansion
, one complete step is defined as expanding each statement in the suite by one step.The macro arguments for
step_expansion
andstepr
can be passed in any order.
Fixed:
-
Fix subscript slice handling in unparser for Python 3.9 and later. Now that
ast.Index
andast.ExtSlice
are gone, anast.Tuple
may appear directly in the slice position, representing multi-dimensional indexing. Such a tuple must be rendered without surrounding parentheses, because the notationa[1,2:5]
is fine, buta[(1,2:5)]
is a syntax error. See https://bugs.python.org/issue34822 -
Fix bug in quasiquoting of constants: support also
...
(theEllipsis
singleton). -
Fix bug in
splice_ast_literals
(a.k.a. run-time part ofq
) that made it crash onast.Nonlocal
andast.Global
nodes. -
Fix bug in type preservation of empty list in
ASTTransformer
. -
Fix bug in copy support of
ASTMarker
objects. Now it is possible to deepcopy ASTs that contain markers. -
Fix bug that caused the
mcpyrate.debug.show_bindings
macro or the REPL consoles to crash upon a specific kind of broken imports in user code. (E.g. accidentally binding a macro name to a module object instead of a function object.) -
Fix bug failing to honor possible overrides to
sys.stderr
in various debug-printing facilities. Alwaysimport sys
and refer tosys.stderr
to resolve the current value, neverfrom sys import stderr
. -
Up to Python 3.8, items in the decorator list cannot be subscripted, so decorator macros could not take macro arguments. In 3.9 this has been fixed, as implied by the grammar. To work around this issue in earlier supported Python versions (3.6, 3.7, 3.8), we now support parentheses as an alternative syntax for passing macro arguments, like in
macropy
. Note that macro arguments must in any case be passed positionally! (Reasons documented in the comments ofmcpyrate.expander
.)
Version 3.2.1
3.2.1 (10 April 2021)
- Fix version metadata in
__init__.py
.
3.2.0 (10 April 2021) - X marks the spot edition:
New:
- Documentation: the staging compiler is now documented.
- Documentation: contribution guidelines now include a section on automated tests.
- Add command-line option to
macropython
to delete bytecode caches:- Use
macropython -c yourdirectory
(equivalent:macropython --clean yourdirectory
), whereyourdirectory
is a path (can be relative or absolute). - For a dry run, use
macropython -c yourdirectory -n
(equivalent:macropython --clean yourdirectory --dry-run
), which just prints the full paths to the directories it would delete. - If you need programmatic access to this functionality, see
mcpyrate.pycachecleaner
.
- Use
Fixed:
- Fix #20, with thanks to @thirtythreeforty for reporting.
mcpyrate
should now support Python 3.9. - Fix bug in
mcpyrate.splicing.splice_expression
. (Only affected that function; the expression mode ofa[]
uses a different code path.) - Fix a crash in the generation of some error messages in
mcpyrate.coreutils.get_macros
. Particularly, the crash could occur if the module is not found insys.modules
, or if an as-import of a macro is attempted with a concrete expander type that doesn't support that feature.
v3.1.0
3.1.0 (12 February 2021) - Compiling on the high seas edition:
New:
-
The
mcpyrate
compiler (implementing the import algorithm) is now exposed inmcpyrate.compiler
for run-time use.- You can just
expand
, or both expand andcompile
code, as needed. - It is now convenient to compile and run macro-enabled quoted code snippets (or source code) at run time, see the functions
mcpyrate.compiler.run
andmcpyrate.compiler.create_module
.- This makes it easier to test macros that are best tested via the behavior of the run-time code they output. (It also makes macro-enabled Python into a poor man's staged language [1] [2].)
- The system allows dynamically creating modules (for executing code snippets in) at run time, as well as running code in the namespace of an existing module.
- These features combine, so you can let
run
automatically create a module the first time, and then re-use that module if you want. - You can also create a module with a specific dotted name in
sys.modules
. The multi-phase compiler itself uses this feature.
- These features combine, so you can let
- Source code input supports dialects, macros, and multi-phase compilation. The source code represents a module.
- Quoted AST input supports macros and multi-phase compilation. No source transforms for this kind of input, because the input is already an AST. (Dialect AST transformers and postprocessors should work.) The top level of the quoted block (i.e. the body of a
with q as quoted:
) is seen by the compiler as the top level of a module. - While the code snippet is running, the module's
__file__
and__name__
attributes are available, as usual. - For extracting results into the surrounding context, just assign them to variables inside the code snippet. The top level of the code snippet is the module's top level. You have that module object available in the surrounding context (where you call
run
), so you can access those variables as its attributes.
- Full documentation is in docstrings for now, see
mcpyrate.compiler
. Usage examples can be found inmcpyrate.test.test_compiler
.
- You can just
-
Add support for PEP 582 - Python local packages directory in the
macropython
bootstrapper. -
The unparser now supports all three top-level node types, and supports also a
list
of AST nodes (e.g. a statement suite in an AST) as input. -
The
StepExpansion
dialect now works in AST-only mode, too.- It will enable
DialectExpander
debug mode in the source transform step, if that runs. If the AST transform step is reached and debug mode is still off, it will now enable debug mode at that time. Only one copy of the unprocessed code is printed regardless.
- It will enable
-
README: add instructions to configure Emacs syntax highlighting.
-
Add
unpyrate.bunch.bunchify
to convert an existing mapping instance into aBunch
.
Changed:
-
Nested quasiquotes now work properly.
Unquoting now only occurs when quote level hits zero. Inner quotes and unquotes are detected, for tracking the quote level, but are then left in the output as-is.
Note as-is means "as unexpanded macro invocations". Because the quasiquote operators are just macros, and in macro-enabled Python, the tradition is that a function actually being a macro is a property of the use site, not of its definition site, it follows that there's no guarantee whether the quote operators are in the expander's bindings at any later time. Even if they are, there is no guarantee whether they still have the names they had at the time when the outermost quote expanded.
What we have now is the result of taking the current design to its logical extreme. A better solution (for next-next-gen) may need a break from tradition, in that maybe a function being a macro should be a property of its definition site, not of its use site. Also, maybe the quasiquote operators should be considered core functionality, and not be renameable (like regular macros are).
However, the current solution does give useful level separation that has real practical applications; see the dynamically generated module example in
mcpyrate.test.test_compiler
.This is not considered a breaking change, because the previous behavior of nested quasiquotes didn't make any sense, so nothing useful could be built on it.
Fixed:
- Fix INTI-CMNB/KiBot#29, with thanks to @skorokithakis and @set-soft.
- Fix #21, with thanks to @thirtythreeforty for reporting.
- Fix bug in
unastify
: drop the run-time part ofq
. - Fix bug in
rename
: handle also module name inImportFrom
nodes. - Fix
SourceLocationInfoValidator
. macropython
now reportsmcpyrate
version separately from the version of themacropython
script itself when run with the-v
(--version
) command-line option.