ENH: Implement support for with-blocks. #59

ssanderson · 2015-11-18T03:38:18Z

I missed the presentation tonight at Boston Python, but a colleague pointed me at this repository.

This PR is a work in progress toward support for with-blocks.

The basic idea here is rewrite the ast of the with-block into two
statements and a try-except, which we already know how to oneline-ify

In general, we can rewrite a with block of the form:

    with <expr> as <ctxname>:
        <body>

as

    __anonymous = <expr>
    <ctxname> = __anonymous.__enter__()
    try:
        <body>
    except:
        if not __anonymous.__exit__(*sys.exc_info()):
            raise
    else:
        __anonymous.__exit__(None, None, None)

Once we've rewritten the AST, we should be able to just delegate to existing the existing functionality for sequencing statements and handling try-except blocks.

This mostly works as-is, but for reasons I don't quite understand, statements after the with-block aren't being executed. I suspect that I'm not correctly doing whatever work is needed to sequence the with-block with subsequent statements. Any guidance here would be appreciated.

The basic idea here is rewrite the ast of the with-block into two statements and try-except, which we already know how to oneline-ify In general, we can rewrite a with block of the form: with <expr> as <ctxname>: <body> as __anonymous = <expr> <ctxname> = __anonymous.__enter__() try: <body> except: if not __anonymous.__exit__(*sys.exc_info()): raise else: __anonymous.__exit__(None, None, None)

ssanderson · 2015-11-18T03:39:43Z

main.py

@@ -177,7 +180,10 @@ def var(self, name):

    def store_var(self, name):
        name = self.mangle(name)
-        sym = self.table.lookup(name)
+        try:
+            sym = self.table.lookup(name)


These were needed because I'm inserting assignments to names that weren't in the original source, which means they aren't in the symbol table.

andersk · 2015-11-18T07:35:57Z

Thanks for looking at this.

Sequencing subsequent statements is the purpose of the {after} template parameter used by all the existing statement translators (except the terminating statements break, continue, raise, and return).

Have you read PEP 343? It gives a slightly different translation of the with statement. The most important difference is the finally: block, which ensures that __exit__ is called even if the body jumps out of the with: block using break, continue, or return. A more subtle difference is when the __exit__ method is looked up—this matters because __getattr__/__getattribute__ could have side effects, and we’ve tried to be careful about preserving evaluation order even to this level of detail (tests/eval_order.py).

In general, I think it would be better to factor out the try…except translation code into functions that can be reused without literally generating and re-parsing fake ast nodes. You’ve already discovered some of the reasons. For example, your var and store_var kludge will make bugs elsewhere harder to find—those KeyErrors are really important for making sure we stay meticulously synchronized with the symtable module by visiting child namespaces in the same order (tests/symtable_torture.py).

Such refactoring could also tremendously simplify the resulting translated code. For example, the sys.exc_info() call could be eliminated because the __exit__ handler used for catching exceptions already gets passed this information.

ssanderson · 2015-11-22T22:55:48Z

@andersk thanks for the thorough read-through. Responses to notes in line:

Have you read PEP 343? It gives a slightly different translation of the with statement. The most important difference is the finally: block, which ensures that exit is called even if the body jumps out of the with: block using break, continue, or return.

Ah, that's a good point. I hadn't considered jumping out of the with block from a loop.

A more subtle difference is when the exit method is looked up—this matters because getattr/getattribute could have side effects, and we’ve tried to be careful about preserving evaluation order even to this level of detail

Makes sense as well.

In general, I think it would be better to factor out the try…except translation code into functions that can be reused without literally generating and re-parsing fake ast nodes.

I'm not sure what you mean here. Are you just suggesting that we replace the big nested AST literals with easier-to-read/change functions? Or do you think the strategy of doing AST-level writing itself is the wrong way to do this. My experience has been that it's significantly easier to do source-to-source transformations on AST trees than on raw strings

andersk · 2015-11-23T00:38:32Z

I’m explaining our reasons to avoid AST-level rewriting, yes. Perhaps things would be different if symtable worked on AST input—though even without such external considerations, I find it better to refactor an AST-to-AST translation

translate(ast.Foo(args)) = translate(ast.Bar(f(args)))
translate(ast.Bar(args)) = some output

as something more like

translate(ast.Foo(args)) = translateBar(f(args))
translate(ast.Bar(args)) = translateBar(args)
translateBar(args) = some output

Some examples of this pattern already in use are assignment_component and comprehension_code. (Also, we aren’t working on raw strings, we’re working on the T class.)

ssanderson force-pushed the with-block branch from 7285071 to 942b45d Compare November 18, 2015 03:38

ssanderson reviewed Nov 18, 2015
View reviewed changes

andersk mentioned this pull request May 31, 2016

Why not support "with" keyword? #67

Open

csvoss force-pushed the master branch from 8a481b6 to 7f6a35e Compare May 19, 2018 23:44

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ENH: Implement support for with-blocks. #59

ENH: Implement support for with-blocks. #59

ssanderson commented Nov 18, 2015

ssanderson Nov 18, 2015

andersk commented Nov 18, 2015

ssanderson commented Nov 22, 2015

andersk commented Nov 23, 2015

ENH: Implement support for with-blocks. #59

Are you sure you want to change the base?

ENH: Implement support for with-blocks. #59

Conversation

ssanderson commented Nov 18, 2015

ssanderson Nov 18, 2015

Choose a reason for hiding this comment

andersk commented Nov 18, 2015

ssanderson commented Nov 22, 2015

andersk commented Nov 23, 2015