Workweek layers

layers

• ibnc: Create a new compositor layer & set to fixed position: scrolling
• simonsapin: why fixed before absolute? fixed is the same as absolute and then you have some scrolling and layering.
• ibnc: positioned to the viewport and doesn't scroll.
• simonsapin: the layout part is pretty much the same as absolute
• jack: it was the easiest thing
• simonsapin: start with aboslute and then add fixed
• jack: other engines do layerizing. have a div that you render, then figure out how to draw this stuff and try to collapse the layers as much as possible. Have another step that figures out if it has to have its own layer (position:fixed, z-index, etc.). They have things that force other layers to be drawn. Shown in the blink videos w.r.t. animations. Like position:fixed, stuff behind will scroll, but drawn separately. In a perfect world, each piece gets its own texture, but not on mobile. GPU memory is short, so we have to have a limit. This changes frame by frame. Animation may get turned on vs. off. Sounds complicated inside gecko. What we have is just one layer. All in same displaylist. Chop it into tiles and dump it into a texture. Need layers for animation, z-index, position:absolute/position:fixed. Do we know the best way to do this?
• larsberg: where do we make the decision on how many layers?
• jack: It's more how much memory than how many tiles. Can't have everything in its own layer. We do iframes as their own thing. Kinda layer-like.
• simonsapin: in gecko, we don't have a displaylist, we have a tree. One of the types can be opacity, and it contains other displayitems.
• jack: Used to be a list, but is now a tree (because of ClipItem, which sets an overflow:hidden stuff with children). Exactly the same thing for opacity.
• simonsapin: Those are typically where you would have a layer. Anywhere you have a displayitem that contains other displayitems is where you put a layer. Does it make sense to have layers inside layers?
• jack: on the compositor side, we have the scene graph with root layers. there's the root layer, there's the container layer, and I think there's a tiled image layer (GPU, in rust-layers code). In servo, we have the display list w/ render buffers, and then the compositor with its layers and then the rust-layers with their own. It gets a little tricky. What should we be doing? Where do you think layers should be created?
• ibnc: Add a separate displayitem and pass it through. So, we have the one display list right now.
• jack: Right now, there's a displaylist and it draws a bunch of stuff. If you have an iframe, it has a separate displaylist. The compositor will composite those on top of each other (each with their own rust-layers GPU layer).
• ibnc: where is that?
• jack: iframes have their own pipeline. Those are the only real GPU layers in there. There are all these other LayerBuffers, etc. in place, but we're not sure if that all makes sense now. This is the LayerBuffers and LayerBufferSets. The middle part we should treat as a blank canvas.
• larsberg: should iframes always have a separate layer?
• jack: it does provide parallelism. We also have a problem with a single layer that covers the whole page, as it takes us forever to lay it all out. Gecko does prefetching (which we will also need to do) with the AsyncPanZoomController.
• jack: So, should there be multiple displaylists? or should we host it all together? Do we need layers support for anything other than
• jdm: Canvas. maybe webgl?
• jack: I assume we just give off a raw layer to those systems?
• simonsapin: Do we even need layers?
• jack: Only to avoid re-rendering stuff. Lots of the CSS animations are just matrix transforms, so if it's in a layer, you just shift the layer and the GPU quickly repaints.
• simonsapin: So, trade-off is between having enough layers to paint quickly vs. running out of GPU memory.
• jack: Ideal would be everything in its own tiny layer. And uploading to the GPU, etc. But lots of the animations take a sprite or some little piece of content and then blit it. Never uploading saves a huge amount of work.
• larsberg: where would position:fixed go in the tree? Has to be on top...
• jack: Seems like we should know where it goes
• simonsapin: to do stacking correctly, you have to pull position:fixed and position:absolute at the level of the containing block. fixed is at viewport. absolute is in its containing block.
• jack: makes sense that absolute should be in the
• simonsapin: Just a special case. absolute's containing block is the closest containing block. can do the same for fixed, since it's always the viewport.
• jack: so we make them children of the flow that is the containing block.
• larsberg: so how do we ensure that the fixed gets painted on top?
• jack: we paint siblings in linear order, so it should just work
• larsberg: sort for z-index?
• simonsapin: yeah, that's how weazyprint works.
• jack: questions for patrick: 1) how does the flow tree need to change? Should containing blocks be their own flows? 2) for stacking contexts, do we need to create new flows?
• simonsapin: What is a flow in servo?
• jack: Good question to ask :-) It basically just exists to let us perform parallel layouts. two kinds: blockflow and inlineflow. those flows have boxes. the flow for a blockflow draws borders, etc. The other stuff inside of it will be drawn by whatever the children are. An inline flow has a whole lot of boxes that represent the inline children's content. The RenderBoxes can change, but the flow tree is static. But, the boxes may change. We start at the bottom to do bubble-widths and start dividing. We don't do stuff like line-breaking until later, at which case it's just in terms of the RenderBoxes (not the flows). So layout doesn't affect the flow tree; just the boxes. We collaborated with gecko on this early in the servo days. Having part of it be static makes lots of things cleaner.
• simonsapin: flow+boxes = frames in gecko?
• jack: Yes. We just have it in two parts. So, when we construct the flow tree, you mainly just look at position and display attributes. Small subset of CSS properties. We may need to look at absolute, z-index, etc.
• simonsapin: for stacking contexts?
• jack: Should the flow tree know what's a stacking context or not? The same way it knows if it's a block or inline or a table, etc. Is that useful? Another is where in the flow tree should these things be? Maybe we should move position:fixed to the top and absolutes to their containing block. But the flow tree needs to have some idea of containing block. Right now, layout doesn't know anything about that. Already complicated because of {ib}-splits. Getting all this right is tricky (and has been a buggy piece of code in the past).
• ibnc: For fixed & absolute, it should be pretty easy. Just need to know where it should be.
• jack: Agreed, just have a flag isContainingBlock, walk up the tree, but breaks parallelism?
• larsberg: we already walk bottom-up!
• simonsapin: in Gecko, we leave a placeholder for where the element would have been in the tree if it was not positioned. If you only specify top, you have to determine the width based on where it would have been in the tree. called "static position"
• ibnc: Yes, it just works for right now because we leave it in-place. We might need that placeholder element.
• jack: Reflow happens right now over the DOM tree. First layout does the CSS cascade (selector matching & cascade) and then flow tree construction ( in parallel). After flow tree construction, we ignore the DOM and rendering is just flow tree & render boxes. Create the display list and render it.
• simonsapin: The static position means that for one DOM element we need two things in the flow tree. The placeholder for static position and the actual element.
• jack: OK. So it's like a "ghost" node in the flow tree. With pointers all over? Parallelism nightmare, maybe. We do script and layout separately and we're hoping to eventually get where we can give up the DOM nodes so script can proceed once we finish. But having to look at the DOM node would break it.
• simonsapin: What about for floats?
• jack: In-order traversal, right.
• simonsapin: similar constraints. Probably need a placeholder as well.
• jack: Your siblings control where you get positioned. So you have to walk the tree in-order. We wanted to lift these elements up so we know where they are.
• simonsapin: For floats edge cases you may need this anyway. e.g., float in the middle of an inline in a paragraph - where you put it depends on where you would have put it if it was not a float
• jack: Can have dynamic problems. e.g., float causes a line break, but it fits,
• simonsapin: float in a line in a paragraph, you have to try to fit it on the left of the line. If there isn't enough space, then it goes on the next line, but then you have text that needs to go on the previous line.
• jack: Hopefully won't have this problem with absolute or fixed. floats already break parallelism.
• jack: Kinda need this info. How bad are floats in real pages? We may need to start clearing floats or something to avoid having them affect the subtree.
• simonsapin: clear only makes a difference if you have other floats before.
• jack: Yeah, if the float is cleared, you can render the subtree in parallel because no other floats can affect it.
• simonsapin: I don't think we can speculatively truncate / clear floats without breaking things...
• jack: Imagine the 100 px sidebar and it's floated - we can probably clear that...
• simonsapin: I'm skeptical.
• jdm: We can get the data (via telementry). John jensen can spider across websites to get uses of things, etc.
• jack: Was hoping to do common crawl. Does it have the data or just the index? I'd like to know stuff like how many floats are clearable, etc.
• pcwalton: Opera did this kind of survey for changing table close to p tag and found out it broke like half the web. I'm working on MutexArc.
• pcwalton: WebKit's approach to this (displaylists) may be better. I talked with roc on this. in Gecko, we build one and split it up with the FrameLayerBuilder. Really complex. Webkit's is based on the DOM. You have a layerizing decision based on the style (maybe active/inactive as well). translate3d gets a layer, position:fixed gets one, now have active/inactive. Nodes can start a layer hierarchy.
• simonsapin: what does that mean active/inactive?
• pcwalton: Has it moved recently? If it's animating, it will be promoted to an active layer, and demote after it stops. Early safari always created a layer.
• simonsapin: don't know if that helps us. To avoid people doing hacks (like translate z 0), there's a proposal for a new CSS property. willChange. Gives hints about what to put in layers or not.
• pcwalton: Webkit has a separate displaylist for every layer. They layer the displaylist on top by caching drawing commands. Render into each layer. Each subtree has a layer you render into. I don't know how stacking/z-index stuff works in webkit with layers. I think that's simpler because you don't need the FrameLayerBuilder, which is pretty complicated and hard to maintain. Then you have a bunch of layers and a bunch of displaylists. You have to keep track of the layer hierarchy that you're building as you build the display list.
• larsberg: placeholder elements? We need them for positioned elements and floats. They have a "static position" based on where they would have laid out.
• pcwalton: worried about messing up parallelism. We should just try it with the placeholder elements and see what happens.
• jack: should we hang position:fixed off of the root?
• pcwalton: yes. flows are designed to hang off of their containing block.
• jack: does this apply for fixed?
• ibnc: Yes, if you don't specify left or right.
• jack: want a separate place for those flows to sit, but then need a placeholder element to get the static properties... which way should the pointers go? How should we compute the width, etc.?
• pcwalton: Yeah, extra data dependency there.
• jack: first pass for anything that's not positioned....
• pcwalton: Not good. Right thing is not to put position:fixed like that. I really don't want extra passes. put position:fixed not parented to containing block but...
• jack: last pass is bottom-up, right? could collect position:absolute & fixed on the way up, since they're "done"
• pcwalton: need the containing block during assign-widths
• jack: fine because
• simonsapin: assign widths is top-down? if you have teh containing block for the content, then you can keep track of it.
• pcwalton: Yeah, webkit does that. there's one right.
• simonsapin: three. one for normal, one for absolute, and one for fixed.
• pcwalton: Don't know of any other way to do it... Is this going to make anything weird in terms of flow construction. Now you have lost the invariant that every flow's parent is its containing block.
• jack: not true now unless they're positioned non-statically. block flow's kid's containing block is the root, not itself.
• simonsapin: why is it useful to have the flow's parent be its containing block?
• pcwalton: don't have to keep track of it.
• jack: that was the idea, but we don't have the interesting containing blocks yet. the containing block is the nearest ancestor with the position that is not static.
• simonsapin: that's for position:absolute. for non-positioned, it's your nearest parent block container. in fixed, it's the viewport. root element's containing block is alos the viewport.
• pcwalton: yeah, I think we can't keep that invariant in place. so, leave fixed in the dom tree. Then lift it up during layer construction.
• jack: can we not lift it right now? and say that displaylist construction will be slower? just leave it where it is?
• pcwalton: should be fine. just construct displaylist for all layers simultaneously. so you have a list of layers. root has a displaylist. add elements. eventually, you see position:fixed and create a new layer. still just have one pass. at the end, you have two layers and two displaylists. so kids can create new layers and new displaylists.
• jack: so not any slower, just appending
• pcwalton: need to be able to insert for z-index, too. not sure if parallelism for displaylist construction is top-down or bottom-up. nice thing about bottom-up is just one insertion
• jack: but in the wrong order: what about borders of the root element going last?
• pcwalton: merge can handle putting them in the wrong order. displaylist bottom-up could work the same way as flow tree construction so they bubble up the tree and at the endyou just pull it off the root as each thing migrated. the problem is that displaylist has to compute final positions, which are dependent upon your parents. layout positions are relative to the parents.
• jack: maybe could introduce TransformingItems... already do that for some things. Assume everything is relative to a containing block. Display items have a transform for all their kids. But, we don't have to do this lifting thing. When we just build the display lists.
• pcwalton: Kids don't worry about where they go; the parent decides to create a layer (stacking context). So you just bubble up the items of your kids and add your own items into your own. when you get to a layer, you put those items into a layer.
• larsberg: going to have nested layers
• pcwalton: yes. fixed doesn't matter, absolute might need a layer
• simonsapin: position:absolute doesn't create a layer unless it's being animated
• pcwalton: top-down traversal doesn't require transforms, etc.
• jack: let's try position:fixed where it is. try to build the displaylist top-down, and got with it.
• pcwalton: it's in-order right now
• jack: Why? Can't we just do top-down?
• pcwalton: yes. will get a linearized list of layers. top-down, when you get to an item that creates a layer, you establish that and then probably have to be passing down an index for what layer they should be writing their displaylists into.
• jack: so pass in the position:fixed layer to everybody?
• pcwalton: kids go to their parents to find it