Any guidance on a reversed virtual list with dynamic elements?

[cmseguin]

We have been using react-virtual for a few months, and love the fact that the library is headless and allow us to do things our way... However, we are trying to build a chat-like virtual list that starts at the bottom, we then prepend elements to the list when we reach the top. Nothing too fancy except that the messages have fairly variable sizes.

The problems started to show up when we started experimenting with bigger messages... It seems that if we used a list that would not start from the bottom we would not have nearly as many issues... Mostly because the height of the list gets calculated and as you usually scroll down, the list height expands but you are left at the same scroll position (so nothing more than a little flicker in the scrollbar). However when the list is inverted and the elements above your overscan are rendered, the height of the elements pushes the elements that are in view downward (happens only when the estimate size is not 100% correct).

Furthermore, we have a functionality to scroll to the bottom of the list which we achieve by scrolling to a big offset. The reason for that is due to the padding at the bottom, scrollToIndex would not work for us. However since our estimate function is not 100% accurate the scroll happens as follow:

1. scroll to the bottom
2. render happens and list height is increased (no longer at bottom)

It seems that this is something inevitable, and that is why we have a double scroll in the scrollToIndex.

So I was wondering if anyone has ever done a reversed list with dynamic elements and could give some guidance, or create an example.

[kidroca]

The only thing that worked for us so far is a double scaleY(-1) hack borrowed from react-native virtual list inversion

As you've noticed both prepending or appending an item would change scroll height, but prepending pushes visible content while appending does not

The scaleY hack would make everything work in reverse

• your first item (index 0) will be rendered on the bottom
• scrollToIndex(0) would scroll to the bottom
• pushing more items would add them to the top, but now that's the natural direction in which the list expands

It's relatively easy to try it out, but it has some potential drawbacks

• accessibility - screen readers would disregard css and would read the non inverted list - it would be as if they read the list from bottom to top
• text selection - if you select some content and copy it, it will be pasted in reverse order

So here it is:

1. Order you items (array) so that index: 0 is the message you expect to be on the bottom
2. Add transform: scaleY(-1) to the parentRef element styles (the element that is scrollable)
3. Add transform: translateY(${({ translateY }) => translateY}px) scaleY(-1); to each row item

Pushing more items would add them to the top but would not cause scroll/content issues

Since your case is a chat - I'm not sure whether there won't be the same problem when new messages arrive and they need to be added to the bottom. It might be possible to just scroll to index 0 (bottom) when new messages arrive (if you were already on the bottom)
Otherwise you'll need to readjust scroll position by the change in totalSize of the list - if the (inverted) list grew by 200px when new message are received then you need to scroll to current scrollTop - 200

For our case the scaleY hack works
We have a virtual list that is dynamically filled with days for 3 years (like 1000 days)
The days/weeks content is only loaded when they are scrolled into view - initially a placeholder element is rendered
We can jump to the middle and skip loading some days (e.g. jump from now to 2 years ago)
As we scroll up/down the days are loaded and they dynamically expand the rows - this used to mess our scroll position
So far nothing but the scaleY had made scrolling reliable without content being pushed unexpectedly by days/rows materializing and changing height

BTW you also have to invert the mouse wheel scroll like:

    useEffect(() => {
      const el = parentRef.current;

      const invertedWheelScroll = (event) => {
        el.scrollTop -= event.deltaY;
        event.preventDefault();
      };

      el.addEventListener('wheel', invertedWheelScroll, false);

      return () => el.removeEventListener('wheel', invertedWheelScroll);
    }, []);

[lexahall]

Have you had any success with keyboard navigation with this method? I tried this and tab works in the reverse order. Arrow keys work in the correct order until you hit the edge of the viewable area, and then it will try to scroll more item into view but scrolls in the opposite direcition. Have you hit either of these issues?

[jpribyl]

Man, this gets hairy- you gotta

1. render the elements invisibly to the real dom via portal
2. use those invisible elements to extract + set the height on you virtualized elements
3. (assuming you want overscan) maintain a list of elements that are currently visible separate from react-virtual (maybe something like react-intersection-observer)
4. build a map between the key / id field of your visible elements and their virtualized indexes so that you can get the figure out what the index is when you prepend new items to your virtual component. Alternatively you might be able to map by the number of items you prepend (OLD_INDEX + PAGE_SIZE or something)
5. probably more that I'm forgetting?

EDIT/UPDATE I think you can actually do 4. with the rangeExtractor which is a good bit simpler + cleaner

I have a loosely passable POC but it's not in a good state to share at the moment. happy to discuss + send snippets if there is still interest.

This blog post was helpful for me
https://medium.com/trabe/measuring-non-rendered-elements-in-react-with-portals-c5b7c51aec25

[jpribyl]

@agamrp let me know if you manage to solve the scrollToOffset rather than scrollToIndex issue so that we can hit exactly the right pixel rather than a small content jump!!

I've flung myself at it a couple times and gotten close but it's super tricky because there is no "wait for scroll to complete" in JS.. so the issue that I have seen is that scroll height jumps around too much with that approach.

One option might be to use both? IE use index to get a rough position and avoid a big jump and then later use offset to fine tune? Haven't really tried that approach yet.

[agamrp]

@jpribyl Have a favor to ask if you have time. Have you tested your solution on iOS and Safari yet?

I've played around with other virtualized libraries that have prepending built in but run into weird behavior where while scrolling up the list will sometimes jump to the top of the new data instead of keeping scroll position, along with some flickering. If you scroll slowly up then it keeps scroll position better. I was hoping you may have figured out a way to solve this? Or a way to cancel scroll events while data is being loaded?

Attaching video to help clarify:

Blinking.And.Jumping.720.mov

[jpribyl]

Hmm, I haven't really played around with it- I do most of my dev work on firefox + chrome but I'll take a look next time I have a mac in front of me.

[agamrp]

Got it thanks, yeah appreciate it when you get a chance and also if you get hold of iOS device to try it on. I'll let you know what I find out on my end

[jpribyl]

@agamrp just FYI I have played around with it a bit on safari

while scrolling up the list will sometimes jump to the top of the new data instead of keeping scroll position

I have not seen this, thankfully

along with some flickering

There is a small amount of occasional flickering. But tbh it's pretty rare and not very noticeable so I don't plan to really look into it more at this point. I think it could be mitigated with better loading strategies.

The largest issue I am still grappling with is images. They aren't available when the initial measurement takes place which means you must either

• know their size before loading them (IE store images with their size) or coerce them into a specific height
• come up with some way to scroll as they load (maybe an onload handler)
• prevent render until they're loaded
• lazy load them
• something else?

Haven't had a ton of time to devote to it but all in all it's working pretty darn well

[piecyk]

I think we can go with this https://codesandbox.io/s/beautiful-meninsky-fr6csu?file=/pages/index.js

The trick is to update scroll offset in same time when new items are prepend, and reverse indexes.

[piecyk]

Ok, the issues was that react in strict mode will call useMemo functions twice instead of once, that basic was breaking syncing scroll. Updated example to check if count changes against virtualizer value.

[abdullaev388]

Could you provide same example but with useVirtualizer (not window)

[piecyk]

Could you provide same example but with useVirtualizer (not window)

@abdullaev388 the idea is the same, only difference is that you need to pass scroll element https://codesandbox.io/p/sandbox/frosty-morse-scvryz

[velidan]

Thank you for your idea Damian. Could you, please, help me to understand what is the purpose of this code?

  const [paddingTop, paddingBottom] =
    items.length > 0
      ? [
          Math.max(0, items[0].start - virtualizer.options.scrollMargin),
          Math.max(0, virtualizer.getTotalSize() - items[items.length - 1].end),
        ]
      : [0, 0];

I debugged it at paddingTop is only one time become to 480px and then it become 0 and never won't be positive number again. I'm just really curious about it. Without it the scroller is incorrectly positioning itself hovewer still can't understand how this prop works

@piecyk

[zirlman]

Here's how the code works:

The items array contains information about the virtual items that are currently visible in the container. Each virtual item represents a paragraph, and contains information about its size and position.

The paddingTop variable is calculated by subtracting the start position of the first virtual item from the scrollMargin option of the virtualizer. The scrollMargin option is used to add extra padding to the top and bottom of the container, to ensure that the first and last virtual items are not cut off when the user scrolls through the container.

The paddingBottom variable is calculated by subtracting the end position of the last virtual item from the total size of the virtualizer. This ensures that the last virtual item is not cut off when the user scrolls to the bottom of the container.

If there are no virtual items in the container, the paddingTop and paddingBottom variables are set to 0.

The paddingTop and paddingBottom variables are then used to set the paddingTop and paddingBottom styles of the chat bubble container, using the style prop.

Overall, this code block is used to calculate the padding for the top and bottom of the container, to ensure that the first and last paragraphs are not cut off when the user scrolls through the container.

Source: Copilot

[novykh]

Hello, I ended up in this issue a couple of days ago searching for an efficient way to have two-direction infinite loader+scrolling with virtual and react-table. Your answers above helped a lot and I ended up with an approach that I believe works well enough. (Still wip - especially from the perf aspect)

I work at netdata and you can check the end result (without signup or anything) here.

480p.mov

I used both react-table and react-virtual.
For the table, I created a flex implementation with a big component tree for every row, which made things more difficult performance wise, but managed to fix it through caching (yeah I know @tannerlinsley you love the table re-rendering freely, but it wasn't possible to allow it in this use case, so please bear with me).

Now for the problem at hand, bi-directional infinite scrolling, I used the usual implementation for infinite loading with the difference of adding a second useEffect to check if the virtualRows are on top of the table, and if there are more items to fetch to issue the request for them. This of course created the issue to the virtualizer, since changing (prepending) the data was instantly changing the top virtualItem to the new one - which in turn was requesting again for more items, resulting in an infinite loop.

The gist of what I used is this:

   const retryRef = useRef(0)
   
   useLayoutEffect(() => {
      // Check if data length changed and we need to check if the response that changed the data was of direction forward (going up that is)
      if (prevDataLengthRef.current !== data.length && latestDirection === "forward") {

        // Find the current scroll offset
        const offset = virtualRef.current.scrollOffset
        let dummyOffset = 0
        let scrolledItem = virtualRef.current.getMeasurements().find(measurement => {
          dummyOffset = dummyOffset + measurement.size
          return dummyOffset > offset
        })

        // If we don't find any, then we get the first element
        if (!scrolledItem) {
          scrolledItem = virtualRef.current.getMeasurements()[0]
        }

        // OK if we don't find an element again then just leave
        if (!scrolledItem) return

        // We keep the scrolled data item (we need it's identifier)
        topElementRef.current = data[data.length - prevDataLengthRef.current + scrolledItem.index]
      }

      const tryScrollToOffset = () => {
        if (topElementRef.current && retryRef.current < 5) {
          retryRef.current = retryRef.current + 1
          prevDataLengthRef.current = data.length
          const anchor = topElementRef.current.id // or an unique identifier that works on your data

          // Try to find the new measurement (we prepended rows, so the previous measurement is not at the same index as before)
          let item = virtualRef.current
            .getMeasurements()
            .find(measurement => measurement.key === anchor) // Supplied a `getItemKey` to the virtualizer to get the data id
          
          // If none found, fallback to blind index searching based on data in our state
          if (!item) {
            const index = data.findIndex(rowData => rowData.id == anchor)
            item = virtualRef.current.getMeasurements()[index]
          }

          // If none found again, then abort and stay on top of the table
          if (!item) {
            retryRef.current = 5 // break
            topElementRef.current = null
            return
          }

          // Try to scroll to offset
          virtualRef.current.scrollToOffset(item.start - topElementRef.current.headerSize)
          
          // Even if you do scroll, retry to be certain - Needed only on dynamic measurement lists
          setTimeout(tryScrollToOffset)
          return
        } else {
          retryRef.current = 5
          topElementRef.current = null
        }
      }
      tryScrollToOffset()

      prevDataLengthRef.current = data.length
    }, [data])

So in our use case:

• The user clicks on the chart to scroll to that timestamp
• After we scroll there, we are in the middle of the table, so we need to fetch more data to prepend
• After prepended data comes in, the useEffect above get's triggered and scrolls user back to the first step's first index.

The same happens when the user keeps scrolling up and fetches more and more data. All of the above are happening with just a small flickering the minute before scrollToOffset gets triggered (especially when there are big differences in the measurements between every line.

Hope this helps anyone who wants to try something similar.

[agnauck]

is there a way to implement this without the flickering?

[piecyk]

@agnauck yep, it boils down to sync scroll in correct time

[agnauck]

@piecyk can you elaborate a little more on that? or point to a small demo?

[avdb13]

Did somebody have success reversing the arrow and home/end keys already?