Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add the ability to "retain" a dismissed screen, to support iOS Picture In Picture overlays. #11097

Open
wants to merge 10 commits into
base: 6.x
Choose a base branch
from

Conversation

jaredly
Copy link

@jaredly jaredly commented Dec 13, 2022

(Feature request fwiw) (corresponding react-native-screens PR)

Thanks for making this wonderful library! I've really enjoyed using it. I'm transitioning a large app from custom native navigation code (written before we adopted react-native) to react-navigation, and the only thing I'm having trouble with is iOS Picture-In-Picture.

If you're unfamiliar, Picture-in-Picture (PIP) refers to popping a video out into an "overlay" that persists as you navigate around the app, until you either close it or "restore" it, which results in the originating screen being pushed back onto the stack, and the video sliding back into its original location.

Here's a little video of that in action, for context:

pip.mp4

There are two things needed to support PIP well:

  1. the ability to dismiss a screen without letting the resources be garbage collected. (when the originating ViewController is garbage collected, the PIP window automatically closes). Additionally, the screen must not be unmounted on the react side of things, so that events will still be handled properly (critically, the onRestoreUserInterfaceForPictureInPictureStop callback for react-native-video).
  2. the ability to bring back the screen that the PIP overlay originated from, in response to the user tapping on the relevant icon in the overlay.

I've made a pull-request to react-native-screens adding a hidden prop to the Screen component, which allows a screen to stay in the react tree (so the resources are retained & events still propagate), but be ignored by the NavigationStack.

On the react-navigation side, we need an additional flag on a given route, to allow a screen to be (a) marked for retention (e.g. if a user dismisses this screen, hang on to it but mark it "hidden"), (b) restored from the hidden state, and (c) dropped from retention (which, if the screen is hidden, means it is removed, and if it is still in the visible stack, it just has the flag cleared).

Test code and steps to reproduce

Here's a minimal repo illustrating the problem behavior: https://github.com/jaredly/rnsdemo
And a video:
https://user-images.githubusercontent.com/112170/207732805-7750c1e8-7276-4f51-9244-f86ec312092b.mp4

@github-actions
Copy link

Hey jaredly! Thanks for opening your first pull request in this repo. If you haven't already, make sure to read our contribution guidelines.

@netlify
Copy link

netlify bot commented Dec 13, 2022

Deploy Preview for react-navigation-example ready!

Name Link
🔨 Latest commit 7ace119
🔍 Latest deploy log https://app.netlify.com/sites/react-navigation-example/deploys/63c86426451c9000089c246b
😎 Deploy Preview https://deploy-preview-11097--react-navigation-example.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

@@ -61,7 +61,7 @@
"react": "*",
"react-native": "*",
"react-native-safe-area-context": ">= 3.0.0",
"react-native-screens": ">= 3.0.0"
"react-native-screens": ">= 3.19.0"
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I set this to a fictitious future version, because it relies on a change to react-native-screens

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jaredly the peer dependency shouldn't be bumped, the package is still compatible with react native screens 3, only the new feature needs a newer version so that can be documented separately

Copy link
Member

@satya164 satya164 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey! Thanks for the PR.

I'm not sure about the approach, this is highly specific to the native stack and native features so I don't think it belongs in the router.

Native Stack should expose this with a different API such as a custom hook and store the additional data regarding screens that need to be retained in its internal state.

It'd also be useful to see some code snippets of how this may be used to understand the APIs needed.

@@ -61,7 +61,7 @@
"react": "*",
"react-native": "*",
"react-native-safe-area-context": ">= 3.0.0",
"react-native-screens": ">= 3.0.0"
"react-native-screens": ">= 3.19.0"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jaredly the peer dependency shouldn't be bumped, the package is still compatible with react native screens 3, only the new feature needs a newer version so that can be documented separately

@jaredly
Copy link
Author

jaredly commented Dec 16, 2022

@satya164 cool, thanks for the feedback! I'll try the custom hook direction.

@jaredly
Copy link
Author

jaredly commented Jan 17, 2023

@satya164 So in order to implement screen restoration, I will need to be able to specify the key of the screen that I'm adding "back on" to the stack, such that it doesn't get a nanoid() key. Would you rather that be a separate action, or an argument added to the PUSH action? (or to the .payload of the PUSH action)

@satya164
Copy link
Member

Would you rather that be a separate action, or an argument added to the PUSH action? (or to the .payload of the PUSH action)

Since this is very specific to native-stack, there shouldn't be any changes to routers or the core package. Ideally, native stack would expose a method from its hook that you can call which would internally use reset action to add the route back to the state - so your key will be preserved in this case.

@jaredly
Copy link
Author

jaredly commented Jan 17, 2023

oh cool, yeah that will work great

/**
* Contexts
*/
export { RetainContext } from './views/RetainContext';
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let me know if you want this to go into a /contexts/ directory, or somewhere else

return (
<ScreenStack style={styles.container}>
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

best viewed with whitespace ignored

Comment on lines +433 to +463
if (!retainedKeys.includes(route.key)) {
setNextDismissedKey(route.key);
}
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this warning fires spuriously when restoring a previously dismissed screen

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jaredly yea this one has some issue, need to address it later

@jaredly
Copy link
Author

jaredly commented Jan 18, 2023

@satya164 ok I've got it working 😅 this repo https://github.com/jaredly/rnsdemo with the native-stack dependency pointed at a local checkout of this branch, has Picture in Picture working!

pip.mp4

@satya164
Copy link
Member

Thanks @jaredly for updating the MR! I'll try to review it when I get some free time

@satya164 satya164 changed the base branch from main to 6.x February 17, 2023 17:38
@jaredly
Copy link
Author

jaredly commented Apr 18, 2023

bump? 🙏

@github-actions
Copy link

Hey @jaredly! Thanks for opening your first pull request in this repo. If you haven't already, make sure to read our contribution guidelines.

@codecov-commenter
Copy link

codecov-commenter commented Sep 4, 2023

Codecov Report

Attention: 46 lines in your changes are missing coverage. Please review.

Comparison is base (46954ce) 74.17% compared to head (55e9f21) 73.82%.

Additional details and impacted files
@@            Coverage Diff             @@
##              6.x   #11097      +/-   ##
==========================================
- Coverage   74.17%   73.82%   -0.35%     
==========================================
  Files         178      179       +1     
  Lines        5618     5670      +52     
  Branches     2212     2216       +4     
==========================================
+ Hits         4167     4186      +19     
- Misses       1402     1435      +33     
  Partials       49       49              
Files Coverage Δ
.../native-stack/src/views/NativeStackView.native.tsx 78.99% <50.00%> (+0.04%) ⬆️
packages/native-stack/src/views/RetainContext.tsx 31.91% <31.91%> (ø)

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@jaredly
Copy link
Author

jaredly commented Oct 17, 2023

@satya164 I'd really like to get this merged so I can stop depending on a fork -- is there anything I can do to help this along?

@satya164 satya164 requested a review from osdnk October 18, 2023 10:13
@satya164
Copy link
Member

satya164 commented Oct 18, 2023

@jaredly I'll try to review it this week. would you be able to write a short usage example with the code so it's easier for me to see the API?

@jaredly
Copy link
Author

jaredly commented Oct 18, 2023

Fantastic, thanks! Here's a little demo repo I made to show how retain works together with PictureInPicture https://github.com/jaredly/rnsdemo/blob/main/App.js
We've been using this in production at Khan Academy for 6 months now, and it's been working great.

Copy link
Member

@osdnk osdnk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jaredly
Thanks for your hard work.
Few things:

  • Please, specify what is the public API here. We prefer not to expose the contexts directly.
  • you should warn/throw if one wants to use that on Android/web
  • Avoid using values from refs in the state update. In general, I cannot see why latestValue is needed here. If it's very needed, please elaborate
  • Avoid shadowing of variables
  • Add some examples.

@@ -373,74 +377,86 @@ function NativeStackViewInner({ state, navigation, descriptors }: Props) {

useInvalidPreventRemoveError(descriptors);

const { retainContext, hiddenRoutes, hiddenDescriptors, retainedKeys } =
useRetainContext(state, navigation, descriptors);
descriptors = { ...hiddenDescriptors, ...descriptors };
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd avoid overriding props, as ot may be confusing for someone reading the code. I guess it's a mental model that we don't mutate values as long as possible.

How about mergedDescriptors and mergedRoutes.

It's a bit confusing (at least, I can imagine, it can be) that you use create an array and object without memorizing. Of course, it makes sense here as we don't pass them as props, but I'd add a comment, e.g., "no need to memorize as we don't pass them directly as props"

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no need to memorize as we don't pass them directly as props

imo the comment isn't necessary since memorizing depends on other factors and not about props. it'd be confusing for me to see such a comment.

);
})}
</ScreenStack>
<RetainContext.Provider value={retainContext}>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about ScreensRetentionContext? I'd prefer to use nouns if possible.

<RetainContext.Provider value={retainContext}>
<ScreenStack style={styles.container}>
{routes.map((route, offsetIndex) => {
const index = offsetIndex - hiddenRoutes.length;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's confusing. I don't think we have any offset here.

Suggested change
const index = offsetIndex - hiddenRoutes.length;
const isHidden = index < hiddenRoutes.length;

and then hidden={isHidden}

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

const isHidden = hiddenRouteKeys.includes(route.key) maybe more descriptive.

Comment on lines +21 to +29
* This context allows you to retain screens in memory, even after
* they have been popped off the stack. This is important for
* supporting native iOS "Picture in Picture" mode, as the
* originating UIViewController must be retained in memory for the
* PiP overlay to continue running.
* Retained screens can also be "restored" to the top of the stack,
* for example when the user taps the button in the PiP overlay
* to return to the app.
*/
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* This context allows you to retain screens in memory, even after
* they have been popped off the stack. This is important for
* supporting native iOS "Picture in Picture" mode, as the
* originating UIViewController must be retained in memory for the
* PiP overlay to continue running.
* Retained screens can also be "restored" to the top of the stack,
* for example when the user taps the button in the PiP overlay
* to return to the app.
*/
* This context allows you to retain screens in memory, even after
* they have been popped off the stack. That is important for
* supporting native iOS "Picture in Picture" mode, as the
* originating UIViewController must be retained in memory for the
* PiP overlay to continue running.
* Retained screens can also be restored to the top of the stack,
* for example, when the user taps the button in the PiP overlay
* to return to the app.
*/

I'm not native so feel free to ignore. I believe there's no need for a quotation as the description is precise enough.

* to return to the app.
*/
export const RetainContext = React.createContext<RetainContextT>({
retain() {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd do something even stronger.

Suggested change
retain() {
get retain() {

This will throw an error on access, without waiting for invoking the function.

const { [key]: _, ...rest } = screens;
return rest;
});
return latestValues.current.retainedScenes[key] != null;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you should not rely on the ref here as concurrent mode can make it flaky. Why can't you rely on the previous state?

// for which we still want to keep the memory allocated.
const hiddenRoutes: Route<any>[] = [];
const hiddenDescriptors: NativeStackDescriptorMap = {};
Object.keys(retainedScenes).forEach((key) => {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd prefer

const {
  hiddenDescriptors: descriptors,
  hiddenRoutes: routes 
} = Object.values(retainedScreen).reduce(() => { ... }, { routes: [], descriptors: {} })

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code will be much simpler if there was just a hiddenRouteKeys array. Seems like the only place that needs it is when we set hidden={true}, and with such an array, it could just check hidden={hiddenRoutes.includes(route.key)} without changing descriptor and routes related logic.

const hiddenRoutes: Route<any>[] = [];
const hiddenDescriptors: NativeStackDescriptorMap = {};
Object.keys(retainedScenes).forEach((key) => {
if (!routeKeys[key]) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would just check the state.routes here directly, even though it's a higher theoretical complexity. This is not a big deal as this array should be small anyway.

Comment on lines +58 to +61
const latestValues = React.useRef({ state, retainedScenes, descriptors });
React.useEffect(() => {
latestValues.current = { state, retainedScenes, descriptors };
});
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm rather against patters like that. The new state should be the result of props and the previous state as react may batch those updates, we should not rely on what is currently in useRef.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe this is not very needed here and it's possible to refactor the code without that.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume the intention is to memoize the retain, release, restore functions. In that case define those functions separately with useLatestCallback and then they can be safely added to dependency array for useMemo.

With that it's unnecessary to do such a ref.

const retainContext = React.useMemo<RetainContextT>(
() => ({
retain() {
const { state, descriptors } = latestValues.current;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why not state? Also, please avoid shadowing variables as it gets harder to understand the code.

@osdnk
Copy link
Member

osdnk commented Oct 22, 2023

Actually, ignore my comment about adding the example. It's not possible now

Copy link
Member

@satya164 satya164 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think an API like the following would be simpler:

const { retain, release, restore } = useRetain();

// ...

<Video
  // ...
  onPictureInPictureStatusChanged={isActive => {
    if (isActive) {
      retain();
    } else {
      release();
    }
  }}
>

It'd be simpler not to have to manage a key. If user wants to retain multiple things (is that even possible?), they can use the hook multiple times. Though looking at the code, the key seems to be just the route key which will always be the same and can be aquired from useRoute inside the hook itself.

Also why does restoreUserInterfaceForPictureInPictureStopCompleted need a timeout? Is it possible to make restore return a promise so consumer can wait until this function can be called?

@@ -373,74 +377,86 @@ function NativeStackViewInner({ state, navigation, descriptors }: Props) {

useInvalidPreventRemoveError(descriptors);

const { retainContext, hiddenRoutes, hiddenDescriptors, retainedKeys } =
useRetainContext(state, navigation, descriptors);
descriptors = { ...hiddenDescriptors, ...descriptors };
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no need to memorize as we don't pass them directly as props

imo the comment isn't necessary since memorizing depends on other factors and not about props. it'd be confusing for me to see such a comment.

<RetainContext.Provider value={retainContext}>
<ScreenStack style={styles.container}>
{routes.map((route, offsetIndex) => {
const index = offsetIndex - hiddenRoutes.length;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

const isHidden = hiddenRouteKeys.includes(route.key) maybe more descriptive.

Comment on lines +58 to +61
const latestValues = React.useRef({ state, retainedScenes, descriptors });
React.useEffect(() => {
latestValues.current = { state, retainedScenes, descriptors };
});
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume the intention is to memoize the retain, release, restore functions. In that case define those functions separately with useLatestCallback and then they can be safely added to dependency array for useMemo.

With that it's unnecessary to do such a ref.

() => ({
retain() {
const { state, descriptors } = latestValues.current;
const route = state.routes[state.routes.length - 1];
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of assuming the last screen, this hook should use the useRoute hook to get the correct route object

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As currently written, the retain function isn't a hook. I'll look into changing the API to a useRetain and see whether that can meet all our needs

Comment on lines +70 to +73
[route.key]: {
route,
descriptor: descriptors[route.key],
},
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should only store the route key. Duplicating the route and descriptor is problematic as those could change after storing here making them out of sync. They should always be read from state.routes and descriptors.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm it can't be read from state.routes if the screen has been popped off the stack; that's why I'm storing it here. In the case where the route hasn't been popped, I use the one from state.routes.

Comment on lines +100 to +107
const index = state.routes.findIndex((r) => r.key === key);
const routes =
index !== -1 ? state.routes.slice(0, index) : state.routes.slice();
routes.push(route);
navigation.dispatch({
type: 'RESET',
payload: { ...state, index: routes.length - 1, routes },
});
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't refer to actions by string. Use the CommonActions.reset action creator instead. In this case it should be possible to just use navigation.reset.

Also, I assume it's moving the screen to the end. Is this necessary/desirable? If the screen wasn't the last one, seems strange to change the order after pip restore. I'd remove this, consumers can do this themselves if they want to.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the screen wasn't the last one, seems strange to change the order after pip restore.

I thought that in order for a screen to be visible on a screen stack, it needed to be the last one. Is that not the case? restore() is supposed to bring the screen back to visibility; either adding it to the end if it had been popped off, or popping off anything on top of it if the screen still exists in the stack.


return true;
},
supported: true,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This boolean seems unnecessary if it's always true.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's true here, but the default value for the RetainContext has supported: false

// for which we still want to keep the memory allocated.
const hiddenRoutes: Route<any>[] = [];
const hiddenDescriptors: NativeStackDescriptorMap = {};
Object.keys(retainedScenes).forEach((key) => {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code will be much simpler if there was just a hiddenRouteKeys array. Seems like the only place that needs it is when we set hidden={true}, and with such an array, it could just check hidden={hiddenRoutes.includes(route.key)} without changing descriptor and routes related logic.

This allows the native stack router to retain some viewControllers in memory but out of sight, ready to be restored at a later time.
This is necessary for a good PictureInPicture experience.
@satya164
Copy link
Member

FYI we have added retain support to regular stack navigator via navigation.retain and would like to use the same method for native stack: #11765

@jaredly
Copy link
Author

jaredly commented Jan 18, 2024

@satya164 oh interesting! Unfortunately it looks like that API doesn't align quite well enough:

  /**
   * Removes a screen from the active routes, at the same time
   * retaining the screen in the preloaded screens list,
   * so it is not getting detached.
   */
  retain(): void;

In order for picture-in-picture to work as users expect, we need to be able to "mark a screen as retained" without also removing it from the active routes, such that when it is removed from the active routes (e.g. by a POP or any number of other events) it does not get detached.

@jaredly
Copy link
Author

jaredly commented Jan 18, 2024

Responding to your questions (thanks for giving it a thorough review!)

It'd be simpler not to have to manage a key.

I realize now that my demo repository doesn't make use of the keys, but a full application (such as ours, the Khan Academy app) that needs to support picture-in-picture does need them to function correctly. Specifically, when viewing another screen with a video on it, the PIP overlay should be automatically dismissed, but when returning to the screen that originated the PIP overlay, it should not be automatically dismissed. In order to implement that behavior, we need to keep track of the screen where the video was retained.

Also why does restoreUserInterfaceForPictureInPictureStopCompleted need a timeout? Is it possible to make restore return a promise so consumer can wait until this function can be called?

In order for the PIP overlay to be restored fluidly (animating smoothly back to the location of the <Video/> element), the screen needs to have finished animating in. The timeout is a simple way to wait for the animation to complete. I couldn't see a way to have restore know the parameters of the screen animation, but if you know how that would be fantastic.

@satya164
Copy link
Member

we need to be able to "mark a screen as retained" without also removing it from the active routes

I think we can make that change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants