Buggy .makeImageSnapshot() on IOS #1811

sravis02 · 2023-08-27T22:01:38Z

Description

When I want to snapshot my Skia canvas (which is not that small), it gets buggy.
These bugs don't occur always. If you try it multiple times, you will see that sometimes it works fine.

Screenrecording of the bug:
https://www.youtube.com/shorts/mH7VuiT7CuU

What bugs?:

For a short period of time, the elements of my canvas get transformed in a weird way
Outputted image doesn't contain all items of canvas sometimes

Does RNSkia somehow rerender the Canvas, when the method makeImageSnapshot gets executed?
If yes, I think, the snapshot is taken before the rerender process is completely finished.

It makes no sense to me, since the only code that gets triggered on the button click is this one:

const handleShare = async () => {
    const image = canvasRef.current?.makeImageSnapshot();

    if (image) {
      const base64Image = image.encodeToBase64();
      const options = {
        mimeType: 'image/jpeg',
        dialogTitle: 'Share the image',
        UTI: 'public.jpeg',
      };
      const fileUri = FileSystem.documentDirectory + 'test.jpeg'; // todo ts name name
      await FileSystem.writeAsStringAsync(fileUri, base64Image, {
        encoding: FileSystem.EncodingType.Base64,
      });

      Sharing.shareAsync(fileUri, options)
        .then((data: any) => {
          console.log('SHARED');
        })
        .catch((err: any) => {
          console.log(JSON.stringify(err));
        });
    }
  };

Version

0.1.197 in repro but also appears in 0.1.202

Steps to reproduce

How to reproduce:

npm install

On Mac simulator
npx expo run:ios

On Iphone with EAS
eas build --profile development --platform ios
npx expo start --dev-client

In the App

1) Select background image from gallery

2) Select overlaying design image from gallery

Snack, code example, screenshot, or link to a repository

I couldn't really reproduce this in any test projects sadly.
This only happened in my big project, i guess its because it has more heavy operations.

https://github.com/sravis02/flashes-app
Branch: issue-react-native-skia
I invited you both (chrfalch & wcandillion) to my repository

The Canvas is in PreviewCanvas.tsx
The Snapshotfunction in preview.tsx

The text was updated successfully, but these errors were encountered:

grillorafael · 2023-10-23T21:14:16Z

I'm having the same issue. I checked the file and it is always empty

wcandillon · 2023-11-23T16:15:43Z

I'm struggling to understand the issue based on the video, could you walk me thought it?
We know have a test suite for this feature any tests you would like to me add there I can add.

ludwighen · 2023-12-22T21:10:23Z

Same issue but I believe it only happens in dev builds. Does the issue happen in a production build for you @sravis02?

What happens for me: I have a canvas with various elements. When I call makeImageSnapshot in the local dev builds, the resulting image sometimes doesn't contain all my elements from the canvas. It quickly blinks (like in your screen recording) and it seems as the elements are scaled down and on the top left corner falsely positioned.

In production I cannot reproduce this so far...

sravis02 · 2023-12-28T03:10:33Z

I'm struggling to understand the issue based on the video, could you walk me thought it? We know have a test suite for this feature any tests you would like to me add there I can add.

thats exactly the issue i am having. it occurs on expo development builds, expo prebuild builds and expo internal preview builds for me.

sravis02 · 2023-12-28T03:12:21Z

I'm struggling to understand the issue based on the video, could you walk me thought it? We know have a test suite for this feature any tests you would like to me add there I can add.

i will take a look into it again and try to provide some minimal reproduceable tests

cesargdm · 2024-02-22T00:56:46Z

I've encountered the same issue. When I call the makeImageSnapshot sometimes there's a small blink in the canvas (image of the frame attached).

2Simulator.Screen.Recording.-.iPhone.15.Pro.-.2024-02-21.at.18.49.05.mov

This is the code I'm using:

export default function ImagePreviewScreen() {
	const navigation = useNavigation()
	const route = useRoute<any>()
	const { t } = useTranslation()
	const { uri, serviceId, isFinal } = route.params ?? {}

	console.log('ImagePreviewScreen')

	const path = useSharedValue<string>('')

	const [isTransitioning, setIsTransitioning] = useState(true)
	const [initialCanvasSize, setInitialCanvasSize] = useState(null as Dimensions)
	const [currentCanvasSize, setCurrentCanvasSize] = useState(null as Dimensions)
	const [hasPath, setHasPath] = useState(false)
	const [isSaving, setIsSaving] = useState(false)

	useAnimatedReaction(
		() => Boolean(path.value),
		(currentValue, previousValue) => {
			if (currentValue !== previousValue) {
				runOnJS(setHasPath)(currentValue)
			}
		},
	)

	const canvasRef = useCanvasRef()

	const image = useImage(uri)

	const handleUndo = useCallback(() => {
		// Remove last line remove until last M
		path.value = path.value.replace(/ ?M[^M]*$/, '')
	}, [path])

	const handleLayout = useCallback(
		(event: LayoutChangeEvent) => {
			const { width, height } = event.nativeEvent.layout

			if (!initialCanvasSize) {
				setInitialCanvasSize({ width, height })
			}

			setCurrentCanvasSize({ width, height })
		},
		[initialCanvasSize],
	)

	const srcRect = useImageRect({ canvas: initialCanvasSize, uri })
	const dstRct = useImageRect({ canvas: currentCanvasSize, uri })

	const handleSave = useCallback(async () => {
		console.log('handleSave')

		setIsSaving(true)

		// Simply save image if there are no annotations
		if (!hasPath) {
			addServiceAttachment(uri, serviceId)
		} else {
			try {
				console.log('Make image snapshot')
				const canvas = canvasRef.current
				if (!canvas) return

				console.log(canvas.state)

				// TODO: use original image size
				// eslint-disable-next-line @typescript-eslint/no-non-null-assertion
				const snapshot = canvas.makeImageSnapshot(dstRct!)

				const base64Image = snapshot.encodeToBase64(ImageFormat.PNG)

				console.log('Image snapshot done')

				if (!base64Image) return

				const fileName =
					`${serviceId}-${Date.now()}.${SIGNATURE_EXTENSION}` as const

				const imageUri = documentDirectory + fileName

				console.log('Write image to file')
				await writeAsStringAsync(
					imageUri,
					base64Image.replace('data:image/png;base64,', ''),
					{ encoding: 'base64' },
				)

				snapshot.dispose()

				console.log('Add attachment to service')

				addServiceAttachment(imageUri, serviceId)
			} catch {
				// TODO: show error
			} finally {
				setIsSaving(false)
			}
		}

		navigation.goBack()
	}, [canvasRef, dstRct, hasPath, navigation, serviceId, uri])

	useLayoutEffect(() => {
		navigation.addListener('transitionEnd' as any, () => {
			setIsTransitioning(false)
		})

		navigation.setOptions({
			headerTintColor: 'white',
			headerStyle: { backgroundColor: 'black', color: 'white' },
			headerLeft() {
				return (
					<Button
						$variant="secondary"
						onPress={() => navigation.goBack()}
						$small
					>
						{t('cancel')}
					</Button>
				)
			},
			headerRight() {
				return isFinal ? null : (
					<Button
						disabled={isSaving}
						$variant="primary"
						onPress={handleSave}
						$small
					>
						{t('screens.image-preview.save')}
					</Button>
				)
			},
		})
	}, [handleSave, handleUndo, isFinal, isSaving, navigation, path.value, t])

	const panGesture = Gesture.Pan()
		.onStart((event) => {
			if (!srcRect || !dstRct) return

			let { x, y } = event

			if (
				JSON.stringify(initialCanvasSize) !== JSON.stringify(currentCanvasSize)
			) {
				const [{ translateX }, { translateY }, { scaleX }, { scaleY }] =
					rect2rect(srcRect, dstRct)

				x = (x - translateX) / scaleX
				y = (y - translateY) / scaleY
			}

			path.value = `${path.value} M ${x},${y}`
		})
		.onUpdate((event) => {
			if (!srcRect || !dstRct) return

			let { x, y } = event

			if (
				JSON.stringify(initialCanvasSize) !== JSON.stringify(currentCanvasSize)
			) {
				const [{ translateX }, { translateY }, { scaleX }, { scaleY }] =
					rect2rect(srcRect, dstRct)

				x = (x - translateX) / scaleX
				y = (y - translateY) / scaleY
			}

			path.value = `${path.value} L ${x},${y}`
		})

	const tapGesture = Gesture.Tap()
		.runOnJS(true)
		.onStart(() => {
			Keyboard.dismiss()
		})

	const composed = Gesture.Race(tapGesture, panGesture)

	return (
		<SafeAreaView edges={SAFE_AREA_EDGES} style={styles.container}>
			<EXStatusBar style="light" />
			{!isFinal && (
				<View style={styles.actionsContainer}>
					<Button
						disabled={!hasPath}
						$variant="secondary"
						onPress={handleUndo}
						$circle
					>
						<Button.Icon as={Ionicons} name="return-up-back-outline" />
					</Button>
					<View />
				</View>
			)}
			<KeyboardAvoidingView
				style={styles.container}
				behavior="padding"
				keyboardVerticalOffset={125}
			>
				<GestureDetector gesture={composed}>
					{isTransitioning ? (
						<View style={styles.empty} />
					) : (
						<Canvas
							ref={canvasRef}
							onLayout={handleLayout}
							mode="continuous"
							style={styles.canvas}
						>
							{dstRct ? (
								<Group clip={dstRct}>
									<Image
										image={image}
										x={dstRct.x}
										y={dstRct.y}
										width={dstRct.width}
										height={dstRct.height}
									/>

									{srcRect && dstRct ? (
										<FitBox src={srcRect} fit="fill" dst={dstRct}>
											<Path
												strokeCap="round"
												path={path}
												strokeWidth={5}
												style="stroke"
												color="red"
											/>
										</FitBox>
									) : null}
								</Group>
							) : null}
						</Canvas>
					)}
				</GestureDetector>

				<View style={styles.textInputContainer}>
					<TextInput
						placeholderTextColor="gray"
						style={{ color: 'white' }}
						$color="white"
						placeholder={
							isFinal ? '' : t('screens.image-preview.description.placeholder')
						}
						readOnly={isFinal}
						hideError
						hideBorder
						multiline
					/>
				</View>
			</KeyboardAvoidingView>
		</SafeAreaView>
	)
}

cesargdm · 2024-02-22T01:13:10Z

This is very similar to the issues @ludwighen had

cesargdm · 2024-02-23T05:54:24Z

Probably related to #2133

ludwighen · 2024-02-25T15:12:36Z

Following up on this, I think I found a workaround for the issue @cesargdm.

Indeed I have the same issue and I first thought it would only happen in my development builds but it also happened in production, only for less powerful devices. It was probably visible in the development builds because they are a bit less performant, and something obviously tries to render the moment the snapshot is taken. I assume in most attempts it's fast enough to take the image after this drawing process.

I figured what if I just delay the makeImageScreenshot by one render cycle using setTimeout. And it works. I have no idea why 😅, but with over 100 attempts, I now can no longer reproduce the issue, so I believe it fixed it. Let me know if it works for you too.

Here's my code:

  const onSavePress = useCallback(async () => {
   //... some other stuff

    setTimeout(() => {
      const image = ref.current?.makeImageSnapshot();
      if (!image) {
        // handle error
        return;
      }

      setSkiaImage(image);
    }, 0);
  }, [ref]);

sravis02 · 2024-02-25T16:58:38Z

thanks a lot, this workaround works for me :) @ludwighen

ludwighen · 2024-02-25T18:38:56Z

Update: Unfortunately the hack doesn't work fully reliable after all..
It causes the issue to disappear when I have just a handful of elements in the canvas.
But when I add say 20 images to the canvas, I still get the glitching.

Feels to me like some sort of race condition where the snapshot happens just before the drawing on the canvas is finished?

wcandillon · 2024-02-25T18:42:54Z

this bug looks interesting and I will investigate it further but in the meantime, I recommend to draw the content offscreen and take a snapshot of that: https://shopify.github.io/react-native-skia/docs/animations/textures#usetexture

ludwighen · 2024-02-26T07:15:49Z

Thanks @wcandillon, indeed that's what I planned to do next... I'll keep you posted

wcandillon · 2024-02-26T15:48:01Z

Gang, I think that this PR may fix the issue: #2254, any chance you could try this patch on your app?

ludwighen · 2024-02-26T21:18:21Z

nice!
Probably dumb question but do I need to build skia locally to do that?
If I install from your branch it get's added as react-native-skia-dev-tools.

wcandillon · 2024-02-26T21:29:47Z

that's a fair question, I was asking just in case, we will publish this bug fix soon so you will be able to test on the publish version. But now I am realizing that these a not related most likely. I will try to investigate this particular issue still

…

On Mon, Feb 26, 2024 at 10:18 PM ludwighen ***@***.***> wrote: nice! Probably dumb question but do I need to build skia locally to do that? If I install from your branch it get's added as react-native-skia-dev-tools. — Reply to this email directly, view it on GitHub or unsubscribe. You are receiving this email because you commented on the thread. Triage notifications on the go with GitHub Mobile for iOS or Android.

ludwighen · 2024-02-28T20:05:13Z

So I tested your fix with 0.1.241 beta but the issue is still there.

Next I tried the useTexture approach but I cannot get the basic example working.
It throws an error: Value is undefined, expected an Object at root.render(element); in Offscreen.tsx.

  const texture = useTexture(
      <Fill color="cyan" />,
    { width, height }
  );

Any idea why?

While looking into all of this, I was wondering if my current approach is even the "right" way, I think I should switch to the imperative API. I know it's not the right thread for it, but given that it would likely mean a significant refactor for me, maybe you could point in me the right direction since the documentation is a bit limited for the imperative API 🥲.

Basically, I have a canvas that can contain lots of different elements from the user (images, svgs, paths, etc.), think something like the Instagram stories editor / stickers example.
I use the declarative API like this:

  <Canvas>
      {elements.map((element, index) => (
        <CanvasElement
          gestureInfo={gestureInfo[element.uid]}
          key={element.uid}
          element={element}
          canvasWidth={canvasWidth}
          canvasHeight={canvasHeight}
          clock={clock}
          color={element.color}
        />
      ))}
 </Canvas>

And then essentially the CanvasElement conditionally renders Skia images, svgs, paths, etc. and applies transforms from GestureHandlers.

The useTexture hook would not allow me to pass the same logic in, would it?
Should I instead create all of the conditional drawing with the imperative API and just display the results on screen? Also performance wise, is there a difference between declarative vs imperative? I can have up to 100 elements/images drawn on the canvas in my case.
Any pointers would be welcome! I also see quite a few people out there that use Skia for a creative "editor" with a preview and an export stage, it could be cool to have resources on how to handle offscreen vs onscreen.

Anyways, thanks a ton @wcandillon as always 🙏

ludwighen · 2024-03-01T19:14:42Z

Update: I finally was able to solve it (this time reliably @sravis02 @cesargdm).

Not using the canvas ref's makeImageSnapshot() but instead drawing offscreen indeed fixes the problem and it also revealed why the items glitched small in the top left corner. It has something to do with the scale/pixel density.

What I did is just putting the Canvas children 1 to 1 but it has the caveat that the sizing is not identical to the onscreen canvas:

import { drawAsImage } from '@shopify/react-native-skia';

const { scale } = useWindowDimensions();

const scaledHeight = canvasHeight * scale;
const scaledWidth = canvasWidth * scale;

const image = drawAsImage(
      <>
        <Rect
          x={0}
          y={0}
          height={scaledHeight}
          width={scaledWidth}
          color={color}
          }
        />
        {elements.map((element, index) => (
          <CanvasElement
            gestureInfo={gestureInfo[element.uid]}
            //... I also needed to adjust the scale of width/height of my elements here
         />
        ))}
      </>,
      {
        width: scaledWidth,
        height: scaledHeight,
      },
    );

 const base64 = image.encodeToBase64();

So basically, when I use the normal canvasWidth/canvasHeight (via height/width from useWindowDimensions), the offscreen canvas is very small because it uses them as absolute pixel values whereas with the declarative API the pixels are scaled according to device (i.e. 3x for iPhones).

So the little glitch we see when using the ref's makeImageSnapshot method is basically the non-upscaled dimensions. Does this somehow help you in debugging @wcandillon ?

Hope that helps

wcandillon · 2024-04-04T07:07:51Z

Thank you for this.
I filed a seperate bug report for the offscreen scaling: #2336
This is gonna sound silly but I knew that of course these need to be scaled to pixel density but couldn't build an example where I would notice the difference: https://github.com/Shopify/react-native-skia/blob/main/package/src/renderer/Offscreen.tsx#L35

But I will definitely revisit.

For makeImageSnapshot, we know offer makeImageSnapshotAsync that runs on the UI thread so it shares the same Skia context than your onscreen canvas: https://shopify.github.io/react-native-skia/docs/canvas/overview/#getting-a-canvas-snapshot

sravis02 added the bug Something isn't working label Aug 27, 2023

wcandillon mentioned this issue Apr 4, 2024

Document the relationship between offscreen canvas and pixel density #2336

Closed

wcandillon closed this as completed Apr 4, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Buggy .makeImageSnapshot() on IOS #1811

Buggy .makeImageSnapshot() on IOS #1811

sravis02 commented Aug 27, 2023 •

edited

grillorafael commented Oct 23, 2023

wcandillon commented Nov 23, 2023

ludwighen commented Dec 22, 2023

sravis02 commented Dec 28, 2023

sravis02 commented Dec 28, 2023

cesargdm commented Feb 22, 2024

cesargdm commented Feb 22, 2024

cesargdm commented Feb 23, 2024

ludwighen commented Feb 25, 2024

sravis02 commented Feb 25, 2024

ludwighen commented Feb 25, 2024

wcandillon commented Feb 25, 2024

ludwighen commented Feb 26, 2024

wcandillon commented Feb 26, 2024

ludwighen commented Feb 26, 2024

wcandillon commented Feb 26, 2024 via email

ludwighen commented Feb 28, 2024

ludwighen commented Mar 1, 2024

wcandillon commented Apr 4, 2024

Buggy .makeImageSnapshot() on IOS #1811

Buggy .makeImageSnapshot() on IOS #1811

Comments

sravis02 commented Aug 27, 2023 • edited

Description

What bugs?:

Version

Steps to reproduce

How to reproduce:

Snack, code example, screenshot, or link to a repository

grillorafael commented Oct 23, 2023

wcandillon commented Nov 23, 2023

ludwighen commented Dec 22, 2023

sravis02 commented Dec 28, 2023

sravis02 commented Dec 28, 2023

cesargdm commented Feb 22, 2024

cesargdm commented Feb 22, 2024

cesargdm commented Feb 23, 2024

ludwighen commented Feb 25, 2024

sravis02 commented Feb 25, 2024

ludwighen commented Feb 25, 2024

wcandillon commented Feb 25, 2024

ludwighen commented Feb 26, 2024

wcandillon commented Feb 26, 2024

ludwighen commented Feb 26, 2024

wcandillon commented Feb 26, 2024 via email

ludwighen commented Feb 28, 2024

ludwighen commented Mar 1, 2024

wcandillon commented Apr 4, 2024

sravis02 commented Aug 27, 2023 •

edited