This idea came from Framer Motion docs. I'll show you the challenges I encountered moving this into my project and fixes I made.

Framer Motion

The Framer docs: Video scroll scrub

Their code:

import { scroll } from 'framer-motion/dom';

const video = document.querySelector('video');
video.pause();

// Scrub through the video on scroll
scroll((progress) => {
  if (video.readyState) {
    video.currentTime = video.duration * progress;
  }
});

The function scroll takes a callback that has a progress value depending on scrollY position. The callback is called on every scroll event and each time we update the video's currentTime based on the progress value.

Now in React

When moving this into the react world I had to handle a few extra things.

1. I needed to translate this to react hooks (easy enough)
2. On mobile, the video doesnt auto load to save on bandwidth/battery. So we needed a hack around that.
3. The scrolling caused a janky experience where the video would freeze on a frame until the scroll stopped. That won't do.

Let's see how we did each of these.

1. Translating to React Hooks

I'm using useScroll in place of scroll. Easy.

I'm using useMotionValueEvent to listen to the scroll progress and update the video's currentTime.

import { useScroll, useMotionValueEvent } from 'framer-motion';

const MyComponent = () => {
  const videoRef = (useRef < HTMLVideoElement) | (null > null);
  const { scrollYProgress } = useScroll();

  useMotionValueEvent(scrollYProgress, 'change', (latest) => {
    if (videoRef.current) {
      videoRef.current.currentTime = videoRef.current.duration * latest;
    }
  });

  return (
    <video
      ref={videoRef}
      preload="auto"
      autoPlay
      loop
      muted
      playsInline
      poster={bridgePlaceholder.src}
    >
      <source
        src={'https://example.com/assets/or/cdn/video.mp4'}
        type="video/mp4"
      />
    </video>
  );
};

Great start, nothing loaded on mobile and the video was freezing on a frame until the scroll stopped.

Let's fix mobile first.

2. On Mobile

On mobile, the video doesn't auto-load to save on bandwidth/battery. We are going to make a little hack around that.

Because our hack will go against the web standard practices, we should accept that this may not work on all devices and could break at any time.

With that said, let's pragmatically play the video on load and then immediately pause it.

import { useScroll, useMotionValueEvent } from 'framer-motion';

const MyComponent = () => {
  const videoRef = (useRef < HTMLVideoElement) | (null > null);
  const { scrollYProgress } = useScroll();

  useMotionValueEvent(scrollYProgress, 'change', (latest) => {
    if (videoRef.current) {
      videoRef.current.currentTime = videoRef.current.duration * latest;
    }
  });

  /* prettier-ignore */
  useEffect(() => { 
    if (videoRef.current) { 
      videoRef.current.play(); 
      videoRef.current.pause(); 
    } 
  }, []); 

  return (
    <video
      ref={videoRef}
      preload="auto"
      autoPlay
      loop
      muted
      playsInline
      poster={bridgePlaceholder.src}
    >
      <source
        src={'https://example.com/assets/or/cdn/video.mp4'}
        type="video/mp4"
      />
    </video>
  );
};

Now the video shows on mobile! Let's fix the janky experience scroll experience.

const MyComponent = () => {
  const videoRef = (useRef < HTMLVideoElement) | (null > null);
  const { scrollYProgress } = useScroll();

  /* prettier-ignore */
  useMotionValueEvent(scrollYProgress, 'change', (latest) => {
    if (videoRef.current && !videoRef.current.seeking) { 
      window.requestAnimationFrame(() => { 
        if (videoRef.current) { 
          videoRef.current.currentTime = videoRef.current.duration * latest * 1; 
        } 
      }); 
    }
  });

  useEffect(() => {
    if (videoRef.current) {
      videoRef.current.play();
      videoRef.current.pause();
    }
  }, []);

  return (
    <video
      ref={videoRef}
      preload="auto"
      autoPlay
      loop
      muted
      playsInline
      poster={bridgePlaceholder.src}
    >
      <source
        src={'https://example.com/assets/or/cdn/video.mp4'}
        type="video/mp4"
      />
    </video>
  );
};

The biggest gotcha was the need to wait for the video to finish seeking before updating the currentTime again.

I'm pretty sure this has to do with the way the video is compressed. The video does NOT exist as a sequence of frames like our mental model might suggest.

Instead, the video has a subset of frames that are key frames and contains data that is used to diff from one frame to the next.