# Efficient playback of video

**URL:** <https://openrndr.discourse.group/t/efficient-playback-of-video/364>\
**Category:** Beginners\
**Created:** [April 18, 2022, 2:06pm UTC](https://openrndr.discourse.group/t/efficient-playback-of-video/364 "2022-04-18T14:06:18Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![capital-G](https://avatars.discourse-cdn.com/v4/letter/c/958977/32.png) [@capital-G](https://openrndr.discourse.group/u/capital-G)\
**Post date:** [April 18, 2022, 2:06pm UTC](https://openrndr.discourse.group/t/efficient-playback-of-video/364/1 "2022-04-18T14:06:18Z")

</div>

Hi,

I currently try to understand how one can efficiently play multiple videos at the same time and also how to seek in them and modulate the playback speed.

A naive approach was

```kotlin
import org.openrndr.application
import org.openrndr.ffmpeg.loadVideo

fun main() = application {
    configure {
        width = 1280
        height = 720
    }

    program {
        val video = loadVideo("data/videos/small/my_video.MOV")

        video.play()
        video.seek(4.0)
        video.pause()
        extend {
            drawer.clear(ColorRGBa.BLACK)
            video.draw(drawer)
        }
    }
}

```

Which results in a blue screen and multiple

```auto
error? -35

```

in the stdout.

Jumping to the same second within the for loop results in a stuttering movement and a CPU load of around 1000% on my macbook pro i9.  
Both of these aspects makes it not feasible IMO to implement a slow playback of the video through a manual step through the seconds of the video.

When I want to play multiple videos I also get the error messages

```auto
 WARN [Thread-6(decoder)] o.o.f.Decoder ↘ video queue is almost full. [video queue: 49, audio queue: 0]

```

and the CPU is beyond 16000%. I already reduced the resolution down to 720p w/ ffmpeg. With ffprobe I get an unnoticeable CPU usiage.

Is OPENRNDR performant enough to control the playback of 6 720p videos, layered on top of each other in 30fps realtime?

---

<div class="post-metadata">

**Author:** ![abe](https://yyz2.discourse-cdn.com/free1/user_avatar/openrndr.discourse.group/abe/32/700_2.png) [@abe](https://openrndr.discourse.group/u/abe)\
**Post date:** [April 18, 2022, 4:37pm UTC](https://openrndr.discourse.group/t/efficient-playback-of-video/364/2 "2022-04-18T16:37:06Z")

</div>

Welcome to the forum 🙂

I haven’t tried playing 6 videos at the time but I do know some codecs are much better for seeks than others. One reason is that with certain codecs frames can not be just read as an image from a video file, but are actually constructed based on previous frames. So requesting a specific frame can be slow because it needs to put it together first. This issue can be avoided by configuring the codec, using a different codec or maybe just loading single images (PNG is probably slower than JPG or uncompressed TIF).

The codec I remember people using is [HAP](https://hap.video/). Maybe if you encode your video with it things are smoother?

---

<div class="post-metadata">

**Author:** ![capital-G](https://avatars.discourse-cdn.com/v4/letter/c/958977/32.png) [@capital-G](https://openrndr.discourse.group/u/capital-G)\
**Post date:** [April 18, 2022, 7:05pm UTC](https://openrndr.discourse.group/t/efficient-playback-of-video/364/3 "2022-04-18T19:05:55Z")

</div>

Thanks for the hint with HAP but here a single video still consumes 900% of a CPU with a reset within the loop to an exact second.

Also the warning

```auto
 WARN [Thread-3(decoder)] o.o.f.Decoder ↘ video queue is almost full. [video queue: 49, audio queue: 0]

```

is present.

edit: it also stutters like the original hvc1 file

---

<div class="post-metadata">

**Author:** ![abe](https://yyz2.discourse-cdn.com/free1/user_avatar/openrndr.discourse.group/abe/32/700_2.png) [@abe](https://openrndr.discourse.group/u/abe)\
**Post date:** [April 18, 2022, 7:23pm UTC](https://openrndr.discourse.group/t/efficient-playback-of-video/364/4 "2022-04-18T19:23:32Z")

</div>

I will ask in the chat and try on my own computer.

BTW those percentages are odd. How can one use more than 100% of a CPU? 🙂

* * *

Update: you can configure advanced settings like this:

```auto
val video = VideoPlayerFFMPEG.fromFile(
    "/Videos/myVid.mp4",
    mode = PlayMode.VIDEO, // no sound
    configuration = VideoPlayerConfiguration().also {
        it.allowArbitrarySeek = true
        //it....
    }
)

```

Question: are you trying to loop video files? If yes, how long are they and how do you loop them?

---

<div class="post-metadata">

**Author:** ![abe](https://yyz2.discourse-cdn.com/free1/user_avatar/openrndr.discourse.group/abe/32/700_2.png) [@abe](https://openrndr.discourse.group/u/abe)\
**Post date:** [April 18, 2022, 8:17pm UTC](https://openrndr.discourse.group/t/efficient-playback-of-video/364/5 "2022-04-18T20:17:16Z")

</div>

This is something I came up with:

```kotlin
import org.openrndr.application
import org.openrndr.color.ColorRGBa
import org.openrndr.ffmpeg.*
import kotlin.math.sin

fun main() = application {
    configure {
        width = 1280
        height = 720
    }

    program {
        val video = VideoPlayerFFMPEG.fromFile(
            "/tmp/a.mp4",
            PlayMode.VIDEO,
            VideoPlayerConfiguration().apply {
                useHardwareDecoding = false
                videoFrameQueueSize = 500
                displayQueueCooldown = 5
                //synchronizeToClock = false
            }
        ) { seconds + sin(seconds) }

        video.play()
        video.ended.listen {
            video.seek(0.0)
        }

        extend {
            drawer.clear(ColorRGBa.BLACK)
            video.draw(drawer)
        }
    }
}

```

This was with trial and error 🙂 It seems like you can pass a function which keeps the time. If the function is for example `{ seconds / 5.0 }` then it plays 5 time slower. In this example I modulated the speed using a sine wave.

I tried setting `videoFrameQueueSize = 500` to keep all frames in memory, not sure if that worked. But I do not see any big hiccups when jumping from end to start.

I believe that if I set `synchronizeToClock = false` it plays as fast as possible, which is at 60 fps by default.

Ah I set `useHardwareDecoding = false` because otherwise it wasn’t working on my system. Maybe related to a recent update.

Maybe some of that helps you?

---

<div class="post-metadata">

**Author:** ![capital-G](https://avatars.discourse-cdn.com/v4/letter/c/958977/32.png) [@capital-G](https://openrndr.discourse.group/u/capital-G)\
**Post date:** [April 18, 2022, 10:59pm UTC](https://openrndr.discourse.group/t/efficient-playback-of-video/364/6 "2022-04-18T22:59:37Z")

</div>

I try to jump to specific points in the video and freeze at specific points in time and seek in the video back and forth - based on received osc messages.

Thanks for giving the hint on using the clock instead of seek. Although I do not fully understand how this clock behaves as using a fixed value does not allow me to seek to specific moments in the video, so something along

```kotlin
clock = {(seconds*speed) + offset}

```

for `speed=0` and `offset=4` still playbacks the video normally to second 4 and then stops. It does not allow me to go back at e.g. `offset=2`.

Would be great to have some documentation on the clock specifics at least within the source code 😕

ps: In Unix you can get over 100% cpu as 100% stands for one core is used fully - but e.g. my i9 has 16 vcores so i have 16000% cpu available.

---

<div class="post-metadata">

**Author:** ![abe](https://yyz2.discourse-cdn.com/free1/user_avatar/openrndr.discourse.group/abe/32/700_2.png) [@abe](https://openrndr.discourse.group/u/abe)\
**Post date:** [April 19, 2022, 6:30am UTC](https://openrndr.discourse.group/t/efficient-playback-of-video/364/7 "2022-04-19T06:30:44Z")

</div>

Which version of OPENRNDR are you using? I [build OPENRNDR](https://github.com/openrndr/openrndr/wiki/Building-OPENRNDR#simple-openrndr-build) from github to have the latest version. On March 6th ffmpeg was updated. I run it on an i5 CPU.

I [converted](https://gist.github.com/dlublin/e4585b872dd136ae88b2aa51a6a89aac) a video to HAP:

```bash
ffmpeg -i a.mp4 -c:v hap /tmp/a.mkv
# /tmp is a ram mounted disk in my case, fwiw
# /etc/fstab
# tmpfs /tmp tmpfs mode=1777,nosuid,nodev 0 0

```

This brings the mp4 file from 50 Mb to 330 Mb.

Then this nicely jitters the video quite reliably. I don’t see any pauses.

```kotlin
        video.play()
        extend {
            video.draw(drawer)
            if (frameCount % 5 == 0) {
                video.seek(seconds.toInt().toDouble())
            }
        }

```

and this provides random pauses in the video

```kotlin
        video.play()
        var pauseFrameCount = 0
        extend {
            video.draw(drawer)
            if(frameCount == pauseFrameCount) {
                if(Random.bool()) video.pause()
            }
            if (frameCount % 60 == 0) {
                video.resume()
                video.seek(Random.double0(video.duration))
                pauseFrameCount = frameCount + 1
            }
        }

```

If you share a minimal example showing the issue (including media) I can try it.

ps. In `src/main/resources/log4j2.yaml` I can hide warnings as I get one per jump:

```nohighlight
  Loggers:
    Root:
      level: error

```