Efficient playback of video

OPENRNDR Beginners
1

Hi,

I currently try to understand how one can efficiently play multiple videos at the same time and also how to seek in them and modulate the playback speed.

A naive approach was

import org.openrndr.application
import org.openrndr.ffmpeg.loadVideo

fun main() = application {
    configure {
        width = 1280
        height = 720
    }

    program {
        val video = loadVideo("data/videos/small/my_video.MOV")

        video.play()
        video.seek(4.0)
        video.pause()
        extend {
            drawer.clear(ColorRGBa.BLACK)
            video.draw(drawer)
        }
    }
}

Which results in a blue screen and multiple

error? -35

in the stdout.

Jumping to the same second within the for loop results in a stuttering movement and a CPU load of around 1000% on my macbook pro i9.
Both of these aspects makes it not feasible IMO to implement a slow playback of the video through a manual step through the seconds of the video.

When I want to play multiple videos I also get the error messages

 WARN [Thread-6(decoder)] o.o.f.Decoder                   ↘ video queue is almost full. [video queue: 49, audio queue: 0]

and the CPU is beyond 16000%. I already reduced the resolution down to 720p w/ ffmpeg. With ffprobe I get an unnoticeable CPU usiage.

Is OPENRNDR performant enough to control the playback of 6 720p videos, layered on top of each other in 30fps realtime?

2

Welcome to the forum :slight_smile:

I haven’t tried playing 6 videos at the time but I do know some codecs are much better for seeks than others. One reason is that with certain codecs frames can not be just read as an image from a video file, but are actually constructed based on previous frames. So requesting a specific frame can be slow because it needs to put it together first. This issue can be avoided by configuring the codec, using a different codec or maybe just loading single images (PNG is probably slower than JPG or uncompressed TIF).

The codec I remember people using is HAP. Maybe if you encode your video with it things are smoother?

3

Thanks for the hint with HAP but here a single video still consumes 900% of a CPU with a reset within the loop to an exact second.

Also the warning

 WARN [Thread-3(decoder)] o.o.f.Decoder                   ↘ video queue is almost full. [video queue: 49, audio queue: 0]

is present.

edit: it also stutters like the original hvc1 file

4

I will ask in the chat and try on my own computer.

BTW those percentages are odd. How can one use more than 100% of a CPU? :slight_smile:

Update: you can configure advanced settings like this:

val video = VideoPlayerFFMPEG.fromFile(
    "/Videos/myVid.mp4",
    mode = PlayMode.VIDEO, // no sound
    configuration = VideoPlayerConfiguration().also {
        it.allowArbitrarySeek = true
        //it....
    }
)

Question: are you trying to loop video files? If yes, how long are they and how do you loop them?

5

This is something I came up with:

import org.openrndr.application
import org.openrndr.color.ColorRGBa
import org.openrndr.ffmpeg.*
import kotlin.math.sin

fun main() = application {
    configure {
        width = 1280
        height = 720
    }

    program {
        val video = VideoPlayerFFMPEG.fromFile(
            "/tmp/a.mp4",
            PlayMode.VIDEO,
            VideoPlayerConfiguration().apply {
                useHardwareDecoding = false
                videoFrameQueueSize = 500
                displayQueueCooldown = 5
                //synchronizeToClock = false
            }
        ) { seconds + sin(seconds) }

        video.play()
        video.ended.listen {
            video.seek(0.0)
        }

        extend {
            drawer.clear(ColorRGBa.BLACK)
            video.draw(drawer)
        }
    }
}

This was with trial and error :slight_smile: It seems like you can pass a function which keeps the time. If the function is for example { seconds / 5.0 } then it plays 5 time slower. In this example I modulated the speed using a sine wave.

I tried setting videoFrameQueueSize = 500 to keep all frames in memory, not sure if that worked. But I do not see any big hiccups when jumping from end to start.

I believe that if I set synchronizeToClock = false it plays as fast as possible, which is at 60 fps by default.

Ah I set useHardwareDecoding = false because otherwise it wasn’t working on my system. Maybe related to a recent update.

Maybe some of that helps you?

6

I try to jump to specific points in the video and freeze at specific points in time and seek in the video back and forth - based on received osc messages.

Thanks for giving the hint on using the clock instead of seek. Although I do not fully understand how this clock behaves as using a fixed value does not allow me to seek to specific moments in the video, so something along

clock = {(seconds*speed) + offset}

for speed=0 and offset=4 still playbacks the video normally to second 4 and then stops. It does not allow me to go back at e.g. offset=2.

Would be great to have some documentation on the clock specifics at least within the source code :confused:

ps: In Unix you can get over 100% cpu as 100% stands for one core is used fully - but e.g. my i9 has 16 vcores so i have 16000% cpu available.

7

Which version of OPENRNDR are you using? I build OPENRNDR from github to have the latest version. On March 6th ffmpeg was updated. I run it on an i5 CPU.

I converted a video to HAP:

ffmpeg -i a.mp4 -c:v hap /tmp/a.mkv
# /tmp is a ram mounted disk in my case, fwiw
# /etc/fstab
# tmpfs /tmp tmpfs mode=1777,nosuid,nodev 0 0

This brings the mp4 file from 50 Mb to 330 Mb.

Then this nicely jitters the video quite reliably. I don’t see any pauses.

        video.play()
        extend {
            video.draw(drawer)
            if (frameCount % 5 == 0) {
                video.seek(seconds.toInt().toDouble())
            }
        }

and this provides random pauses in the video

        video.play()
        var pauseFrameCount = 0
        extend {
            video.draw(drawer)
            if(frameCount == pauseFrameCount) {
                if(Random.bool()) video.pause()
            }
            if (frameCount % 60 == 0) {
                video.resume()
                video.seek(Random.double0(video.duration))
                pauseFrameCount = frameCount + 1
            }
        }

If you share a minimal example showing the issue (including media) I can try it.

ps. In src/main/resources/log4j2.yaml I can hide warnings as I get one per jump:

  Loggers:
    Root:
      level: error