How to debug a Ffmpeg --11 error

OPENRNDR How to?
1

My current OPENRNDR program does the following:

  1. loads videos and image files from a ‘playlist’ folder and assigns them to Scenes on a timeline
  2. plays these files sequentially after the project is loaded and I press space bar.

On the surface, it’s current state works perfectly.
Problem arises when I try to load two videos.
When a second video is loaded and played, the console log shows a repeated “error? -11” message.
This only happens when more than one video is loaded and played.
Strange thing is that the error occurs but the visual output is perfect and the program doesn’t crash. For the life of me, I couldn’t find anywhere what “error? -11” means.

I’m not asking for a fix or anything, but guidance on where to look/research to fix this. Thanks in advance.

Here’s the relevant code parts:

// File: utils/VideoManager.kt
package utils

import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.withContext
import models.Scene
import models.VideoState
import org.openrndr.ffmpeg.VideoPlayerFFMPEG
import java.io.File
import java.util.logging.Logger

object VideoManager {
    private val logger = Logger.getLogger(VideoManager::class.java.name)
    private val loadedVideos = mutableMapOf<File, VideoPlayerFFMPEG>()
    private val loadingVideos = mutableSetOf<File>()

    suspend fun loadVideo(file: File): VideoPlayerFFMPEG? {
        return when {
            loadedVideos.containsKey(file) -> {
                logger.fine("Video already loaded: ${file.name}")
                loadedVideos[file]
            }
            loadingVideos.contains(file) -> {
                logger.fine("Video is currently loading: ${file.name}")
                null
            }
            else -> {
                loadingVideos.add(file)
                try {
                    logger.info("Loading video: ${file.name}")
                    val video = withContext(Dispatchers.IO) {
                        VideoPlayerFFMPEG.fromFile(file.absolutePath)
                    }
                    loadedVideos[file] = video
                    logger.info("Successfully loaded video: ${file.name}")
                    video
                } catch (e: Exception) {
                    logger.warning("Failed to load video: ${file.name} - ${e.message}")
                    null
                } finally {
                    loadingVideos.remove(file)
                }
            }
        }
    }

    fun releaseVideo(scene: Scene) {
        scene.videoFile?.let { file ->
            logger.info("Attempting to release video: ${file.name}")
            loadedVideos[file]?.let { video ->
                stopVideo(scene)
                video.dispose()
                loadedVideos.remove(file)
                logger.info("Successfully released video: ${file.name}")
            }
        }
    }

    fun releaseAllVideos(scenes: List<Scene>) {
        logger.info("Releasing all videos")
        scenes.forEach { scene ->
            scene.videoFile?.let { file ->
                loadedVideos[file]?.let { video ->
                    stopVideo(scene)
                    video.dispose()
                    logger.info("Released video: ${file.name}")
                }
            }
        }
        loadedVideos.clear()
        loadingVideos.clear()
        logger.info("All videos released")
    }

    private fun stopVideo(scene: Scene) {
        scene.videoPlayer?.let { player ->
            logger.info("Stopping video for scene ${scene.number}")
            player.pause()
            player.seek(0.0)
            scene.videoState = VideoState.STOPPED
        }
    }
}

import org.openrndr.application
import org.openrndr.color.ColorRGBa
import org.openrndr.draw.loadFont
import org.openrndr.draw.loadImage
import org.openrndr.draw.ColorBuffer
import org.openrndr.dialogs.openFolderDialog
import kotlinx.coroutines.*
import models.Project
import models.VideoState
import renderers.TimelineRenderer
import utils.*
import java.io.File
import java.util.logging.Logger
import java.util.logging.Level

private val logger = Logger.getLogger("MainApplication")

fun main() = application {
    configure {
        width = 960
        height = 540
        title = "ChannelFactory Video Editor"
    }

    program {
        logger.info("Initializing ChannelFactory Video Editor")
        val font = loadFont("data/fonts/default.otf", 16.0)
        var project: Project? = null
        var projectFolder: File? = null
        var message: String = "ChannelFactory Video Editor"
        var timelineRenderer: TimelineRenderer? = null
        val loadedImages = mutableMapOf<File, ColorBuffer>()

        fun loadImageIfNeeded(file: File): ColorBuffer? {
            return loadedImages[file] ?: run {
                try {
                    logger.info("Loading image: ${file.name}")
                    val image = loadImage(file)
                    loadedImages[file] = image
                    logger.info("Image loaded successfully: ${file.name}")
                    image
                } catch (e: Exception) {
                    logger.log(Level.WARNING, "Failed to load image: ${file.name}", e)
                    null
                }
            }
        }

        fun reloadProject(projectFolder: File) {
            logger.info("Reloading project from folder: ${projectFolder.absolutePath}")
            runBlocking {
                launch(Dispatchers.Default) {
                    val loadedProject = loadProject(projectFolder)
                    if (loadedProject != null) {
                        project = loadedProject
                        timelineRenderer = TimelineRenderer(loadedProject.scenes)
                        message = "Project reloaded successfully!"
                        loadedImages.clear()
                        logger.info("Releasing all videos before reloading")
                        VideoManager.releaseAllVideos(loadedProject.scenes)

                        logger.info("Preloading videos asynchronously")
                        loadedProject.scenes.forEach { scene ->
                            scene.videoFile?.let { file ->
                                launch(Dispatchers.IO) {
                                    logger.info("Preloading video for scene ${scene.number}: ${file.name}")
                                    VideoManager.loadVideo(file)
                                }
                            }
                        }
                        logger.info("Project reloaded successfully")
                    } else {
                        logger.warning("Failed to reload project. Invalid folder selected.")
                        message = "Failed to reload project. Please select a valid folder."
                    }
                }
            }
        }

        extend {
            val currentTime = project?.let { proj ->
                if (proj.playing) System.currentTimeMillis() - proj.startTime else proj.startTime
            } ?: 0L

            drawer.clear(ColorRGBa.BLACK)
            drawer.fill = ColorRGBa.WHITE
            drawer.fontMap = font

            drawer.text(message, 20.0, 30.0)
            drawer.text("Press 'L' to load a project", 20.0, 60.0)
            drawer.text("Press 'Space' to play/pause", 20.0, 90.0)
            drawer.text("Press 'R' to stop", 20.0, 120.0)

            project?.let { proj ->
                drawer.text("Total Scenes: ${proj.scenes.size}", 20.0, 150.0)
                proj.scenes.forEachIndexed { index, scene ->
                    val minutes = scene.startTime / 60000
                    val seconds = (scene.startTime % 60000) / 1000
                    val firstWords = scene.text.split(" ").take(5).joinToString(" ") + "..."
                    drawer.text("Scene ${index + 1}: ${minutes}m ${seconds}s - $firstWords", 20.0, 180.0 + index * 20.0)
                }

                timelineRenderer?.draw(drawer, width.toDouble(), height.toDouble(), mouse.position)

                val activeScene = proj.scenes.findLast { it.startTime <= currentTime }
                activeScene?.let { scene ->
                    logger.fine("Processing active scene ${scene.number}")
                    proj.sceneMediaMap[scene.number]?.forEach { file ->
                        when (file.extension.lowercase()) {
                            in listOf("jpg", "jpeg", "png", "bmp") -> {
                                loadImageIfNeeded(file)?.let { image ->
                                    val imageScale = minOf(width.toDouble() / image.width, height.toDouble() / image.height)
                                    val imageWidth = image.width * imageScale
                                    val imageHeight = image.height * imageScale
                                    val imageX = (width - imageWidth) / 2.0
                                    val imageY = (height - imageHeight) / 2.0
                                    drawer.image(image, imageX, imageY, imageWidth, imageHeight)
                                }
                            }
                            in listOf("mp4", "avi", "mov") -> {
                                runBlocking {
                                    logger.fine("Processing video file: ${file.name}")
                                    VideoManager.loadVideo(file)?.let { player ->
                                        when (proj.playing) {
                                            true -> {
                                                if (scene.videoState == VideoState.STOPPED) {
                                                    logger.info("Starting video from beginning for scene ${scene.number}")
                                                    player.seek(0.0)
                                                    player.play()
                                                    scene.videoState = VideoState.PLAYING
                                                } else if (scene.videoState == VideoState.PAUSED) {
                                                    logger.info("Resuming paused video for scene ${scene.number}")
                                                    player.play()
                                                    scene.videoState = VideoState.PLAYING
                                                }
                                            }
                                            false -> {
                                                if (scene.videoState == VideoState.PLAYING) {
                                                    logger.info("Pausing playing video for scene ${scene.number}")
                                                    player.pause()
                                                    scene.videoState = VideoState.PAUSED
                                                }
                                            }
                                        }
                                        player.draw(drawer, 0.0, 0.0, width.toDouble(), height.toDouble())
                                    }
                                }
                            }
                        }
                    }
                }

                // Check if we need to move to the next scene
                val nextSceneIndex = proj.scenes.indexOfFirst { it.startTime > currentTime }
                if (nextSceneIndex != -1 && nextSceneIndex != proj.currentSceneIndex) {
                    logger.info("Moving to next scene. Current: ${proj.currentSceneIndex}, Next: $nextSceneIndex")
                    stopVideo(proj.scenes[proj.currentSceneIndex])
                    proj.currentSceneIndex = nextSceneIndex
                    if (proj.playing) {
                        playVideo(proj.scenes[proj.currentSceneIndex])
                    }
                }
            }
        }

        keyboard.keyDown.listen { key ->
            when (key.name) {
                "l" -> {
                    logger.info("Load project key pressed")
                    openFolderDialog { folder ->
                        folder?.let {
                            logger.info("Project folder selected: ${it.absolutePath}")
                            projectFolder = it
                            reloadProject(it)
                        }
                    }
                }
                "space" -> {
                    logger.info("Play/Pause key pressed")
                    project?.let { proj ->
                        proj.playing = !proj.playing
                        if (proj.playing) {
                            logger.info("Starting playback")
                            proj.startTime = System.currentTimeMillis() - (proj.startTime.takeIf { it > 0 } ?: 0L)
                            playVideo(proj.scenes[proj.currentSceneIndex])
                        } else {
                            logger.info("Pausing playback")
                            pauseVideo(proj.scenes[proj.currentSceneIndex])
                        }
                    }
                }
                "r" -> {
                    logger.info("Reload project key pressed")
                    projectFolder?.let {
                        reloadProject(it)
                    } ?: run {
                        logger.warning("No project loaded to reload")
                        message = "No project loaded to reload."
                    }
                }
            }
        }

        ended.listen {
            logger.info("Application ending")
            project?.let {
                logger.info("Releasing all videos and resources")
                VideoManager.releaseAllVideos(it.scenes)
                loadedImages.values.forEach { it.destroy() }
                loadedImages.clear()
            }
        }
    }
}

Relevant part of the Log:

out. 02, 2024 11:00:27 AM MainKt$main$1$2$reloadProject$1$1 invokeSuspend
INFORMAÇÕES: Preloading videos asynchronously
out. 02, 2024 11:00:27 AM MainKt$main$1$2$reloadProject$1$1 invokeSuspend
INFORMAÇÕES: Project reloaded successfully
out. 02, 2024 11:00:28 AM MainKt$main$1$2$2 invoke
INFORMAÇÕES: Play/Pause key pressed
out. 02, 2024 11:00:28 AM MainKt$main$1$2$2 invoke
INFORMAÇÕES: Starting playback
out. 02, 2024 11:00:28 AM utils.VideoManager loadVideo
INFORMAÇÕES: Loading video: 01.mp4
out. 02, 2024 11:00:30 AM utils.VideoManager loadVideo
INFORMAÇÕES: Successfully loaded video: 01.mp4
out. 02, 2024 11:00:30 AM MainKt$main$1$2$1$1$2$1$2 invokeSuspend
INFORMAÇÕES: Starting video from beginning for scene 1
out. 02, 2024 11:00:31 AM MainKt$main$1$2$1 invoke
INFORMAÇÕES: Moving to next scene. Current: 0, Next: 1
out. 02, 2024 11:00:35 AM utils.VideoManager loadVideo
INFORMAÇÕES: Loading video: 02.mp4
out. 02, 2024 11:00:35 AM utils.VideoManager loadVideo
INFORMAÇÕES: Successfully loaded video: 02.mp4
out. 02, 2024 11:00:35 AM MainKt$main$1$2$1$1$2$1$2 invokeSuspend
INFORMAÇÕES: Starting video from beginning for scene 2
out. 02, 2024 11:00:35 AM MainKt$main$1$2$1 invoke
INFORMAÇÕES: Moving to next scene. Current: 1, Next: 2
error? -11
error? -11
error? -11
error? -11
error? -11
error? -11
error? -11
error? -11
error? -11
error? -11
error? -11
error? -11
error? -11
2

Hi @dedos !

I wrote a quick program to test playing multiple videos:

import org.openrndr.application
import org.openrndr.ffmpeg.PlayMode
import org.openrndr.ffmpeg.loadVideo
import java.io.File

fun main() = application {
    program {
        val l = File("/home/funpro/datasea/Videos/abstract/")
        val files = l.listFiles()
        //files?.shuffle()

        val videos = files?.take(5)?.map {
            loadVideo(it.absolutePath, PlayMode.VIDEO)
        }

        videos?.forEachIndexed { i, v ->
            v.ended.listen {
                v.restart()
                println("restart $i -> ${v.duration} seconds long (now: $seconds)")
            }
            v.play()
        }

        extend {
            videos?.forEachIndexed { i, v ->
                v.draw(drawer, i.toDouble() * width / videos.size, 0.0)
            }
        }
    }
}

One can change the number of videos by changing the 5 (there should be that many or more videos in the folder).

By doing this I discovered that the restarting has an issue, and it’s being worked on currently. At least on the GitHub version, which is what I used (I don’t use the official release but build it myself to get all the latest). The CPU usage is low even if I play 10 videos, but when it begins restarting videos it goes up to 100%, which should not happen.

I haven’t tested your code yet, but I see two odd things. Maybe object should be class. And I haven’t used suspend. Don’t know what kind of effect it may have. I would try removing it see what happens.

In my test I don’t see error -11, but it can be because I use a newer version.

I’ll take a look again when I have a moment :slight_smile:

3

Hello again, Abe. Thanks a lot, again, for your quick response and helpfulness.
I have written a sample code based on your example to try to isolate the matter:

import org.openrndr.application
import org.openrndr.ffmpeg.PlayMode
import org.openrndr.ffmpeg.loadVideo
import java.io.File

fun main() = application {
    program {
        val l = File("C:\\videos\\godgang\\play")
        val files = l.listFiles()
        val videos = files?.take(5)?.map {
            loadVideo(it.absolutePath, PlayMode.VIDEO)
        }

        videos?.forEachIndexed { i, v ->
            v.ended.listen {
                v.restart()
                println("restart $i -> ${v.duration} seconds long")
            }
            v.play()
        }

        extend {
            videos?.forEachIndexed { i, v ->
                v.draw(drawer, i.toDouble() * width / videos.size, 0.0)
            }
        }
    }
}

I don’t have the error? -11 code.
Your suspicions seem to be correct, I will make some tests and come back with my findings.

About the restarting part: shouldn’t we be using Loop, as specified here?
https://guide.openrndr.org/drawing/video.html

// Loop: restart when reaching the end
videoPlayer.ended.listen {
    videoPlayer.restart()
}

extend {
    videoPlayer.draw(drawer)
}

I’m new to this so forgive me if I’m asking foolish questions.
Thanks again!

4

Based on your code, I was able to fix the error -11 issue.
Here’s my updated Main.kt file:

import org.openrndr.application
import org.openrndr.color.ColorRGBa
import org.openrndr.draw.loadFont
import org.openrndr.draw.loadImage
import org.openrndr.draw.ColorBuffer
import org.openrndr.dialogs.openFolderDialog
import org.openrndr.ffmpeg.PlayMode
import org.openrndr.ffmpeg.VideoPlayerFFMPEG
import kotlinx.coroutines.*
import models.Project
import models.Scene
import models.VideoState
import renderers.TimelineRenderer
import utils.*
import java.io.File
import java.util.logging.Logger
import java.util.logging.Level

private val logger = Logger.getLogger("MainApplication")

fun main() = application {
    configure {
        width = 960
        height = 540
        title = "ChannelFactory Video Editor"
    }

    program {
        logger.info("Initializing ChannelFactory Video Editor")
        val font = loadFont("data/fonts/default.otf", 16.0)
        var project: Project? = null
        var projectFolder: File? = null
        var message: String = "ChannelFactory Video Editor"
        var timelineRenderer: TimelineRenderer? = null
        val loadedImages = mutableMapOf<File, ColorBuffer>()
        val loadedVideos = mutableMapOf<File, VideoPlayerFFMPEG>()

        fun loadImageIfNeeded(file: File): ColorBuffer? {
            return loadedImages[file] ?: run {
                try {
                    logger.info("Loading image: ${file.name}")
                    val image = loadImage(file)
                    loadedImages[file] = image
                    logger.info("Image loaded successfully: ${file.name}")
                    image
                } catch (e: Exception) {
                    logger.log(Level.WARNING, "Failed to load image: ${file.name}", e)
                    null
                }
            }
        }

        fun loadVideoIfNeeded(file: File): VideoPlayerFFMPEG? {
            return loadedVideos[file] ?: run {
                try {
                    logger.info("Loading video: ${file.name}")
                    val video = VideoPlayerFFMPEG.fromFile(file.absolutePath, PlayMode.VIDEO)
                    video.ended.listen {
                        video.seek(0.0)
                        video.play()
                    }
                    loadedVideos[file] = video
                    logger.info("Video loaded successfully: ${file.name}")
                    video
                } catch (e: Exception) {
                    logger.log(Level.WARNING, "Failed to load video: ${file.name}", e)
                    null
                }
            }
        }

        fun reloadProject(projectFolder: File) {
            logger.info("Reloading project from folder: ${projectFolder.absolutePath}")
            val loadedProject = loadProject(projectFolder)
            if (loadedProject != null) {
                project = loadedProject
                timelineRenderer = TimelineRenderer(loadedProject.scenes)
                message = "Project reloaded successfully!"
                loadedImages.clear()
                loadedVideos.values.forEach { it.dispose() }
                loadedVideos.clear()

                logger.info("Preloading videos")
                loadedProject.scenes.forEach { scene ->
                    scene.videoFile?.let { file ->
                        loadVideoIfNeeded(file)
                    }
                }
                logger.info("Project reloaded successfully")
            } else {
                logger.warning("Failed to reload project. Invalid folder selected.")
                message = "Failed to reload project. Please select a valid folder."
            }
        }

        extend {
            val currentTime = project?.let { proj ->
                if (proj.playing) System.currentTimeMillis() - proj.startTime else proj.startTime
            } ?: 0L

            drawer.clear(ColorRGBa.BLACK)
            drawer.fill = ColorRGBa.WHITE
            drawer.fontMap = font

            drawer.text(message, 20.0, 30.0)
            drawer.text("Press 'L' to load a project", 20.0, 60.0)
            drawer.text("Press 'Space' to play/pause", 20.0, 90.0)
            drawer.text("Press 'R' to stop", 20.0, 120.0)

            project?.let { proj ->
                drawer.text("Total Scenes: ${proj.scenes.size}", 20.0, 150.0)
                proj.scenes.forEachIndexed { index, scene ->
                    val minutes = scene.startTime / 60000
                    val seconds = (scene.startTime % 60000) / 1000
                    val firstWords = scene.text.split(" ").take(5).joinToString(" ") + "..."
                    drawer.text("Scene ${index + 1}: ${minutes}m ${seconds}s - $firstWords", 20.0, 180.0 + index * 20.0)
                }

                timelineRenderer?.draw(drawer, width.toDouble(), height.toDouble(), mouse.position)

                val activeScene = proj.scenes.findLast { it.startTime <= currentTime }
                activeScene?.let { scene ->
                    logger.fine("Processing active scene ${scene.number}")
                    proj.sceneMediaMap[scene.number]?.forEach { file ->
                        when (file.extension.lowercase()) {
                            in listOf("jpg", "jpeg", "png", "bmp") -> {
                                loadImageIfNeeded(file)?.let { image ->
                                    val imageScale = minOf(width.toDouble() / image.width, height.toDouble() / image.height)
                                    val imageWidth = image.width * imageScale
                                    val imageHeight = image.height * imageScale
                                    val imageX = (width - imageWidth) / 2.0
                                    val imageY = (height - imageHeight) / 2.0
                                    drawer.image(image, imageX, imageY, imageWidth, imageHeight)
                                }
                            }
                            in listOf("mp4", "avi", "mov") -> {
                                loadVideoIfNeeded(file)?.let { video ->
                                    when {
                                        proj.playing && scene.videoState != VideoState.PLAYING -> {
                                            logger.info("Starting video from beginning for scene ${scene.number}")
                                            video.seek(0.0)
                                            video.play()
                                            scene.videoState = VideoState.PLAYING
                                        }
                                        !proj.playing && scene.videoState == VideoState.PLAYING -> {
                                            logger.info("Pausing playing video for scene ${scene.number}")
                                            video.pause()
                                            scene.videoState = VideoState.PAUSED
                                        }
                                    }
                                    video.draw(drawer, 0.0, 0.0, width.toDouble(), height.toDouble())
                                }
                            }
                        }
                    }
                }

                val nextSceneIndex = proj.scenes.indexOfFirst { it.startTime > currentTime }
                if (nextSceneIndex != -1 && nextSceneIndex != proj.currentSceneIndex) {
                    logger.info("Moving to next scene. Current: ${proj.currentSceneIndex}, Next: $nextSceneIndex")
                    stopVideo(proj.scenes[proj.currentSceneIndex])
                    proj.currentSceneIndex = nextSceneIndex
                    if (proj.playing) {
                        playVideo(proj.scenes[proj.currentSceneIndex])
                    }
                }
            }
        }

        keyboard.keyDown.listen { key ->
            when (key.name) {
                "l" -> {
                    logger.info("Load project key pressed")
                    openFolderDialog { folder ->
                        folder?.let {
                            logger.info("Project folder selected: ${it.absolutePath}")
                            projectFolder = it
                            reloadProject(it)
                        }
                    }
                }
                "space" -> {
                    logger.info("Play/Pause key pressed")
                    project?.let { proj ->
                        proj.playing = !proj.playing
                        if (proj.playing) {
                            logger.info("Starting playback")
                            proj.startTime = System.currentTimeMillis() - (proj.startTime.takeIf { it > 0 } ?: 0L)
                            playVideo(proj.scenes[proj.currentSceneIndex])
                        } else {
                            logger.info("Pausing playback")
                            pauseVideo(proj.scenes[proj.currentSceneIndex])
                        }
                    }
                }
                "r" -> {
                    logger.info("Reload project key pressed")
                    projectFolder?.let {
                        reloadProject(it)
                    } ?: run {
                        logger.warning("No project loaded to reload")
                        message = "No project loaded to reload."
                    }
                }
            }
        }

        ended.listen {
            logger.info("Application ending")
            project?.let {
                logger.info("Releasing all videos and resources")
                loadedVideos.values.forEach { it.dispose() }
                loadedVideos.clear()
                loadedImages.values.forEach { it.destroy() }
                loadedImages.clear()
            }
        }
    }
}

I don’t know what drives you to be so kind and helpful, but I’m grateful. Thanks again!

1 Like
5

Nice that you’re making progress! :slight_smile:

Haha I wrote that looping part of the guide myself :laughing:

If you look at my example code, that’s what I wrote there too:

videos?.forEachIndexed { i, v ->
    v.ended.listen {
        v.restart()
        println("restart $i -> ${v.duration} seconds long")
    }
    v.play()
}

In English: for each video, listen to its end event, and then restart it and print some debug info. Also, play each video.

The issue I’m having with restarting may have been introduced in a newer version of the frameworks. Looking at it.

6

Do you know exactly what changed fixed it?

Thank you for your nice words :pray:

7

It has something to do with suspend.
Based on your example, I switched to a simpler rendering solution without coroutines and that stuff I was trying to do. Then it worked.

Now I am almost where I want to be, just struggling with another kind of error, LOL. But I won’t keep bugging you here, you have already helped me too much and I don’t want to take more of your time.

I’m trying lots of different approaches, if I really can’t solve it I’ll come back here but I hope it won’t be necessary

1 Like
8

I literally took your solution, pasted it into Claude and said “look. the guy who wrote this is a genius. you should follow his example to fix this shit” and then it took me to the right direction, LOL

1 Like