# Video mapping 2d

**URL:** <https://openrndr.discourse.group/t/video-mapping-2d/720>\
**Category:** How to?\
**Created:** [December 15, 2024, 5:01pm UTC](https://openrndr.discourse.group/t/video-mapping-2d/720 "2024-12-15T17:01:38Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![supernihil](https://yyz2.discourse-cdn.com/free1/user_avatar/openrndr.discourse.group/supernihil/32/764_2.png) [@supernihil](https://openrndr.discourse.group/u/supernihil)\
**Post date:** [December 15, 2024, 5:01pm UTC](https://openrndr.discourse.group/t/video-mapping-2d/720/1 "2024-12-15T17:01:38Z")

</div>

hi i am trying to figure out how to do the following:

1. a video is played using ffmpeg and into a render target (A)
2. four coordinates (Q) describe a polygon inside the render target (A)
3. four other coordinates (E) describe a polygon inside an “output render target” (B)
4. use the Q cords to extract texture data from (A) and use it as texture inside polygon (E) on render target (B)

or in other terms : how do i immitate madmapper in order to have input textures, texture polygons, surface polygons and map from texture cords to surface cords?

---

<div class="post-metadata">

**Author:** ![supernihil](https://yyz2.discourse-cdn.com/free1/user_avatar/openrndr.discourse.group/supernihil/32/764_2.png) [@supernihil](https://openrndr.discourse.group/u/supernihil)\
**Post date:** [December 15, 2024, 5:26pm UTC](https://openrndr.discourse.group/t/video-mapping-2d/720/2 "2024-12-15T17:26:25Z")

</div>

here is an example taken from ofxpimapper

> <https://github.com/kr15h/ofxPiMapper/blob/master/src/Surfaces/TriangleSurface.cpp#L18>

my problem is i am new to glsl and cant figure out how to specify the texture coordinates and the surface coordinates inside openrndr, as i like terminology understanding (tried hard but doesnt stick)

thinking about how to have a texture plus 4 coordinates each a corner in a rectangular like polygon and then use that info as “texcoords” to tell where to get tex data for another polygon

---

<div class="post-metadata">

**Author:** ![supernihil](https://yyz2.discourse-cdn.com/free1/user_avatar/openrndr.discourse.group/supernihil/32/764_2.png) [@supernihil](https://openrndr.discourse.group/u/supernihil)\
**Post date:** [December 16, 2024, 6:43am UTC](https://openrndr.discourse.group/t/video-mapping-2d/720/3 "2024-12-16T06:43:38Z")

</div>

i now have learned that vertexbuffers contain the coordinates of the shape as well as the normalized coordinates of the texture (where in the texture is each shape coordinate) so i guess i need to find a way to edit those vectors seperately .

my first goal is to be able to have a texture and a triangle shape and be able to edit the texture coordinates as well as the shape coordinates seperately with the mouse. and see the textured shape being rendered in the output

---

<div class="post-metadata">

**Author:** ![abe](https://yyz2.discourse-cdn.com/free1/user_avatar/openrndr.discourse.group/abe/32/700_2.png) [@abe](https://openrndr.discourse.group/u/abe)\
**Post date:** [December 16, 2024, 8:30am UTC](https://openrndr.discourse.group/t/video-mapping-2d/720/4 "2024-12-16T08:30:07Z")

</div>

It sounds like you’re on the right path 🙂

I quickly wrote this, maybe it helps. The `write` method expects 4 screen coordinates and 4 texture coordinates. The deformation looks odd because the quad is made out of two triangles. But at least it should show the components you might need.

```kotlin
import org.openrndr.application
import org.openrndr.draw.*
import org.openrndr.math.Polar
import org.openrndr.math.Vector2
import org.openrndr.math.Vector3
import kotlin.math.sin

class QuadWriter(val tex: ColorBuffer) {
    private val vf = vertexFormat {
        position(3)
        textureCoordinate(2)
    }
    private val quad = vertexBuffer(vf, 6)
    private val shader = shadeStyle {
        fragmentTransform = "x_fill = texture(p_tex, va_texCoord0.xy);"
        parameter("tex", tex)
    }

    fun write(positions: List<Vector3>, textureCoords: List<Vector2>) {
        quad.put {
            write(positions[0])
            write(textureCoords[0])
            write(positions[1])
            write(textureCoords[1])
            write(positions[3])
            write(textureCoords[3])

            write(positions[1])
            write(textureCoords[1])
            write(positions[2])
            write(textureCoords[2])
            write(positions[3])
            write(textureCoords[3])
        }
    }

    fun draw(drawer: Drawer) {
        drawer.shadeStyle = shader
        drawer.vertexBuffer(quad, DrawPrimitive.TRIANGLES)
    }
}

fun main() = application {
    program {
        val tex = loadImage("data/images/pm5544.png")
        val quadWriter = QuadWriter(tex)

        extend {
            val center = drawer.bounds.center
            quadWriter.write(
                List(4) {
                    (center + Polar(it * 90.0, sin(seconds + it) * 100.0 + 150.0).cartesian).xy0
                }, listOf(
                    Vector2(0.1, 0.1),
                    Vector2(0.3, 0.2),
                    Vector2(0.4, 0.6),
                    Vector2(0.2, 0.5)
                )
            )
            quadWriter.draw(drawer)
        }
    }
}

```

ps. The quad coordinates are clockwise.

---

<div class="post-metadata">

**Author:** ![supernihil](https://yyz2.discourse-cdn.com/free1/user_avatar/openrndr.discourse.group/supernihil/32/764_2.png) [@supernihil](https://openrndr.discourse.group/u/supernihil)\
**Post date:** [December 16, 2024, 8:55am UTC](https://openrndr.discourse.group/t/video-mapping-2d/720/5 "2024-12-16T08:55:16Z")

</div>

thank you so much! i am not great at math and since my departure from software development (due to stress) i am having a hard time grasping new concepts 🙂 so it means a lot to get some code i can fuzz about with until i understand

---

<div class="post-metadata">

**Author:** ![supernihil](https://yyz2.discourse-cdn.com/free1/user_avatar/openrndr.discourse.group/supernihil/32/764_2.png) [@supernihil](https://openrndr.discourse.group/u/supernihil)\
**Post date:** [December 16, 2024, 6:12pm UTC](https://openrndr.discourse.group/t/video-mapping-2d/720/6 "2024-12-16T18:12:51Z")

</div>

i have tried to read up on the issue of the texture being warped down the diagonal but the closest i’ve come to a definition of the issue is this stack overflow question: [Quad texture stretching on OpenGL - Stack Overflow](https://stackoverflow.com/questions/27204865/quad-texture-stretching-on-opengl)  
talking about the interpolation of each of the two triangles in the mesh being rendered on their own. and therefore you get skewed quad texture

can anybody send me in a direction of getting the quad texture to be “not warped down the diagonal”

---

<div class="post-metadata">

**Author:** ![supernihil](https://yyz2.discourse-cdn.com/free1/user_avatar/openrndr.discourse.group/supernihil/32/764_2.png) [@supernihil](https://openrndr.discourse.group/u/supernihil)\
**Post date:** [December 16, 2024, 6:15pm UTC](https://openrndr.discourse.group/t/video-mapping-2d/720/7 "2024-12-16T18:15:44Z")

</div>

this article also talks about the issue, reading it now! 🙂

> **[Quadrilateral Interpolation, Part 1 – Nathan Reed’s coding blog](https://www.reedbeta.com/blog/quadrilateral-interpolation-part-1/)**
>
> Pixels and polygons and shaders, oh my!

---

<div class="post-metadata">

**Author:** ![supernihil](https://yyz2.discourse-cdn.com/free1/user_avatar/openrndr.discourse.group/supernihil/32/764_2.png) [@supernihil](https://openrndr.discourse.group/u/supernihil)\
**Post date:** [December 16, 2024, 6:57pm UTC](https://openrndr.discourse.group/t/video-mapping-2d/720/8 "2024-12-16T18:57:29Z")

</div>

this repo is also talking about the “nonaffine” transformations needed to get a proper non-affine quad texture going

> **[GitHub - v0idv0id/PerspectiveTexture: This example shows how to render a perspective...](https://github.com/v0idv0id/PerspectiveTexture?tab=readme-ov-file#notes)**
>
> This example shows how to render a perspective texture on a quad in only 2-dimensions.

> <https://github.com/v0idv0id/PerspectiveTexture/blob/master/PerspectiveTexture.cpp#L263>

---

<div class="post-metadata">

**Author:** ![abe](https://yyz2.discourse-cdn.com/free1/user_avatar/openrndr.discourse.group/abe/32/700_2.png) [@abe](https://openrndr.discourse.group/u/abe)\
**Post date:** [December 16, 2024, 9:01pm UTC](https://openrndr.discourse.group/t/video-mapping-2d/720/9 "2024-12-16T21:01:59Z")

</div>

Very interesting issue 🙂

There’s also a part 2:

> **[Quadrilateral Interpolation, Part 2 – Nathan Reed’s coding blog](https://www.reedbeta.com/blog/quadrilateral-interpolation-part-2/)**
>
> Pixels and polygons and shaders, oh my!

---

<div class="post-metadata">

**Author:** ![supernihil](https://yyz2.discourse-cdn.com/free1/user_avatar/openrndr.discourse.group/supernihil/32/764_2.png) [@supernihil](https://openrndr.discourse.group/u/supernihil)\
**Post date:** [December 17, 2024, 5:31am UTC](https://openrndr.discourse.group/t/video-mapping-2d/720/10 "2024-12-17T05:31:30Z")

</div>

I have learned that a cheap way to get around the issue is to have more triangles. The subject is called tesselation. And apparently can be done in a shader

---

<div class="post-metadata">

**Author:** ![abe](https://yyz2.discourse-cdn.com/free1/user_avatar/openrndr.discourse.group/abe/32/700_2.png) [@abe](https://openrndr.discourse.group/u/abe)\
**Post date:** [December 17, 2024, 7:49am UTC](https://openrndr.discourse.group/t/video-mapping-2d/720/11 "2024-12-17T07:49:56Z")

</div>

That gave me an idea: [orx/orx-shapes at master · openrndr/orx · GitHub](https://github.com/openrndr/orx/tree/master/orx-shapes) has a bezier patch that could be used for this. Unfortunately I discovered issues [350](https://github.com/openrndr/orx/issues/350) and [351](https://github.com/openrndr/orx/issues/351). Once they are fixed I think that might be an easy way to map irregular (even curved) shapes.

Mmm… would it be enough for you if only the target is irregular, but the source is rectangular? Or do you want both to be irregular?

Meanwhile, I think it would be simple to create a 3D Plane generator with configurable x and y resolution and uv texture coords, which can be used in 2D (and then add it to orx-shapes).

---

<div class="post-metadata">

**Author:** ![supernihil](https://yyz2.discourse-cdn.com/free1/user_avatar/openrndr.discourse.group/supernihil/32/764_2.png) [@supernihil](https://openrndr.discourse.group/u/supernihil)\
**Post date:** [December 17, 2024, 9:42am UTC](https://openrndr.discourse.group/t/video-mapping-2d/720/12 "2024-12-17T09:42:45Z")

</div>

feels like i’m nearly there. my solution (albeit rookie) is to calculate interpolated points first on the vertical sides of the quad and then from between those calculate and interpolate the horizontal points.  
i do it twice, once for position and once for texture

```auto
import org.openrndr.application
import org.openrndr.draw.*
import org.openrndr.ffmpeg.VideoPlayerConfiguration
import org.openrndr.ffmpeg.VideoPlayerFFMPEG
import org.openrndr.math.Polar
import org.openrndr.math.Vector2
import org.openrndr.math.Vector3
import kotlin.math.sin
import org.openrndr.ffmpeg.PlayMode
import org.openrndr.math.mix

class QuadWriter(val vid:VideoPlayerFFMPEG) {
    private val vf = vertexFormat {
        position(3)
        textureCoordinate(2)
    }
    private val segments =8
    private val quad = vertexBuffer(vf, 6*segments*segments)

    fun write(positions: List<Vector3>, textureCoords: List<Vector2>) {
        // b
        // 1 -- 2
        // | / |
        // a | / | c
        // | / |
        // 0 -- 3
        // d
        quad.put {
            for (v in 0 until segments) {
                for (u in 0 until segments) {

                    // vertical first

                    val p0y = mix(positions[0], positions[1], (1.0 * v) / segments)
                    val p3y = mix(positions[3], positions[2], (1.0 * v) / segments)
                    val p1y = mix(positions[0], positions[1], (1.0 * (v+1)) / segments)
                    val p2y = mix(positions[3], positions[2], (1.0 * (v+1)) / segments)

                    val p0 = mix(p0y, p3y, (1.0 * u) / segments)
                    val p3 = mix(p0y, p3y, (1.0 * (u+1)) / segments)
                    val p1 = mix(p1y, p2y, (1.0 * u) / segments)
                    val p2 = mix(p0y, p3y, (1.0 * (u+1)) / segments)

                    val t0y = mix(textureCoords[0], textureCoords[1], (1.0 * v) / segments)
                    val t3y = mix(textureCoords[3], textureCoords[2], (1.0 * v) / segments)
                    val t1y = mix(textureCoords[0], textureCoords[1], (1.0 * (v+1)) / segments)
                    val t2y = mix(textureCoords[3], textureCoords[2], (1.0 * (v+1)) / segments)

                    val t0 = mix(t0y, t3y, (1.0 * u) / segments)
                    val t3 = mix(t0y, t3y, (1.0 * (u+1)) / segments)
                    val t1 = mix(t1y, t2y, (1.0 * u) / segments)
                    val t2 = mix(t0y, t3y, (1.0 * (u+1)) / segments)

                    write(p2)
                    write(t2)
                    write(p1)
                    write(t1)
                    write(p0)
                    write(t0)

                    write(p0)
                    write(t0)
                    write(p3)
                    write(t3)
                    write(p2)
                    write(t2)

                }
            }
        }

    }

    fun draw(colorBuffer: ColorBuffer, drawer: Drawer) {
            drawer.shadeStyle = shadeStyle {
                fragmentTransform = "x_fill = texture(p_tex, va_texCoord0.xy);"
                parameter("tex", colorBuffer)
            }
            drawer.vertexBuffer(quad, DrawPrimitive.TRIANGLES)

    }
}

fun main() = application {
    program {
        val videoPlayer = VideoPlayerFFMPEG.fromFile("data/video2.mov",
            PlayMode.VIDEO,
            VideoPlayerConfiguration().apply {
                useHardwareDecoding = false
                videoFrameQueueSize = 500
                displayQueueCooldown = 5
                //synchronizeToClock = false
            })
        videoPlayer.play()
        videoPlayer.ended.listen {
            videoPlayer.restart()
        }
        val renderTarget = renderTarget(width, height) {
            colorBuffer()
        }
        val quadWriter = QuadWriter(videoPlayer)

        extend {
            drawer.withTarget(renderTarget) {
                videoPlayer.draw(drawer)
            }
            val center = drawer.bounds.center
            quadWriter.write(
                listOf(
                    Vector3(50.0, 450.0,0.0),
                    Vector3(50.0,100.0 ,0.0),
                    Vector3(550.0, 50.0,0.0),
                    Vector3(300.0, 400.0,0.0)
                ), listOf(
                    Vector2(0.0, 0.0),
                    Vector2(0.0, 1.0),
                    Vector2(1.0, 1.0),
                    Vector2(1.0, 0.0)
                )
            )
            quadWriter.draw(renderTarget.colorBuffer(0), drawer)
        }
    }
}

```

 ![Screenshot from 2024-12-17 10-40-13](https://global.discourse-cdn.com/free1/uploads/openrndr/original/1X/1763a877dfb09d1ac02fcfd3f5998af298be9ed5.png)

now i just need to figure out why its only half of my triangles that are showing…?

any ideas @abe ?

---

<div class="post-metadata">

**Author:** ![supernihil](https://yyz2.discourse-cdn.com/free1/user_avatar/openrndr.discourse.group/supernihil/32/764_2.png) [@supernihil](https://openrndr.discourse.group/u/supernihil)\
**Post date:** [December 17, 2024, 10:07am UTC](https://openrndr.discourse.group/t/video-mapping-2d/720/13 "2024-12-17T10:07:29Z")

</div>

figured it out 🙂  
here is the diff gist:

> <https://gist.github.com/sloev/a7492af4ccecf7b99f552da0fff660cc>

 ![Screenshot from 2024-12-17 11-06-23](https://global.discourse-cdn.com/free1/uploads/openrndr/original/1X/e6fb8352518c1d5eb4c9a2d08f615bd7ea819267.png)

---

<div class="post-metadata">

**Author:** ![abe](https://yyz2.discourse-cdn.com/free1/user_avatar/openrndr.discourse.group/abe/32/700_2.png) [@abe](https://openrndr.discourse.group/u/abe)\
**Post date:** [December 17, 2024, 10:20am UTC](https://openrndr.discourse.group/t/video-mapping-2d/720/14 "2024-12-17T10:20:06Z")

</div>

Nice!! Congratulations 🎉

About performance: if you are not animating the target shape, you can move the `quadWriter.write()` out of `extend` so it only calculates the `vertexBuffer` once (maybe useful if you draw hundreds of such shapes 🙂 )

Is this for doing projection mapping with a projector?

---

<div class="post-metadata">

**Author:** ![supernihil](https://yyz2.discourse-cdn.com/free1/user_avatar/openrndr.discourse.group/supernihil/32/764_2.png) [@supernihil](https://openrndr.discourse.group/u/supernihil)\
**Post date:** [December 17, 2024, 10:29am UTC](https://openrndr.discourse.group/t/video-mapping-2d/720/15 "2024-12-17T10:29:31Z")

</div>

yup, yeah it only needs to be calculated when either the texture quad or the surface quad changes its cords.

yeah i am making a projection mapping vj app for linux:

- 4 video channels
- 1 output via udp broadcast to mpv (video player clients) each taking a quadrant of the udp video (think videosplitter)
- a mapping editor where you create texture quads and surface quads.

basically like madmapper but with udp network output instead

on that note @abe is there an example of a gui creating objects that then can be selected in a drop down?

i need some way to create new texture/ surface quad pairs and be able to delete them as well as edit their corners with a mouse

---

<div class="post-metadata">

**Author:** ![supernihil](https://yyz2.discourse-cdn.com/free1/user_avatar/openrndr.discourse.group/supernihil/32/764_2.png) [@supernihil](https://openrndr.discourse.group/u/supernihil)\
**Post date:** [December 17, 2024, 10:47am UTC](https://openrndr.discourse.group/t/video-mapping-2d/720/16 "2024-12-17T10:47:31Z")

</div>

i can see that you can create data driven ui’s like this here:

> [@How to keep .kt files not too long](https://openrndr.discourse.group/t/how-to-keep-kt-files-not-too-long/55/7):
>
> We recently developed tools for orx-panel to simplify the creation of data-driven UIs. The tools are called WatchListDiv, WatchPropertyDiv and WatchObjectDiv and fit in nicely with the state class example I demonstrated before. A demo for WatchListDiv including the loading and saving of state to json files: A demo for WatchObjectDiv:

---

<div class="post-metadata">

**Author:** ![abe](https://yyz2.discourse-cdn.com/free1/user_avatar/openrndr.discourse.group/abe/32/700_2.png) [@abe](https://openrndr.discourse.group/u/abe)\
**Post date:** [December 17, 2024, 12:31pm UTC](https://openrndr.discourse.group/t/video-mapping-2d/720/17 "2024-12-17T12:31:54Z")

</div>

`orx-gui` has a dropdown, but it uses an `enum` and I don’t think elements can be added / removed. Maybe we should add one based on `mutableList`…

---

<div class="post-metadata">

**Author:** ![supernihil](https://yyz2.discourse-cdn.com/free1/user_avatar/openrndr.discourse.group/supernihil/32/764_2.png) [@supernihil](https://openrndr.discourse.group/u/supernihil)\
**Post date:** [December 28, 2024, 7:17pm UTC](https://openrndr.discourse.group/t/video-mapping-2d/720/18 "2024-12-28T19:17:08Z")

</div>

code: [init · sloev/fremkalder@413d522 · GitHub](https://github.com/sloev/fremkalder/commit/413d5227d7db4b884dfce7c0a94fac14800d2a5a)

@abe i have now made some progress. but i am lost as to make the whole “edit surface points” thing going correctly as can be seen in the video, do you maybe have some ideas?

---

<div class="post-metadata">

**Author:** ![abe](https://yyz2.discourse-cdn.com/free1/user_avatar/openrndr.discourse.group/abe/32/700_2.png) [@abe](https://openrndr.discourse.group/u/abe)\
**Post date:** [December 29, 2024, 12:05pm UTC](https://openrndr.discourse.group/t/video-mapping-2d/720/19 "2024-12-29T12:05:16Z")

</div>

Nice progress 🙂

By looking at the video it seems like the bottom one is working correctly, but the top one is inverted vertically.

If I do

```kotlin
...
val t0 = mix(t0y, t3y, (1.0 * u) / segments).vFlip()
val t3 = mix(t0y, t3y, (1.0 * (u + 1)) / segments).vFlip()
val t1 = mix(t1y, t2y, (1.0 * u) / segments).vFlip()
val t2 = mix(t1y, t2y, (1.0 * (u + 1)) / segments).vFlip()
...

private fun Vector2.vFlip() = Vector2(x, 1 - y)

```

or

```kotlin
            fragmentTransform = """
                vec2 c = va_texCoord0.xy;
                x_fill = texture(p_tex, vec2(c.x, 1.0 - c.y));
            """.trimIndent()

```

Then the image on the bottom looks as expected, but flipped vertically.  
It’s almost there but not sure how to fix this right now.

I played a bit with the code. Most of it just simplifying things, but I think one should not create many vertexBuffers but reuse them. Also I believe they should be destroyed when no longer needed.

[Surface.kt](https://openrndr.discourse.group/uploads/short-url/7LabmyENBxCybdex56In3RMabxf.kt) (3.2 KB)  
[Surfaces.kt](https://openrndr.discourse.group/uploads/short-url/gFq0UeoSYyrJWESWjwWRP7kWxVB.kt) (3.4 KB)  
[TemplateProgram.kt](https://openrndr.discourse.group/uploads/short-url/sDsF8aogfdAHDli0jUVqw78cPf8.kt) (7.3 KB)

---

<div class="post-metadata">

**Author:** ![supernihil](https://yyz2.discourse-cdn.com/free1/user_avatar/openrndr.discourse.group/supernihil/32/764_2.png) [@supernihil](https://openrndr.discourse.group/u/supernihil)\
**Post date:** [December 29, 2024, 5:11pm UTC](https://openrndr.discourse.group/t/video-mapping-2d/720/20 "2024-12-29T17:11:05Z")

</div>

thank you so much. @abe  
i’ve only written kotlin since i wrote on this forum some weeks ago so i’m still getting my hands dirty on structuring the codebase plus how oop works in kotlin. so your splitting up, cleaning and reorging of my code is so appreciated.  
i’ve added your files with a link back to here in this commit: [add files from abe :-) see: https://openrndr.discourse.group/t/video-… · sloev/fremkalder@4df47f2 · GitHub](https://github.com/sloev/fremkalder/commit/4df47f2c9432877a93137b0787e601cd2655c87e)

[Next page](https://openrndr.discourse.group/t/video-mapping-2d/720.md?page=2)