OPENRNDR and sound (Minim, Beads and TarsosDSP libraries)

OPENRNDR Tutorials
1

It won’t be a question, at least not asked by me. My friend asked me during the OPENRNDR workshop how to use sound with OPENRNDR. and then she immediately googled it. There was no answer over the Internet. So I will write how I do it in my projects with Minim library.

Despite what you might think Minim is not bound directly to Processing. It can be integrated with any java derived code. In case of OPENRNDR you need to add this dependency to build.gradle.kts:

dependencies {
    // ...
    compile("net.compartmental.code:minim:2.2.2") {
        exclude(group = "org.apache.maven.plugins", module = "maven-javadoc-plugin")
    }
}

And then something like this will do the trick:

  program {
    val minim = Minim(object : Object() {
      fun sketchPath(fileName: String): String {
        return fileName
      }
      fun createInput(fileName: String): InputStream {
        return FileInputStream(File(fileName))
      }
    })
    val lineIn = minim.lineIn
    val fft = FFT(lineIn.bufferSize(), lineIn.sampleRate())
    //...

The sketchPath and createInput functions should reflect structure of music resources in your project.

1 Like
2

Thanks for the example :slight_smile:

The beads audio library is also not Processing dependent, and they seem to be migrating to gradle:

I haven’t tried it recently, but I used it in the past.

3

Thanks for sharing @kazik, Beads doesn’t have so many examples out there (not that I’ve found). Minim seems like an easier to use alternative.

For anyone wanting to use beads just add the following to build.gradle.kts:

    repositories {
       maven( url = "https://jitpack.io")
    }

    dependencies {
      compile("com.github.orsjb", "beads", "migrate_to_gradle-SNAPSHOT")
    }

If you want to pair it with Open-AL:

    class OpenALIO : AudioIO() {
        private var aqs: AudioSource? = null
        val source: AudioSource
            get() {
                return aqs?: error("not started")
            }
        override fun start(): Boolean {
            println("starting OpenALIO")
            aqs = AudioSystem.createQueueSource(queueSize = 2) {
                this.update()
                val bs = context.bufferSize
                val buffer1 = context.out.getOutBuffer(0)
                val bb = ByteBuffer.allocateDirect(bs * 2)
                bb.order(ByteOrder.nativeOrder())
                for (i in 0 until bs) {
                    bb.putShort((buffer1[i].coerceIn(-1.0f, 1.0f) * 32767).toShort())
                }
                bb.rewind()
                AudioData(AudioFormat.MONO_16, 48000, bb)
            }
            (aqs as AudioQueueSource).play()
            return true
        }
        override fun getAudioInput(p0: IntArray?): UGen {
            TODO("not implemented") //To change body of created functions use File | Settings | File Templates.
        }
    }

and use it with:

    val context: AudioContext = AudioContext(
            OpenALIO(),
            48000/60,
            IOAudioFormat(48000.0f, 16, 0, 2)
        )

PS: This is courtesy of @edwin… just sharing the solution.

4

I didn’t realize Minim has a synthesis framework too. That was mostly my reason for going with Beads, plus that it was easy to drop-in an audio back-end. I used OpenAL to achieve positional audio with a 4 speaker setup.

5

Just want to mention I had good experience using TarsosDSP library for Pitch Estimation (tho it does other sound stuff too).

I downloaded the TarsosDSP-2.4.jar from releases and put it in src/lib.

The you need to add it to dependencies in build.gradle.kts

    dependencies {
      …
      implementation(files("$projectDir/src/lib/TarsosDSP-2.4.jar"))
      …
    }

I didn’t get it to work until I also added `java-library` to plugins section:

    plugins {
        …
        `java-library`
        …
    }

Finally here’s a simple example using pitch estimation

Imports 
    import org.openrndr.application
    import be.tarsos.dsp.io.jvm.AudioDispatcherFactory
    import be.tarsos.dsp.pitch.PitchProcessor
    import be.tarsos.dsp.pitch.PitchProcessor.PitchEstimationAlgorithm
    import kotlin.concurrent.thread

    fun main() = application {
        program {
            var pitch = -1f

            // Based on https://stackoverflow.com/questions/31231813/tarsosdsp-pitch-analysis-for-dummies

            // On separate thread to not block drawing process
            // Use daemon thread to allow the app to quit when main thread finishes
            thread(isDaemon = true) {
                val bufferSize = 4080
                val audioDispatcher = AudioDispatcherFactory.fromDefaultMicrophone(bufferSize, 0)
                val pitchProcessor = PitchProcessor(PitchEstimationAlgorithm.YIN, 44100f, bufferSize) {
                    detectionResult, _ -> pitch = detectionResult.pitch
                }
                audioDispatcher.addAudioProcessor(pitchProcessor)
                audioDispatcher.run()
            }

            extend {
                // -1f means no tone detected
                if (pitch != -1f) {
                    val x = (pitch / 1000.0) * width
                    drawer.rectangle(x, 0.0, 5.0, height.toDouble())
                }
            }
        }
    }

I gotta say that you can just easily use Java libraries (of which there are a lot!) yet essentially write nice Kotlin is a really nice feature of OPENRNDR.

It would maybe it’d be nice to have something about how to add JARs in the guide? I used way more time simply trying to get the JAR to work than I did actually using the library.

PS: TarsosDSP is GPL which might affect your project license.

2 Likes
6

Cool and thank you for sharing your findings!

The guide tapers off quickly at the point where other documentation exist. I see your point though, I think at least pointing towards other texts to read when dealing with these kind of problems would definitely help.

7

Thanks for sharing! I think it might be nice to convert your reply into an independent post, maybe titled “how to use .jar libraries in OPENRNDR”. Later we could even do a sticky post listing useful posts like that one, a bit like a “user made, easy to update and discuss” guide.

1 Like