We will all have fun with the following:
- Learning the basics of various environments for graphics programming and thinking about the strategies they use for syncing with audio. This will include shader programming and ray marching.
- Watching videos / discussing strategies for how to “best” combine audio and visuals together. We will look at classic examples from the field of visual music and more recent contemporary examples.
Charlie Roberts – [email protected]
We’ll start by taking quick dive into shader programming using The Force by Shawn Lawson. In addition to introducing us to how shaders work (shaders will be the basis for some of the other applications we’ll use) we’ll get to play around with FFT analysis on the GPU.
Then we’ll watch some videos of live-coding shader performances to get an idea of what’s possible using this medium, and think about how performers consider synchronization. We’ll also look at some earlier audiovisual classics and consider how they approach the combination of these mediums.
After this we’ll look at Olivia Jack’s Hydra environment, which will let us live code graphics using JavaScript. Hydra uses the metaphor of analog video synthesis, so we’ll watch a couple of visual music pieces that use analog video synthesis before playing with it.
Then we’ll look at ray marching using marching.js, and discuss how the demoscene community uses sync in their audiovisual practice. This will lead into some work with GIbber, which features some novel affordances for audiovisual mapping that we’ll discuss.
By the end everyone should have a nice introduction into some fun tools for exploring this topic further, and hopefully participated in some interesting discussions!
Our first shader in GLSL:
// we have to define a main function (like C)
void main () {
// gl_FragColor determines our final color output
// it is a list of four numbers (vec4) with r,g,b,a values
gl_FragColor = vec4( 1., 0., 0., 1. );
}It is important to note that most numbers are floating point and must have a decimal point to indicate this. It is really easy to forget this, especially coming from JavaScript, so be careful!
The Force comes pre-programmed with special uniforms (a uniform is a value transferred from the CPU to the GPU). One such value is time. We can use this to make a simple sine wave:
void main () {
float frequency = 4.;
float color = sin( time * frequency );
gl_FragColor = vec4( color,color,color, 1. );
}However, you’ll notice that this spends a half of each cycle as black, because the sin() function returns values between -1 – 1. We can provide a bias and a scalar to get this to a range of 0–1:
void main () {
float frequency = 4., bias = .5, gain = .5;
float color = bias + sin( time * frequency ) * gain;
gl_FragColor = vec4( color,color,color, 1. );
}The Force comes with a function uv() that returns the x and y coordinates of the current pixel being calculated. We can use this to make a classic video oscillator on our screen.
void main () {
vec2 p = uv();
float frequency = 10.;
float color = sin( p.x * frequency + time );
gl_FragColor = vec4( color,0.,0., 1. );
}… and now’s where things start to get fun. We can easily use the x and y values to create all sorts of interesting variations on this. Try this one:
void main () {
vec2 p = uv();
float frequency = 10.;
float color = sin( p.x / p.y * frequency + time );
gl_FragColor = vec4( color,0.,0., 1. );
}What happens when you multiply p.x by p.y instead of dividing it? What happens when you multiply the entire output of the sin() function by p.y? All sort of fun variations to play with.
Hexler - codelife - glsl live-coding editor test #5 on Vimeo Shawn Lawson - RISC Chip - Mint: Kindohm. Visuals: Obi Wan Codenobi on Vimeo
void main() {
vec2 p = uvN();
float color = 0.;
float frequency = 2.;
float gain = 1.;
float thickness = .05;
p.x += sin( p.y * frequency + time * 4.) * gain;
color = abs( thickness / p.x );
gl_FragColor = vec4( color );
}OK, that’s fun, but the only thing better than a sine wave is twenty sine waves:
void main() {
vec2 p = uv();
float color = 0.;
float frequency = 2.;
float gain = 1.;
float thickness = .025;
for( float i = 0.; i < 10.; i++ ) {
p.x += sin( p.y + time * i) * gain;
color += abs( thickness / p.x );
}
gl_FragColor = vec4( color );
}In The Force, we can get access to the data from our laptop microphones by clicking on the microphone button at the bottom of the screen. Once you’ve granted Firefox access to use the microphone, the FFT data will then be available in a four-item uniform named bands, where bands[0] is the low-frequency content and bands[3] is the high frequency content. Try setting the value of gain in the above script to use one of these bands and have fun watching the results… and then try messing around with the formula in other ways, dividing instead of of multiplying, use fewer or more iterations of the for loop etc.
For a more basic introductionn on how to visualize FFT data on the canvas, see: materials/lecture7.markdown at master · cs4241-19a/materials · GitHub
Evelyn Lambart / Norman McLaren : Begone Dull Care - YouTube Norman McLaren: Norman McLaren - Dots (1940) - YouTube
Dan Sandin / Tom DeFanti / Mimi Shevitz: https://www.youtube.com/watch?v=hw9kY85DkfE Brian O’Reilly (start at 3:15 for video synthesis example) - Point Line Cloud (selections) on Vimeo
You can find Hydra at: < hydra >
The easiest way to get started is to start changing numbers in existing sketches (you can load new random sketches by hitting the reload button in the upper right corner) and hitting ctrl+shift+enter to see how your results change after modifying the code.
There’s lots of commented examples to look at here: hydra-examples/0-getting-started.js at master · ojack/hydra-examples · GitHub
And there’s documentation for the various functions at: hydra/funcs.md at master · ojack/hydra · GitHub
Let’s start by creating an oscillator similar to what we did in The Force:
frequency = 10
horizontalScan = .1
osc( frequency, horizontalScan ).out()osc() also accepts a third parameters which colorizes the output. Let’s apply some transformations to the output of our oscillator.
osc( 16, .15, .5 )
.rotate( Math.PI/3 )
.pixelate( 10, 20 )
.out()OK, hopefully that’s enough to get people making interesting patterns… make sure to look at hydra/funcs.md at master · ojack/hydra · GitHub for some other functions to try out. Let’s bring in some FFT data so that we can make our visuals audio reactive.
The audio data for Hydra is stored in the tersely named a variable. We can access the “bins” of our FFT analysis via array access, e.g. a.bins[0]. The number of bins in the analysis can be set using a call to a.setBins().
Hydra will accept a function for any parameter. For example:
__time = 0
osc( () => __time++ % 100 ).out()
We can use these functions to read the current results of our FFT analysis and assign that result to parameters. In the example below we’ll control pixellation with the FFT.
// show the FFT analysis in the bottom-right corner
a.show()
osc(16, .15, .5)
.rotate(Math.PI/3)
.pixelate( () => a.fft[0] * 10, 20)
.out()
// control a lowpass filter on the fft values
// a value of 0 will contain no smoothing
a.setSmooth( .95 )
// set the number of "bins" to use
a.setBins(2)Hydra uses the Meyda.js library GitHub - meyda/meyda: Audio feature extraction for JavaScript. under the hood for FFT analysis. One of the nice features of Medya is that you can easily apply a lowpass filter to the FFT results, so that the output is less hectic.
It wouldn’t be fair to leave Hydra without getting some video feedback into the mix. In order to use feedback, we have to assign our output to a buffer… these buffers are named o0 , o1 etc. We can then use the buffers for blending / mixing etc.
osc( 16, .15, .5 )
.modulate( o0, () => mouse.x * .0001 )
.rotate( Math.PI/3 )
.out( o0 )The demoscene is a culture that began in the 1980s, when hackers created “cracktros" for the software they cracked… these were short audiovisual demos that showed off the skills of the hacker and often contained shoutouts to their various friends.
Over time, some hackers became more interested in the audiovisual demos than they were in pirating software, and this became the birth of the “demoscene”, where programmers would create audiovisual “demos” that ran in realtime, often on severely constrained hardware. The demoscene continues today with may events held annually around the world (for example, http://atparty-demoscene.net/2019/05/09/party-2019-performers/ and https://2019.cookie.paris)
A technique used in many (but certainly not all) demos is ray marching, which is a physically informed graphical rendering technique that can create very different graphics from the classic OpenGL rendering pipeline; see Ray Marching and Signed Distance Functions for more information about how ray marching works. Here is a classic demoscene video from 2007 that shows some of these techniques in action:
Chaos Theory by Conspiracy | 64k intro (FullHD 1080p HQ demoscene demo) - YouTube
These types of effects are very different from what we’ve looked at so far, but certainly a lot of fun! We can experiment with them using a JavaScript library, marching.js playground, that lets us work at a high-level without worrying writing complex shader programs. These graphics can also be audio-reactive.
Let’s start by just putting a sphere on the screen. We can pass a Sphere() to the march() function, and then call .render() on the result. Highlight the code below and hit ctrl+enter to execute it.
march( Sphere() ).render()There are many more interesting forms to render in marching.js. For example, fractals:
march( Julia() ).render()You can get an idea of what’s available by looking through the reference or the various examples in the playground.
One of the classic raymarching techniques is to repeat forms in space… this is a very simple operation on the shader. Let’s create a field of repeating spheres:
march(
Repeat(
Sphere(),
4
)
).render()Note that the spheres end after about 10 spheres or so; in order to reduce rendering time, marching.js only renders to a certain distance on the horizon. However, if we decrease the size of the sphere (which defaults to 1) and the repeat size, we can get a much larger number of repetitions in the space available.
march(
Repeat(
Sphere(.1),
.4
)
).render()At this point, they basically fade infinitely to the horizon. Unfortunately this results in a bit of a mess, it looks better if we fade them out in the distance using fog.
march(
Repeat(
Sphere(.1),
.4
)
)
.fog( .1, Vec3(0) )
.render()We can easily animate the distance between the spheres by storing a reference to our repeat object and then manipulating its .distance property. However, in order to do this we need to tell marching.js that we want to animate our scene and specify a quality setting. Try a value of 3 for quality and if that easily works, maybe raise it to 4 or 5. Most laptops won’t be able to handle more than 5.
march(
rpt = Repeat(
Sphere(.1),
.4
)
)
.fog( .1, Vec3(0) )
.render(4,true)
onframe = time => {
rpt.distance = .25 + time/10 % 1
}By defining an onframe function on the global window object, we can setup basic animations.
As you might have guessed given this topic of this workshop, we can also use this animation function to read FFT analysis data and then assign it to properties.
FFT.start()
march(
r = Repeat(
Sphere(.1),
.4
)
)
.fog( .1, Vec3(0) )
.render(4,true)
onframe = t => {
r.distance.x = FFT.low
r.distance.y = FFT.high
r.distance.z = FFT.mid
}
FFT.windowSize *= 2We can increase our window size to get a smoother response out of the FFT.
Marching.js is built-in to the new, pre-alpha version of Gibber, which can be found at:
Mapping in Gibber v2 is fairly simple, you simply assign a audio object to a visual property. For example:
k = Kick()
k.trigger.seq( 1,1/4 )
s = Sphere()
s.radius = k
However, many times the range of values created by the audio object is not appropriate for the mapping you’re trying to create. You can add a bias (offset) and a scalar (multiplier) to control this.
k = Kick()
k.trigger.seq( 1,1/4 )
s = Sphere()
s.radius = k
s.radius.offset = .5
s.radius.multiplier = 10
There’s an audiovisual tutorial included in v2 that walks through some other examples of mappings.
The older (current?) version of gibber at gibber.cc has a more powerful mapping system builtin to it; hopefully I will be able to integrate this into the new version of Gibber soon.
In v1, every property has metadata associated with it that tells you the range of values it expects to receive. This means that when you perform mappings they’re more likely to automatically be in the correct range of values.
For example:
a = Cube()
a.scale = 1.5
b = XOX( 'x*o*x*o-' )
a.rotation.x = b.kick.out
a.rotation.y = b.snare.outYou can also map to other audio properties like frequency instead of only using amplitude.