How to Sync Actions with Audio in Unity
It’s certainly been a while since you’ve heard from me so I’ll just jump right into it.
Objective: Sync material emission with music in Unity
Breakdown:
- Return some kind of value that we can use to determine heights in the audio clip as it plays in the scene
- Perform an action using that value. In this case, emit light from a material
Steps:
- Step 1: Use audioSource.GetSpectrumData(array, channel, FFTWindow) to get the audio clips spectrum data as it plays
- Step 2: Calculate the average spectrum height
- Step 3: If the average spectrum is above our threshold then execute an event.
- Step 4: Subscribers handle event
Full Code Tutorial
Step 1: GetSpectrumData()
Unity has this super handy method for the AudioSource and AudioListener classes called GetSpectrumData(). This method fill an array with spectrum data from the audio clip as it goes along. The key here is that the array must be a power of 2. So 2, 4, 8…, 256, 512 etc. I would personally recommend a minimum of 1024 as values lower than that didn’t really give good results. The implementation of this is very simple. In code it looks like this:
Why the if statement? This allows the script a little bit of flexibility. If the user wants to have a specific AudioSource to listen to than they can declare it, but if not then the Audio Syncer will just listen to the AudioListener in the scene. However, be aware that listening to the AudioListener will return specturm data for ALL sounds heard including sound effects. So if you only want to sync only to music then you’ll have to define which AudioSource the music is playing from. That’s Step 1 done! Easy, right?
Step 2: Spectrum Average
Who here remembers how to calculate the average of a set of numbers? Luckily this step is exactly what you think it is. Nothing fancy or mind blowing. Just simple math.
For those of us who don’t remember, the average is calculated like so:
sum of all values/the numbers of values
The sum of all values in the set divided by the number of values in the set. Since we’re using an array we can simply loop through, add everything and then divide that number by the array’s length. Like so:
Simple enough right? This value is the key for pretty much everything we’re going to be doing so without this nothing would work.
Step 3: Execute Event
In our case, we’ll be emitting light based off of whatever the average was of the Spectrum Data. I personally used an Observer Pattern because it was more efficient code wise. If you’re unsure of how to setup an event in your code then you can check out this video by Unity.
So the code for our Event Caller is simple enough. Create an event and then execute it after it calculates the average.
Why are we multiplying _spectrumAverage by _emissionMultiplier? This is entirely a quality of life feature for the end user. The spectrum data values tend to be very small so the average is very small which makes fine tuning the threshold really awkward.
What’s the threshold for? The threshold is for omitting sections of the Audio Clip that are quieter than the threshold. For example, if you don’t want anything to happen when it’s quiet then you can increase the threshold to a suitable value
Why are we checking Sync != null? This bool expression is to make sure that the Sync() event has subscribers so that it doesn’t execute to nothing. It’s generally good practice to check first before executing an event.
Step 4: Event Handling
Since this is another script let’s do another breakdown.
Breakdown:
- Subscribe to Event Holder
- Get Material
- Multiply current emission color by new color, emission multiplier, and spectrum average
Steps:
- Step 4.1: SyncCenter.event += Update()
- Step 4.2: GetComponent<MeshRenderer>().material
- Step 4.3: material.SetColor(“_Emission”, newColor * _emissionMultiplier * _spectrumAverage
Step 4.1: Subscribe to Sync Event
Since this is an observer pattern we need to subscribe our handler (or listeners) to our script that’s executing the event. In my case, I named the Event Holder “SyncCenter” which has an event called “Sync”. Most intermediate and beyond C# coders this will be a no brainer but here is the code for anyone who may not know yet.
Step 4.2: Get the Material from the MeshRenderer
Why get the MeshRenderer? We get the MeshRenderer instead of the material directly because during runtime Unity creates instances of materials for each MeshRenderer in the scene. This means that each material in the MeshRenderer’s material array is unique to that MeshRenderer. If we were to get the material directly then we would run into the problem of adjusting all objects in the scene that share the same material instead of just one object in the scene that we want to manipulate.
There is one thing we need to check for. MeshRenderers carry an array of materials. MeshRenderer.material simply returns the material at [0]. So if we need to specify which material to adjust the emission then we need to define the index. This can be a variable in your inspector for ease of use like so:
Step 4.3: Manipulating the Emission Colour
After getting the desired material we need to enable the “_EmissionColor” keyword on the Unity Shaders. Here’s a quick code snippet on how to do it
If you don’t know how to find the keywords then you can refer to my other article here about it.
Now that we’ve enabled the Keyword the only thing to do is wait for the event and set the colour of the emission accordingly.
Conclusion:
That’s the gist of it there! There are some more advanced use cases that I didn’t cover, but for the sake of this tutorial this should cover the basics that you’ll need to implement your own working version.
Here are the full scripts for how I did it:
