3D Music Visualization with Spinning Display

Introduction

The primary goal of our ECE 4760 final project is to build a system that generates 3D visual effects on a spinning LED screen according to the beats and tempo of a given song.

As a device that is primarily for entertainment purposes, we want the final design to be simple to control, portable, easy to use and cool. Thus, we decided to generate 3D visual effects by spinning an LED screen around its middle vertical axis. The screen is 10x10 and is a mix of red and yellow LEDs. In order to use the system, the user would run a Matlab script with the song file in the same directory as the script. Matlab will analyse the song and write appropriate parameters to a file for the microcontroller program to use. Then, the user would program the microcontroller and turn on two switches when the song starts. The visual effects will change throughout the song to best reflect the tune.

High Level Design

Rationale and Source of Our Project Idea

We wanted our final project to be an entertainment device that is easy to use and cool at the same time. As great lovers of music, we wanted to do something that would help people enjoy music more by seeing the music in addition to hearing it. Also since both of our team members like signal processing, we wanted to do a project that involves some of that. As we were looking through the final projects of previous years, we came across the "5x5x5 LED Cube - Orientation Independent 3D Display project" of Spring 2008. This project inspired us to build something with LEDs since they are cheap, easily portable and can generate awesome visual effects. We also wanted the visual effects to be 3D. We learnt from Bruce that the hardware of the 5x5x5 LED cube project was very tedious to implement. Since we wanted the display to be bigger, around 10x10, making a 10x10x10 cube is not exactly feasible. After consulting Bruce with some other implementations for 3D, we decided to spin a 10x10 LED screen around its middle vertical axis to display scenes of depth. Throughout the song, the visual effects will constantly change according to the tune. The cost would be low since theres only two microcontrollers, LEDs, a motor, a H bridge and decoders are needed. This project allows users to have a mini light show at home while listening to music. It brings some of the excitement from a concert to the user’s house. This project will also show that 3D display is not something that is very far off or something that needs a lot of money to make.

Background Math

The maths involved in our project is very simple.

• The internal time kept in C program is in units of 50 milliseconds
• Basic knowledge of binary representation is used to generate square waves for the motor and control the motor's speed

Logical Structure

At a high level, the system has five main blocks: the matlab program that analyses the song and identifies certain characteristics; the C program that controls the visual effects; the C program that controls the speed of the motor; the LED hardware set-up that includes LED screen, PCB board, and a microcontroller; the motor hardware that contains a microcontroller with a motor.

High Level Diagram

Hardware/Software Tradeoffs

There are many other possible ways that we could have implemented our project. For example, in order to control the visual display by the audio file and synchronize the two, we could have used hardware but that would increase the complexity of our project significantly. This complex implementation would need an audio sensor and real time analysis to detect the beats of the tune. This would also be be hard to achieve because of the noise in the environment. Thus, we chose to use Matlab to do preprocessing of the audio signal. For the other components of the project, the software and hardware separation is apparent: The LED software controls the decoders that choose which lights on the screen should turn on. The motor software outputs 4 signals to drive the motor. The spinning of the stand is done by the motor.

Standards

Our microcontroller code conforms to the ANSI C standard.

Patents, copyrights and trademarks

We have done a lot of research on existing 3D visual displays and have not seen anything similar. There are a lot of videos about LED screens and cubes that generate cool visual effects. Some of the videos are recorded to show effects while a song is being played. There have also been projects that spin a rod that has LEDs attached to one end. However, we have never seen a project that spins an LED screen around its own axis. We also have never seen a system that shows obviously different visual effects at the strong beats or high notes of a song. To the best of our knowledge, our project does not use ideas from existing patents and does not violate any copyright laws.

Hardware

This project is very hardware heavy. First the LED display screen must be made. The 10 x 10 LED screen is constructed so that the rows have a common ground and the columns have a common Vdd, as shown in the appendix. This is so that each LED can be individually switched on when its two terminals are connected to Vdd and ground. The switch that controls which lines gets selected is done by a NFET which gets switched on when its gate gets pulled high.

Once the LED screen is done, we need a custom LED driver circuit. This is done with decoders as shown in the appendix. Since there are 10 Vdd and ground to choose from, two 4-16 decoders are needed. The decoder takes in a 4 bit command to know which line to drive high. Unfortunately, the decoders we used are active low, therefore we needed to use place inverters after all 10 lines from the decoder as to drive the NFET switch. For example if we wanted to drive the LED on row 4 and column 5, PORTC would be set to 3 as to drive the 4th line to ground and PORTA set to 4 as the drive 5th line to ground. Then the inverters will invert the grounds to Vdd and close the NFET switches for the 4th row and 5th column to turn on that particular LED. This circuit needs to be implemented on a PC board.
Since the LED screen needs to be spinning to create the 3D visual effects, the whole LED driver circuit, LED screen, and the microcontroller will have to be mounted together so that they will all spin as not to tangle the wires. This was initially designed to be done with a ball bearing. However the motor was not powerful enough; thus the ball bearing was not used. The motor is connected to a rod which is then mounted perpendicularly to the PC board that have the LED driver circuitry. The PC board will then be mounted to a piece of wood that have the LED screen on top of it. Make sure that the PC board’s back is facing the piece of wood to make soldering to the LED screen easier.

Once the LED structure is completed, the motor circuitry needs to be designed as shown above. The motor needs four square waves that are 90 degrees phase shifted from each other. This can be done with two squares waves 90 degrees shifted and their inverses. These four signals will be generated on another microcontroller. The signals from the microcontroller will go to an H bridge as to give the signals the power needed to drive the motor. The motor needs 2.7 V and a 1A power supply but the microcontroller cannot supply so much current. Therefore the microcontroller and H bridge need two power supplies that have a common ground. The enable on the H bridge is always high as so that the output always the save as the input voltage-wise. The four outputs of the H bridge that are 90 degree shifted are wired to the motor as shown in the appendix.
There are buttons and switches attached to both microcontroller. The mircocontroller that controls the motor have a switch that tell the it to generate the square waves when set to high. The microcontroller that controls the LED screen has a button and switch. The switch functions as a pause when set to high and the button functions like a fast forwards to sync the visual display and song.

Software

Matlab audio analyser

Since the goal of this project is to visualize music, we want changing visual effects throughout the song. More importantly, at strong beats or high notes of the song, we want to display a special visual effect.
In order to achieve this, we decided to separate the input tune into pieces of three seconds. For all the three second periods, the mean and variance of the signal is calculated. These two values are used to create a hash function that would choose 1 of then 9 visual effects to display by the equation: ((mean + standard deviation)*20) % 9 in the C program on the microcontroller. This equation was chosen because the result it gives will pick the visual effect with a certain amount of randomness.
We also want to locate every local maximum in the signal at an appropriate granularity so we can display some special visual effects at the local maximums. To achieve this, we first find the magnitude of the absolute maximum in the signal and saved it as Amax. Then, we find the local maximum every 250 ms since any finer granularity will make the local maximum too clustered for visual effects. For each local maximum, if it is greater than one fourths of the Amax, we save the time of local maximum in units of 100 ms. We decided to use the threshold of one fourths Amax because this gave a reasonable amount of local maximums that were spread apart well.

Plot of a single channel of audio with the detected local maximums annotated with a magnitude of 0.5

Integrating the MCU with MATLAB

In order to communicate the results from MATLAB to the microcontroller program, Matlab generates a header file for the C program. The header folder contains several pieces of information: the duration of the song, a 2D array for the mean and variance of every 3 second piece of the song and a 2D array that contains the time of the local maximums in units of 100 ms and the amplitude of the local maximum. The length of the song is written to the variable duration. The length of the 2 2D arrays are stored in length_mstd, length_pulse respectively.

MCU software

Motor control on MCU
The motor requires four square waves that are 90 degrees out of phase with each other as input. In order to generate the four signals on the microcontroller, we use PORTA0-PORTA3. The frequency of the square wave that is generated is around 1000Hz. This is done by setting the timer to have a prescalar of 128. OCR0A is set to 31 and the ISR interrupts on timer compare match. This means that the ISR will be executed every 31*128/16000000 = 0.000248s. In the ISR, a variable 'count' is cycled from 0 to 3 to 0 again. The motor control has a main while loop. The while loop will execute the following commands when a switch is on. When the switch is off, nothing is done.
As shown in the table, if count is 0, PORTA is set to 9. If count is 1, PORTA is set to 10. If count is 2, PORTA is set to 6. If count is 3, PORTA is set to 5.
This generates a square wave because as shown in the following table, there are 2 bits that are high at each moment. The bits that are high alternate to produce 4 square waves that are exactly out of phase with each other. Thus the square wave has a frequency of 1/(4*0.000248) = 1008 Hz. This corresponds to a speed of 1008/200 = 5.04 rps as the motor has an angle step of 1.8 degrees. While the microcontroller can generate faster square waves, 5 rps is the fastest the motor can go.

	Count = 0	Count = 1	Count = 2	Count = 3
PORTA0	1	1	0	0
PORTA1	0	0	1	1
PORTA2	0	1	1	0
PORTA3	1	0	0	1

Port control on MCU
In order to make programming easier, PORTA0 to PORTA3 are the inputs to the decoder for the rows. PORTC0 to PORTC3 are the inputs to the decoder for the columns. PORTD0 is for a switch that when turned off, freezes the timing bookkeeping on the MCU. When the switch is off, the display of visual effects is halted. When the switch is turned on, the visual effect will continue where it left off. This implementation is chosen because when the user wants to stop the music and display, he or she can just turn off the switch. When the music is played again and the switch is turned back on, the music and display will still be in sync. PortD2 is connected to a button. When the button is pressed, the internal time of the micro controller will be increased by 100 ms. This design choice was made because it is possible that the user would notice that the display lags slightly behind the music. When this happens, the user can press the switch as many times he or she desires till the display is in sync with the music again.
Visual effect timing control on MCU
In the program, time is kept in units of 0.5 ms since the nature of our system does not require finer granularity. Through experimenting, 0.5 ms is sufficient for the timing controls for the visual effects. In order to set 50ms as the time base, timer 0 is set up with a prescalar of 64. The OCR0A register is given a value of 124. Thus, timer 0 would interrupt every 0.5 ms.
For the timing control, the main function executes as long as the internal time (will be referred to as maintime) is smaller than the duration that is written in the header file by matlab. In the main function, when the switch is off, nothing happens and the maintime is also stopped. The following discussion applies to when the switch is on. The button for fast forward is debounced every 105 ms to see if it is pressed. Also, there is a variable: mstd_index that is initialied to 0 and incremented to go through the 2D array of mean and standard deviations. The program checks to see if the time is at a multiple of 3 seconds. If it is, (the mstd_index + 1) * 60 is checked against maintime. If it is, mstd_index is incremented since this means that while displaying visual effects, the program is behind on schedule because it was playing other visual effects when the most recent 3 second mark passed. The 3 second visual effect functions are written to take in an integer as an input. The reason for this implementation choice is explained in the section: implementations in software that did not work. On a high level: every 3 second visual effect is separated into 10 portions because there are 10 rows and columns. This input index to the function chooses which rows and/or columns will be displaying a part of the visual effect. In order for human eyes to see these individual portions, each individual portion will be displayed for 260ms. Thus, when a visual effect is first chosen, a variable that signals the beginning of the counting of 260 ms for timer 0:i_n will be set. Then, in the while loop, we will always check if the 260 ms is up. If it is, i_n will be incremented. i_n will be set to 0 once it reaches 40 to avoid overflow. If the 260ms is not up, i_n will be unchanged. Also, in the while loop, another variable: lm_index is initialized to 0 and is used to go through the 2D array that contains the local maximum times. In the while loop, we always check if the lm_index entry in the local maximum time array is later than maintime. If it is, we will check the lm_index+1 entry in the local maximum array to determine if we are too far behind on schedule recursively. We stop when we are at the entry that is the most recent local maximum. Then, a local maximum effect will be chosen according to the magnitude of the local maximum. The local maximum effects are different from the 3 second effects in that they are shorter and more intense.
Visual effect generation on MCU
The layout of the LED screen with column and row numbers are as followed to help the explanation

The column LEDs are connected to Vdd and the row LEDs are connected to ground. In order to turn on one LED, we need to choose the corresponding row and column. Therefore the MCU needs to cycle each LED in order to display an image. In other words, the MCU has already cycled 100 times in order to display an image that involves every LED. Thus if we want to turn on a whole row, we will choose the ground of 1 row and cycle through choosing the columns very quickly. We implemented the following visual effects:
The effects that are going to be displayed in the three second time periods:

A square outline appears at border, moves towards the center of the screen hence shrinks in size till only the middle four LEDs light up and then grows out again.

A square expands from the middle of the screen till it reaches the borders

Same as the first effect 1 but the square changes size at half the speed

Seemingly random LEDs light up in each row starting from the top row. This effect looks like raindrops

Whole columns light up one after another sequentially

Starting from the (0,0) point, all the LEDs to the top and left of LED (i,i) light up. i is incremented from 0 to 9

Diagonal lines of the function: y = x - b, where b is decremented from 19 to 0. Given the coordinate in the above LED layout

Divide the screen into four sections with the 2 diagonal lines. The 2 sections on the left and right light up.

Divide the screen into four sections with the 2 diagonal lines. The right most section gradually lights up.

The effects that are going to be displayed at the local maximums:

Fast firework effect: The middle 2 columns of the screen light up starting from the bottom row. When the middle of the top most row lights up, lights sparkle in each row starting from the top.

Slow firework effect: A slower implementation of the above effect

Sparkle effect: Seemingly random LEDs light up throughout the whole screen for short bursts

Full beat effect: The whole screen lights up and turns off very fast for 2 times

The above effects are the descriptions in 2D. When the screen spins, the effects gives the perception of collimated depth. Following are the visual effects in 2D and the 3D images. We tried to capture the different effects separately. However, at any one instance only 1 LED is on. This makes the effects very hard to capture on camera.