Lab session 3 : Experimenting with the sound sensor

Date: 16. September 2010

Duration of activity: 3,5 hours (11:00 - 14:30)

Group members participating : all group members, that is Anders Dyhrberg, Guillaume Depoyant, Michaël Ludmann and Michal Owsinski.

I. Objectives and summary

I. 1. Objectives

Build a robot with sound sensor

Experiment with the sound sensor

Extract measures with the Data Logger

Implement the sound controlled car

Implement the clap controlled car

I. 2. Summary

During this lab session, we managed to :

• Build the LEGO robot with sound sensor and test it to see how the sensor measures the noise using SonicSensorTest.java;
• Extract data to analyse sounds with SoundSampling.java and DataLogger.java;
• Implement and test the sound controlled car with SoundCtrCar.java and try to improve his behaviour namely with the ESCAPE button;
• Implement and test the clap controlled car, by having our robot react to the sound of clapping, and nothing else (no other loud sound allowed)

II. Results

II. 1. dB and dBA

While reading the details of the class SoundSensor [1] that we need to use for testing the sound sensor, we noticed that there are two methods for setting two modes for the sound sensor. They enable us to set dB mode or dBA mode.

dB stands for decibel : basically, it is a logarithmic unit used to measure a ratio between two powers - and it is mostly known for measuring sound level. The dB scale for sound covers the whole spectrum of sound (it ranges from 0 dB to 194 dB), whereas the dBA scale is provided by a sound filter, "used to approximate the human ear's response to sound, although the measurement is still in dB (SPL). These measurements usually refer to noise and noisome effects on humans and animals, and are in widespread use in the industry with regard to noise control issues, regulations and environmental standards." [2]

In fact, by using the dBA mode, the sound sensor will filter high and low frequencies that are roughly not perceived by the human ear, and it focuses on a frequency range about 1 kHz to 4 kHz, where our ears are much more sensitive [3].

So, what should we use when it comes to programming our robot ? Well, it really depends on what we try to do. dB mode seems to be generally more usefull, as it describes the real world and we can make more accurate algorithms while detecting some given sounds (see the last part of this lab note, with the clap detection), whereas dBA may be efficient with human voice recognition, when we need to reduce the "noise" of the real world.

II. 2. Test of the sound sensor

After reprogramming sonicSensorTest.java program and uploading it into the NXT controller we placed the robot at a fixed position and by using a constant volume sound we tested it. The robot was driving away from the sound for 1 meter with placed on top 2 sound sensors. The test was done in a small closed room almost without any sounds from the outside.

Blue line is representing the db value from the sound sensor 1, red line is representing the dba value from the sound sensor 2.

II. 3 Data logger

During our lab work we wanted to experiment with the differences between dB and dB(A). To compare them we added two microphones and instantiated two DataLoggers. But this gave us a very weird behaving log file. Seemed like the two logging instances were interfering with each other. Nothing obvious in the DataLogger code indicated there should be a problem with two co-existing loggers. To overcome this fast we implemented our own logger "Multilogger" which is a little more flexible. It will allow us to log several values at the same time for comparison and tag it with a time stamp.

public void log( int sample ){    appendLogStart(m_stringBuilder);     m_stringBuilder.append(sample);    streamToFile(m_stringBuilder);}    public void log( Object objs[] ){    appendLogStart(m_stringBuilder);    for(int i=0; i<objs.length; i++)    {     m_stringBuilder.append(objs[i].toString());           //Append ',' after each value, except the last one      if(i<objs.length-1)      m_stringBuilder.append(',');    }         streamToFile(m_stringBuilder);}    public void log( String string ){    appendLogStart(m_stringBuilder);     m_stringBuilder.append(string);    streamToFile(m_stringBuilder);}

We will probably use this logger instead for the rest of the course and extend it as it is required. It is currently exposing the three logging options as shown in the code above.
An obvious feature to implement fairly soon would be to log directly to PC using Bluetooth instead of files. Done right this would allow is in many scenarios to see the log during program execution, making debugging faster

Important Note: We never did discovered why the two DataLogger's could not co exist, we abandoned the issue, and worked around it.

II. 4. Sound controlled car

The first interactive application of the sound sensor was made using the program SoundCtrCar.java [4]. This program basically makes the robot wait for a first loud sound, then makes it move forward while waiting for another sound to make it turn right, then left, and finally the fourth loud sound makes it stop. Then the loop starts again.

A loud sound is defined by the program as a sound which has a value larger than a static threshold (here 90), given that the sensor is in dB mode and the values are rescaled in percent scale. So, theoretically, the louder the sound, the greater the value. In fact, it is the case most of the time, and the threshold was high enough to prevent the surrounding noise to interfere with the movement of the car. The most efficient way to make a "loud sound" was to bang on the floor, clapping or moving a nearby chair. Still, whistling gently in the sensor or simply blowing on it is seen as a loud noise, probably because it saturates the measurement in some way. So, it is sometimes hard to control perfectly the robot if you do not have a full control on the surrounding noise environment.

On the whole, it was satisfactory, but one problem remained. The program was written such as it was possible to stop it only at some precise times, because of many while loops. We therefore needed to implement some event listeners, so as to have the robot stop when the escape button is pressed. One easy solution would have been to add one event for the escape button, and write a system exit line code in the listener. But this is not a satisfactory solution, as it usually is an ugly thing to interrupt the proper running of a program and skipping last instructions. That is why we chose to write something a bit more sophisticated, by adding a listener for the sound sensor and some state variables.

The program listens now to the values of the sound sensor and to the state of the escape button. Each sample form the sound sensor is checked, and if its value is larger than the threshold, we increment a counter and do the corresponding action. Every four incrementation, we reset the counter to begin a new cycle. Meanwhile, if the escape button is pressed, a state variable is set to false, and the main program stops immediately, because we are just running a while loop which relies on the state of this variable.

However, this second try was not so conclusive at first glance. We indeed saw that a listener on a sensor port uses only the raw value returned by the sensor. As we were unsure of the way leJOS converts a raw value coming from a sound sensor into a percent value (we didn't know if the raw value was already in decibel, or if there was some kind of hidden processing), we checked the leJOS source code. We found in the SoundSensor.java [5] file that it was just re-scaled to fit in a percent scale, like that :

public int readValue()
{
    return ((1023 - port.readRawValue()) * 100/ 1023);
}

...which by the way makes us doubt about the relevance of the so called dB unit in leJOS (when is it real decibel ? when we read a raw value or a "percent" value ?)

Anyway, once this little line of code added in our program, it was better, but still not better than the first try. It had to be due to the sampling frequency, which was higher this time (because we always listen to the sensor port). So the same loud sound would often be sampled twice, and the robot would do some erratic moves. Our solution was to use a state variable which systematically ignores a loud sound if the previous sample was already considered as a loud sound. An other solution could have been to wait for a few milliseconds after each sample.

Here is our final code :

public class SoundCtrCar
{
    private static int soundThreshold = 90;
    private static boolean m_running = true;
    private static int m_state = 0;
    private static boolean m_allowNewReaction = true;
    public static void main(String [] args) throws Exception
    {
        LCD.drawString("dB level: ",0,0);
        LCD.refresh();
        Button.ESCAPE.addButtonListener( new ButtonListener () {
            @Override
            public void buttonPressed(Button arg0) {
                m_running = false;
            }
            @Override
                public void buttonReleased(Button arg0) {
                //do nothing
            }
        });
        SensorPort.S1.addSensorPortListener( new SensorPortListener() {
            @Override
            public void stateChanged(SensorPort arg0, int arg1, int arg2) {
            if(SensorPort.S1 == arg0)
            {
                int SoundSensorNormalizedValue = ((1023 - arg2) * 100/ 1023);
                LCD.drawInt(arg2,4,10,0);
                if(SoundSensorNormalizedValue < m_allownewreaction =" true;"> soundThreshold && m_allowNewReaction)
                {
                    //Notify program that we are above threshold
                        switch (m_state)
                        {
                            case 0:
                                LCD.drawString("Forward ",0,1);
                                Car.forward(100, 100);
                            break;
                            case 1:
                                LCD.drawString("Right   ",0,1);
                                Car.forward(100, 0);
                            break;
                            case 2:
                                LCD.drawString("Left    ",0,1);
                                Car.forward(0, 100);
                            break;
                            case 3:
                                LCD.drawString("Stop    ",0,1);
                                Car.stop();
                            break;
                    }
                    m_allowNewReaction = false;
                    m_state++;
                    if(m_state > 3)
                    m_state = 0;
                    }
                    }
                    }
                    });
while (m_running)
{
}
Car.stop();
LCD.clear();
LCD.drawString("Program stopped", 0, 0);
Thread.sleep(2000);
}
}

And our third try with all our improvements was a success ! The robot behaved more efficiently than with the original code.

II. 5 Clap Controlled Car

We implemented clap detections algorithm according to the method of Sivan Toledo.

The implementation was fairly simple it was implemented using a simplified State Structure evaluating all available samples according to the current state. Basically it is just evaluated if the sample will allow the program to stays in the current state or progress into the next. And if neither could be achieved, it will break out and reset to the 1st state again looking for the start of a new possible clap.

case NOCLAP: if(soundSensorNormalizedValue > CLAP_START_INDICATION_LEVEL) {  m_ClapState = ClapDetectionState.START_DETECTED;  m_LatestClapStateChangeTime = System.currentTimeMillis();     logger.log("NOCLAP -> START, " + soundSensorNormalizedValue + ", " + msSinceLastState + "ms"); } else {  //Do nothing } break;case START_DETECTED: if(msSinceLastState < CLAP_RISE_INDICATION_DELAY) {  if(soundSensorNormalizedValue > CLAP_PEAK_INDICATION_LEVEL)  {   m_ClapState = ClapDetectionState.PEAK_DETECTED;   m_LatestClapStateChangeTime = System.currentTimeMillis();          logger.log("START -> PEAK, " + soundSensorNormalizedValue + ", " + msSinceLastState + "ms");  } } else {  //Time is up, does not match clap profile  m_ClapState = ClapDetectionState.NOCLAP;  m_LatestClapStateChangeTime = System.currentTimeMillis();         logger.log("START -> NOCLAP, " + soundSensorNormalizedValue + ", " + msSinceLastState + "ms"); } break;case PEAK_DETECTED: if(msSinceLastState < CLAP_FALL_INDICATION_DELAY) {  if(soundSensorNormalizedValue < CLAP_END_INDICATION_LEVEL)  {   //Clap detected   m_clapCount++;   m_multiClapCount++;   LCD.drawInt(m_clapCount,4,12,0);           m_ClapState = ClapDetectionState.MULTI_CLAP_POTENTIAL;   m_LatestClapStateChangeTime = System.currentTimeMillis();
logger.log("PEAK -> MULTICLAP, " + soundSensorNormalizedValue + ", " + msSinceLastState + "ms");  } } else {  //Time is up, does not match clap profile  m_ClapState = ClapDetectionState.NOCLAP;  m_LatestClapStateChangeTime = System.currentTimeMillis();       logger.log("PEAK_DETECTED -> NOCLAP, " + soundSensorNormalizedValue + ", " + msSinceLastState + "ms"); } break;

The advantage of this structuring, makes it fairly simple to extend the program with more states for more advanced detection algorithms.

To verify this we added a multi clap, detection state. This state will evaluate if there have been more consecutive claps. This increase the usage of the algorithm a lot, since this will then allow us to listen for different multiclap counts and interpret them as different commands.
e.g single clap - start, double clap - stop, triple clap - turn.

case MULTI_CLAP_POTENTIAL: if(msSinceLastState < MULTIBLE_CLAP_DETECT_DELAY) {  if(soundSensorNormalizedValue > CLAP_START_INDICATION_LEVEL)  {   m_ClapState = ClapDetectionState.START_DETECTED;   m_LatestClapStateChangeTime = System.currentTimeMillis();   logger.log("MULTICLAP -> START, " + soundSensorNormalizedValue + ", " + msSinceLastState + "ms");  } } else {  //Evaluate multi clap scenario, no more are accepted since delay was to long  LCD.drawInt(m_multiClapCount,4,12,1);        //Cleanup  m_multiClapCount = 0;  m_ClapState = ClapDetectionState.NOCLAP;  m_LatestClapStateChangeTime = System.currentTimeMillis();
logger.log("MULTICLAP -> NOCLAP, " + soundSensorNormalizedValue + ", " + msSinceLastState + "ms"); } break;

The acceptable time between claps in order to accept them as a multiclap is configured using the MULTIBLE_CLAP_DETECT_DELAY const in top of the program. By experimenting 1 sec seems like a well working value.

III. References

[1] Class SoundSensor in leJOS documentation.

[2] db(A), dB(B) and dB(C) definitions provided by Wikipedia.

[3] Filters used for dBA and dBC, University of New South Wales, Australia

[4] A sound controlled car, Ole Caprani

[5] Code source for the sound sensor, leJOS developer documentation
[6] Clap Detection Algorithm by Sivan Toledo