Bubble Blower (Sound Detection)

Admin

12 years ago

Bubble Blower enables users to actually blow on the phone to make bubbles to appear on the screen. These bubbles grow from the bottom of the screen and pop when they reach the top. This magical effect is made possible by leveraging the phone’s microphone to detect the sound of blowing.

Bubble Blower has an application bar for exposing instructions, settings, and an about page, but it can be hidden (or brought back) by tapping the screen.

About the Microphone

The microphone API is technically an XNA feature; the relevant Microphone class resides in the Microsoft.Xna.Framework.Audio namespace in the Microsoft.Xna.Framework assembly. As with the sound effects APIs from the preceding part of the book, Silverlight apps can use the microphone seamlessly as long as XNA’s FrameworkDispatcher.Update is called regularly.

You can get an instance of the microphone with the static Microphone.Default property. From this instance, you can call Start, Stop, and check its State at any time (which is either Started or Stopped). To retrieve the raw audio data from the microphone, you attach a handler to its BufferReady event. By default, BufferReady is raised every second, providing access to the last second of audio data, but its frequency can be changed by setting the BufferDuration property.

You need the ID_CAP_MICROPHONE capability in your application manifest in order to use the microphone!

If you have removed this from your manifest, Microphone.Default is always null, and the Microphone.All collection is always empty. By requesting this capability, Microphone.Default is guaranteed to be non-null whenever your app runs.Of course, this is only a concern at development time because marketplace certification automatically adds this capability to your manifest if needed.

BufferDuration only supports 101 distinct values!

BufferDuration, defined as a TimeSpan,must be set to a value between 100 milliseconds and 1 second (inclusive). Furthermore, the duration must be a multiple of 10 milliseconds. If you attempt to set it to an invalid value, an ArgumentOutOfRangeException is thrown.

If I plug in a headset with a microphone, does that show up as a second microphone in the Microphone.All collection?

No. Although a headset microphone can be used, it automatically takes over as the default microphone. As far as apps are concerned, there is only ever one microphone.

Inside a BufferReady event handler, you can call Microphone’s GetData method to fill a buffer (a byte array) with the latest audio data. If you want to capture all the audio since the last event, you must make sure the buffer is large enough. Microphone’s GetSampleSizeInBytes method can tell you how large it needs to be for any passed-in TimeSpan, so calling it with Microphone.Default.BufferDuration gives you the desired size.

What can I do with the raw audio data once I receive it as an array of bytes? How do I interpret it?

The bytes represent the audio encoded in the linear pulse-code modulation (LPCM) format, the standard Windows format for raw and uncompressed audio.This format is used in .wav files, and it is often just referred to as PCM encoding rather than LPCM.

Each 2-byte value in the buffer represents an audio sample for a very small slice of time.Windows phones capture 16,000 samples per second (revealed by Microphone’s SampleRate property), so when BufferReady is called every second, the buffer size is 32,000 bytes (16,000 samples x 2 bytes per sample). If the audio were recorded in stereo,which does not happen on the phone, the data would be twice as long and each sample would alternate between left and right channels. With each 2-byte value, zero represents silence. When sound occurs, it creates a waveform that oscillates between positive and negative values.The larger the absolute value is (no matter whether it’s positive or negative), the louder the sound is.

You can do several things easily with these raw data values. For example, you can detect the relative volume of the audio over time (as this app does), and you can play back or save the audio (as done in the next two chapters). With more work, you can do advanced things like pitch detection.You could even turn the data into an actual .wav file by adding a RIFF header. See http://bit.ly/wavespec for more details.

Ignore Microphone’s IsHeadset property!

This property was designed for the PC and Xbox and doesn’t work for the phone. It is always false, even if the microphone actually belongs to a headset.

Can I process audio during a phone call?

No, you cannot receive any data from the microphone during a phone call (or while the phone is locked).

The Bubble User Control

This app needs bubbles—lots of bubbles—so it makes sense to create a user control to represent a bubble. Listing 34.1 contains the visuals for the Bubble user control. Figure 34.1 shows what Bubble looks like.

LISTING 34.1 Bubble.xaml—The User Interface for The Bubble User Control

[code]

[/code]

Although the ellipse is straightforward, the gradient-filled path creating the arc on top of the ellipse is something that would most likely need to be crafted in a tool like Expression Blend. I certainly didn’t come up with those numbers by hand!
The purpose of this control is simply to encapsulate the visuals for a bubble. It has no built-in behavior, so the Bubble.xaml.cs code-behind file contains nothing other than the required call to InitializeComponent inside its constructor.

FIGURE 34.1 The Bubble user control, shown
against a black background.

The Main User Interface

Listing 34.2 contains the XAML for the main page. It is simple because there’s nothing other than the application bar on the page until bubbles appear.

LISTING 34.2 MainPage.xaml—The User Interface for Bubble Blower’s Main Page

[code]

<phone:PhoneApplicationPage x:Class=”WindowsPhoneApp.MainPage”
xmlns=”http://schemas.microsoft.com/winfx/2006/xaml/presentation”
xmlns:x=”http://schemas.microsoft.com/winfx/2006/xaml”
xmlns:phone=”clr-namespace:Microsoft.Phone.Controls;assembly=Microsoft.Phone”
xmlns:shell=”clr-namespace:Microsoft.Phone.Shell;assembly=Microsoft.Phone”
SupportedOrientations=”Portrait”>
<!– The application bar, with 1 button and 2 menu items –>
<phone:PhoneApplicationPage.ApplicationBar>
<shell:ApplicationBar Opacity=”.5”>
<shell:ApplicationBarIconButton
IconUri=”/Shared/Images/appbar.instructions.png”
Text=”instructions” Click=”InstructionsButton_Click”/>
<shell:ApplicationBar.MenuItems>
<shell:ApplicationBarMenuItem Text=”settings”
Click=”SettingsMenuItem_Click”/>
<shell:ApplicationBarMenuItem Text=”about” Click=”AboutMenuItem_Click”/>
</shell:ApplicationBar.MenuItems>
</shell:ApplicationBar>
</phone:PhoneApplicationPage.ApplicationBar>
<!– Hard-coded background! Grid is only here because
the page can’t be given a background. –>
<Grid Background=”Black”>
<!– Size and background are there to enable tapping
anywhere except close to the application bar –>
<Canvas x:Name=”RootCanvas” Width=”480” Height=”700” VerticalAlignment=”Top”
MouseLeftButtonDown=”RootCanvas_MouseLeftButtonDown”
Background=”Transparent”/>
</Grid>
</phone:PhoneApplicationPage>

[/code]

The background is hard-coded to black because the bubbles would not appear on a white background. This is set on the grid because setting the page’s background property has no effect.
The application bar is half-opaque, so bubbles show through underneath yet the text on the button and menu items remains readable. An opacity of zero would not work (without hardcoding the application bar foreground to a color such as white) because the application bar text would be black on top of the hard-coded black background when the light theme is used. As always, you need to be careful when using any hardcoded colors in your app!
The canvas has a MouseLeftButtonDown event handler for toggling the application bar’s visibility when tapped. It has an explicit transparent background, so it responds to taps anywhere within its area.
The canvas only covers the top 700 pixels of the screen because if it gets too close to the application bar, users might accidentally tap it (and hide the application bar) when trying to tap the bar, especially its ellipsis.

The Main Code-Behind

The code-behind in Listing 34.3 is responsible for starting the microphone, determining when to produce bubbles, producing them, and animating them from the bottom of the screen to the top. This bottom-to-top animation only makes sense if the microphone is below the screen, but that’s a pretty safe assumption for all phone models considering how phones work.

The technique for determining when the user is blowing is simply checking for loudenough sounds. (Blowing on a microphone produces a pretty loud sound.) If the sound picked up by the microphone is below a certain threshold, it assumes the sound is background noise. Of course, this approach can be fooled by just talking loud enough or making any other noises.

LISTING 34.3 MainPage.xaml.cs—The Code-Behind for Bubble Blower’s Main Page

[code]

using System;
using System.Windows;
using System.Windows.Controls;
using System.Windows.Input;
using System.Windows.Media;
using System.Windows.Media.Animation;
using Microsoft.Phone.Controls;
using Microsoft.Xna.Framework.Audio;
namespace WindowsPhoneApp
{
public partial class MainPage : PhoneApplicationPage
{
Random random = new Random();
byte[] buffer;
int currentVolume;
public MainPage()
{
InitializeComponent();
// Get called on every frame
CompositionTarget.Rendering += CompositionTarget_Rendering;
// Prevent off-screen bubble parts from being
// seen when animating to other pages
this.Clip = new RectangleGeometry { Rect = new Rect(0, 0, 480, 800) };
// Required for XNA Microphone API to work
Microsoft.Xna.Framework.FrameworkDispatcher.Update();
// Configure the microphone with the smallest supported BufferDuration (.1)
Microphone.Default.BufferDuration = TimeSpan.FromSeconds(.1);
Microphone.Default.BufferReady += Microphone_BufferReady;
// Initialize the buffer for holding microphone data
int size = Microphone.Default.GetSampleSizeInBytes(
Microphone.Default.BufferDuration);
this.buffer = new byte[size];
// Start listening
Microphone.Default.Start();
}
void Microphone_BufferReady(object sender, EventArgs e)
{
int size = Microphone.Default.GetData(this.buffer);
if (size > 0)
this.currentVolume = GetAverageVolume(size);
}
void CompositionTarget_Rendering(object sender, EventArgs e)
{
// Required for XNA Microphone API to work
Microsoft.Xna.Framework.FrameworkDispatcher.Update();
if (this.currentVolume > Settings.VolumeThreshold.Value)
AddBubble(false);
if (this.currentVolume > Settings.VolumeThreshold.Value * 5)
AddBubble(true); // Add an extra (& faster) bubble for extra-hard blowing
}
// Returns the average value among all the values in the buffer
int GetAverageVolume(int numBytes)
{
long total = 0;
// Buffer is an array of bytes, but we want to examine each 2-byte value.
// [SampleDuration for 1 sec (32000) / SampleRate (16000) = 2 bytes]
// Therefore, we iterate through the array 2 bytes at a time.
for (int i = 0; i < numBytes; i += 2)
{
// Cast from short to int to prevent -32768 from overflowing Math.Abs:
int value = Math.Abs((int)BitConverter.ToInt16(this.buffer, i));
total += value;
}
return (int)(total / (numBytes / 2));
}
void AddBubble(bool fast)
{
// Choose a scale for the bubble between .1 (10%) and 1 (100%)
double scale = (double)random.Next(10, 100) / 100;
// Set the vertical animation duration based on the scale
// (from .55 sec for scale==.1 to 1 sec for scale==1)
double duration = .5 + scale / 2;
// If this isn’t a “fast” bubble, lengthen the duration of the animation
if (!fast)
duration *= 1.5;
// Create a new bubble, set its location/size & add it to the root canvas
Bubble bubble = new Bubble();
Canvas.SetLeft(bubble, random.Next(-100, (int)this.ActualWidth + 100));
Canvas.SetTop(bubble, this.ActualHeight + 50);
bubble.RenderTransform = new ScaleTransform { ScaleX = scale,
ScaleY = scale };
bubble.RenderTransformOrigin = new Point(.5, .5);
this.RootCanvas.Children.Add(bubble);
// Dynamically create a new storyboard for the bubble with four animations
Storyboard storyboard = new Storyboard();
Storyboard.SetTarget(storyboard, bubble);
storyboard.Completed += delegate(object sender, EventArgs e)
{
// “Pop” the bubble when the storyboard is done
this.RootCanvas.Children.Remove(bubble);
};
// Animate the vertical position from just below the bottom of the screen
// (set earlier) to just above the top of the screen
DoubleAnimation topAnimation = new DoubleAnimation { To = -100,
Duration = TimeSpan.FromSeconds(duration),
EasingFunction = new QuadraticEase() };
Storyboard.SetTargetProperty(topAnimation,
new PropertyPath(“(Canvas.Top)”));
// Animate the horizontal position from the center
// to the randomly-chosen position previously set
DoubleAnimation leftAnimation = new DoubleAnimation {
From = this.ActualWidth / 2, Duration = TimeSpan.FromSeconds(.5),
EasingFunction = new QuadraticEase() };
Storyboard.SetTargetProperty(leftAnimation,
new PropertyPath(“(Canvas.Left)”));
// Animate the horizontal scale from 0 to the
// randomly-chosen value previously set
DoubleAnimation scaleXAnimation = new DoubleAnimation { From = 0,
Duration = TimeSpan.FromSeconds(.5),
EasingFunction = new QuadraticEase() };
Storyboard.SetTargetProperty(scaleXAnimation, new PropertyPath(
“(UserControl.RenderTransform).(ScaleTransform.ScaleX)”));
// Animate the vertical scale from 0 to the
// randomly-chosen value previously set
DoubleAnimation scaleYAnimation = new DoubleAnimation { From = 0,
Duration = TimeSpan.FromSeconds(.5),
EasingFunction = new QuadraticEase() };
Storyboard.SetTargetProperty(scaleYAnimation, new PropertyPath(
“(UserControl.RenderTransform).(ScaleTransform.ScaleY)”));
// Add the animations to the storyboad
storyboard.Children.Add(topAnimation);
storyboard.Children.Add(leftAnimation);
storyboard.Children.Add(scaleXAnimation);
storyboard.Children.Add(scaleYAnimation);
// Start the storyboard
storyboard.Begin();
}
void RootCanvas_MouseLeftButtonDown(object sender, MouseButtonEventArgs e)
{
// Toggle the application bar visibility
this.ApplicationBar.IsVisible = !this.ApplicationBar.IsVisible;
}
// Application bar handlers
void InstructionsButton_Click(object sender, EventArgs e)
{
this.NavigationService.Navigate(new Uri(“/InstructionsPage.xaml”,
UriKind.Relative));
}
void SettingsMenuItem_Click(object sender, EventArgs e)
{
this.NavigationService.Navigate(new Uri(“/SettingsPage.xaml”,
UriKind.Relative));
The Main Code-Behind 767
}
void AboutMenuItem_Click(object sender, EventArgs e)
{
this.NavigationService.Navigate(new Uri(
“/Shared/About/AboutPage.xaml?appName=Bubble Blower”, UriKind.Relative));
}
}
}

[/code]

The constructor attaches a handler to the Rendering event, which is used for more than just calling Update on XNA’s FrameworkDispatcher. Next, the constructor clips any off-screen content for the benefit of performance and avoiding strange artifacts when navigating away from MainPage. (The page’s ActualWidth and ActualHeight properties can’t be used until layout occurs, so this code simply hard-codes the 480×800 dimensions.) It also initializes and starts the microphone. FrameworkDispatcher.Update is called in the constructor to avoid occasional exceptions while debugging, complaining that Update hasn’t been called. That can happen because during debugging, too much time can pass between starting the microphone and the first call to CompositionTarget_Rendering.
The microphone’s BufferDuration is set to its smallest allowable value—one-tenth of a second. This is the frequency with which the BufferReady event handler will be called. The buffer (the array of bytes) is given the appropriate size for the time period of one-tenth of a second. (This size happens to be 3,200 bytes.)
Inside Microphone_BufferReady, the BufferReady event handler, GetData is called to fill the buffer with a fresh set of bytes representing the last .1 seconds of audio. The currentVolume field is then set to the average magnitude of the samples in the buffer, if the call got any data. This average is calculated in GetAverageVolume. This is certainly not the only approach for determining volume of the audio sample, nor is the value expressed in standard units (like decibels).
GetAverageVolume has a few subtleties. Rather than finding the average magnitude of each byte, which would be meaningless, it must find the average magnitude of each 16-bit (2-byte) audio sample. Therefore, the loop increments i by 2 each time to retrieve each 2-byte chunk. The size of each sample ideally wouldn’t be hardcoded, but rather calculated by dividing a 1-second SampleDuration by SampleRate. However, this would complicate the rest of the logic. BitConverter.ToInt16 is used to convert the two bytes at the current position of the buffer array to a single numeric value. The absolute value of each sample is used because it doesn’t matter whether it is positive or negative, just how large the absolute value is. And if we didn’t take the absolute value of each sample, the resultant average would always be close to zero due to values cancelling each other out!
CompositionTarget_Rendering, besides calling Update, adds a new bubble if the average volume in the current buffer is loud enough. This means that a new bubble is added during every frame that the volume is loud enough. (Adding just one new bubble every relevant .1 second would not be enough.) If the average volume is greater than an even higher threshold, a second bubble is added. Note that this handler continues to get called even after navigating to another page. This is a bit wasteful processing-wise, but it is not a big deal for this app because visits to the other pages should be short-lived. VolumeThreshold is defined as follows with a default value of 400, arrived at by trial and error:
[code]
namespace WindowsPhoneApp
{
public static class Settings
{
public static readonly Setting<int> VolumeThreshold =
new Setting<int>(“VolumeThreshold”, 400);
}
}
[/code]
AddBubble creates a new Bubble and adds it to the canvas just below the bottom of the screen (the page’s height + 50). It picks a random horizontal position for the bubble and a random size (leveraging ScaleTransform). It then dynamically creates and starts a storyboard with four animations. topAnimation moves the bubble to 100 pixels above the top of the screen using a duration relative to the size of the bubble (so smaller bubbles move faster). Therefore, bubbles smaller than 100 pixels travel all the way off the screen, but most bubbles are still partially visible when they “pop.” The pop effect—removing the element when the animation is done without any kind of animated transition—is simple but effective. leftAnimation animates the bubble’s horizontal position from the center of the screen to the randomly chosen position for a hard-coded duration shorter than the length of the vertical animation. This helps to give the effect of all bubbles being blown out of a small wand slightly below the screen. The rest of that effect comes from the last two animations, which make the bubble grow from a size of zero to the randomly chosen size with the same duration as leftAnimation. All animations use QuadraticEase to give a realistic motion, causing the bubbles to linger slightly before they pop.

The Settings Page

The settings page, shown in Figure 34.2, enables adjusting of the “microphone sensitivity.” This changes the value of VolumeThreshold. Although the default value of 400 works well for my phone, other phones from other manufactures might have microphones with slightly different characteristics. In addition, allowing users to adjust this value is helpful to make the app work as well as possible in a variety of environments. Blowing bubbles in a car might require a higher VolumeThreshold value than normal in order to ignore background noise, and blowing bubbles at a sporting event likely requires an even-higher value.

When providing an experience based on a hardware feature—such as the microphone or accelerometer—it’s a good idea to provide end-user calibration, just in case different phone models have slightly different characteristics than the phone(s) used for testing.

FIGURE 34.2 The settings page for Bubble Blower enables changing and resetting the microphone’s sensitivity.

The XAML for Figure 34.2 is shown in Listing 34.4.

LISTING 34.4 SettingsPage.xaml—The Settings User Interface for Bubble Blower

[code]

<phone:PhoneApplicationPage x:Class=”WindowsPhoneApp.SettingsPage” x:Name=”Page”
xmlns=”http://schemas.microsoft.com/winfx/2006/xaml/presentation”
xmlns:x=”http://schemas.microsoft.com/winfx/2006/xaml”
xmlns:phone=”clr-namespace:Microsoft.Phone.Controls;assembly=Microsoft.Phone”
xmlns:shell=”clr-namespace:Microsoft.Phone.Shell;assembly=Microsoft.Phone”
xmlns:local=”clr-namespace:WindowsPhoneApp”
FontFamily=”{StaticResource PhoneFontFamilyNormal}”
FontSize=”{StaticResource PhoneFontSizeNormal}”
Foreground=”{StaticResource PhoneForegroundBrush}”
SupportedOrientations=”PortraitOrLandscape”
shell:SystemTray.IsVisible=”True”>
<Grid Background=”{StaticResource PhoneBackgroundBrush}”>
<Grid.RowDefinitions>
<RowDefinition Height=”Auto”/>
<RowDefinition Height=”*”/>
</Grid.RowDefinitions>
<!– The standard settings header –>
<StackPanel Grid.Row=”0” Style=”{StaticResource PhoneTitlePanelStyle}”>
<TextBlock Text=”SETTINGS” Style=”{StaticResource PhoneTextTitle0Style}”/>
<TextBlock Text=”bubble blower”
Style=”{StaticResource PhoneTextTitle1Style}”/>
</StackPanel>
<!– A stacked text block, slider, and button –>
<StackPanel Grid.Row=”1” Margin=”{StaticResource PhoneMargin}”>
<TextBlock Text=”Microphone sensitivity”
Foreground=”{StaticResource PhoneSubtleBrush}”
Margin=”{StaticResource PhoneMargin}”/>
<Slider x:Name=”SensitivitySlider” Maximum=”4000” LargeChange=”100”
IsDirectionReversed=”True”
Value=”{Binding Threshold, Mode=TwoWay, ElementName=Page}”/>
<Button Content=”reset” Click=”ResetButton_Click”
local:Tilt.IsEnabled=”True”/>
</StackPanel>
</Grid>
</phone:PhoneApplicationPage>

[/code]

The concept of microphone sensitivity is more user-friendly than the idea of a threshold, so the user interface uses this terminology. However, microphone sensitivity is the inverse of the volume threshold, so the slider’s behavior must be reversed by setting its IsDirectionReversed property to true. This makes the slider report its maximum value when it is “empty” (when its thumb is all the way to the left) and decrement its value as the slider fills up and reaches its minimum value.
The slider’s value is allowed to vary from 4,000 to 0. It is given a corresponding LargeChange value of 100, so the value changes at a reasonable rate when the user holds a finger on the slider. Its current value is bound to a Threshold property on the current page instance, with two-way data binding.
As mentioned in previous chapters, it’s always a good idea to have a reset button when sliders are involved.

The code-behind for the settings page is shown in Listing 34.5.

LISTING 34.5 SettingsPage.xaml.cs—The Settings Code-Behind for Bubble Blower

[code]

using System.Windows;
using Microsoft.Phone.Controls;
namespace WindowsPhoneApp
{
public partial class SettingsPage : PhoneApplicationPage
{
public SettingsPage()
{
InitializeComponent();
}
// Simple property bound to the slider
public int Sensitivity
{
get { return Settings.VolumeThreshold.Value; }
set { Settings.VolumeThreshold.Value = value; }
}
void ResetButton_Click(object sender, RoutedEventArgs e)
{
this.SensitivitySlider.Value = Settings.VolumeThreshold.DefaultValue;
}
}
}

[/code]

Sensitivity is not a dependency property, so changes to its value do not automatically update the slider; changes only flow from the slider to the property. (The initial value is fetched by the slider when the page loads, however.) This is why ResetButton_Click updates the slider’s value rather than Sensitivity. By doing so, it updates the user interface and the value of the VolumeThreshold setting with one line of code.

The Finished Product