Fixing the Mango Microphone issue – part 2

In the first part of this short series I have demonstrated that Mango has somewhat broken XNA Microphone object, at least when it is used in Silverlight. Even the standard Microsoft’s own Silverlight Microphone Sample generates a broken sound record with some data missing, if you change the buffer length from default 500 ms to 100 ms.

Look at the sound sample recorded with 500 ms buffer on Mango, or with 100 ms buffer on NoDo:

Joined 100 ms NoDo or 500 ms Mango Debug Wave records on both phone and emulator look similar to this graph (click on it for a bigger picture).

The same sound sample recorded with 100 ms buffer on Mango:

Joined 100 ms Mango Debug Wave records on both phone and emulator look similar to this one. The wider (louder) marker waves are unevenly spaced, so some data is missing.

My next step was to measure the real rhytm of the buffers generated by the Microphone. Note that you cannot use System.DateTime.Now property, because of its ca 50 ms resolution. Instead, we will use System.Diagnostics.Stopwatch object, which provides better than 1 ms accuracy.

In theory, the Microphone should fire the BufferReady event every 100 ms. But the reality is quite different.

This is the list of the intervals – measured in milliseconds – between the subsequent 100 ms BufferReady events in the (emulated) NoDo:

101
51
151
101
51
150
51
152
101
51
151
51
151
51
101
151
51
152
51
101
152
51

As you can see, there is a regular pattern of the three 100+, 50+ and 150+ ms intervals, with some swaps. Ten intervals usually make ca 1011 ms. The length of the buffer is always 3200 B and the buffer contains 1600 x 16bit samples, which make 100 ms of sound. Buffers are perfectly connected to each other and no sample is missing or superfluous. (I haven’t yet cracked the interesting question where are going these surplus 11 milliseconds.)

Let’s switch to Mango:

97
139
69
105
68
135
70
104
138
69
139
35
138
138
35
139
69
105
137
70
138
69

This is recorded on Samsung Omnia 7. The emulator produces different values, but with a similar pattern: This time there are four, not three typical lengths – ca 100, ca 140, ca 70, ca 35. The sum of ten intervals is usually ca 1005 ms now, so it’s better, but the output data is incomplete. It seems that the intervals longer than 100 ms overflow the buffer.

The Microphone object is part of Microsoft.Xna.Framework.Audio namespace. It is dependent on the XNA Framework game loop. You can use it in the Silverlight Windows Phone project, but you have to simulate the game loop with a code like this:

DispatcherTimer dt = new DispatcherTimer();
dt.Interval = TimeSpan.FromMilliseconds(33);
dt.Tick += new EventHandler(dt_Tick);
dt.Start();

void dt_Tick(object sender, EventArgs e)
{
     try { FrameworkDispatcher.Update(); }
     catch { }
}

33 milliseconds is a timer interval used in the Microsoft’s Silverlight Microphone Sample. I have used the same value, but my next step was to experiment with different timer intervals. Indeed, the resulting Microphone tick lengths were somewhat different, but there was always some data missing, overflowing the buffer – the fundamental issue hasn’t disappeared.

Perhaps you already suspect the solution: We just need a bigger buffer! So, let’s record it with a 6400 B buffer.

Folks, this looks promising! At first look, when we erase the gaps, and join the signal segments, the resulting record may be perfect… Let’s try it in Excel…

Oops! 😦 What could have gone wrong???

Why are some segments too long, but no segment is too short? These 35 ms segments should have been short, I guess? Wait! Do I clear the buffer every time? No, it wasn’t necessary in NoDo, because the buffer was always filled up to its fullness in every Microphone tick. In Mango, the Microphone buffer must be cleared before every BufferReady event. 😦 Let’s go on!

Bingo!

So the winning process is:

1. Allocate bigger than nominal buffers.
2. Clear them.
3. Fill them.
4. Join only the filled parts of them.

This seems weird. The Microphone object has GetSampleSizeInBytes method. So we should just invoke this method after (or before) every tick and it should return the actual buffer size… but no, it doesn’t work this way. I have played with it for some time, but it always returns just the BufferDuration x 3200 value. It doesn’t reflect the real tick record size, just its ideal size, which is always matched in NoDo, but never in Mango…

The issue with the XNA Microphone object in Mango with 100 ms buffer is that the real buffer length is variable and there is no working API to get its length!

The good news is that no data is really lost. We just don’t know we actual size of the one tick record, so we have to figure it with some workaround. The way to go is to count zeros from the end of the buffer backwards. It may be slightly wrong in some cases, when the sound samples are actually zero at the end of the partial records, for example when the wave crosses the X axis and has exactly zero value at the end of the record. But these cases are quite rare and we can neglect them – at least if we are building a tuner.

In the next and the last part of this mini-series I will publish the source code of the Carousel object, which uses the above described methods to deliver solid data from the capricious Mango Microphone. The living proof of its reliability is my app Accurate Tuner Pro, probably the best pitch tuner for Windows Phone.

INSTALL ACCURATE TUNER PRO UNLIMITED TRIAL FOR FREE

What should Microsoft do now? Change GetSampleSizeInBytes of the Microphone object to return the actual size of the just recorded buffer.

Let’s hope it’s fixed in Tango. But still, Tango rollout is going to take some time, so the Carousel object may help you.

To be continued.

Fixing the Mango Microphone issue – part 1

Last september, when Windows Phone 7.5 Mango update was released, it broked my Accurate Tuner Beta. I’ve analysed the issue and was able to release the update that fixed it.

The two videos below illustrate the situation. First the trouble (Mango is installed on the phone, while the emulator still uses NoDo and works properly):

Then the solution:

I promised to explain it in detail back then, but I lacked the time to do it. Now, when I have sent Accurate Tuner Pro to the Marketplace certification, I can fulfill my promise and write a short series about this issue.

Perhaps it is not so important now, because new Tango update is behind the corner and may fix it. In the last part of this series (3rd or 4th) I will publish the source code for the Carousel object, which is used by Accurate Tuner for harvesting solid sound data from somewhat broken Mango microphone.

To map the problem I have created a simple tone generator. It produces a 500 Hz wave (B4 + 21 cents) with a specific shape: “Normal” wave has the amplitude of  5.000, while every 10th wave has the amplitude of 15.000. The resulting “hedge clipper” shape makes a growling sound and looks like this:

Source Debug Wave 500 Hz with 50 Hz markers

Click on it for a bigger picture.

Wave parameters:

• Sampling rate: 16 000 Hz – the default sampling rate of Windows Phone OS 7.0 and 7.1
• Base frequency: 500 Hz, wave length 32 samples
• Frequency of the markers: 50 Hz, wave length 320 samples

Tone Generator source code:

/// <summary>
/// Creates the DEBUG Wave with the debug parameters
/// </summary>
public void CreateDebugWave()
{
    // DEBUG PARAMETERS
    _samplingRate = 16000;
    _frequency = 500;
    double _normalAmplitude = _amplitude = 5000;
    double markerWaveAmplitude = 15000;
    int markerWaveStep = 10;

    int waveLength = (int)(_samplingRate / _frequency);
    int markerSampleStep = markerWaveStep * waveLength;

    double doubleSample = 0;
    Int16 intSample = 0;
    byte[] byteSample = new byte[2];
    double b = (Math.PI * 2) * _frequency / _samplingRate;

    int j = 0;

    // The wave buffer is one second long
    //
    // This works for whole number of waves (= integer 
    // frequencies) only. 500 Hz or 440 Hz is OK, while
    // 261.63 Hz will make a snapping sound every second.
    //
    for (int i = 0; i < _samplingRate; i++)
    {
        // Marker wave creation
        //
        // This is simple and clear, but not very efficient.
        // It doesn't matter here.
        //
        if (i % markerSampleStep == 0)
            _amplitude = markerWaveAmplitude;
        else
            if (i % markerSampleStep == waveLength)
                _amplitude = _normalAmplitude;

        doubleSample = Math.Sin(i * b) * _amplitude;
        intSample = (Int16)Math.Round(doubleSample);
        byteSample = BitConverter.GetBytes(intSample);

        _buffer[j] = byteSample[0];
        _buffer[j + 1] = byteSample[1];

        // DEBUG For debug visualization purposes only
        _bufferInt16[i] = intSample;

        j += 2;
    }
}

/// <summary> 
/// Creates and plays the wave.
/// </summary> 
public void Start()
{
     // DEBUG 
    CreateDebugWave();

    SoundEffect sound = new SoundEffect(_buffer,
                        _samplingRate, AudioChannels.Mono);

    if (_isPlaying)
    {
        _soundInstance.Stop();
        _soundInstance.Dispose();
        _soundInstance = null;
    }

    _soundInstance = sound.CreateInstance();
    _soundInstance.IsLooped = true;
    _soundInstance.Volume = _volume;

    _isPlaying = true;
    _soundInstance.Play();
}

/// <summary>
/// Stops playing the wave.
/// </summary>
public void Stop()
{
    if (!_isPlaying)
        return;

    _soundInstance.Stop();
    _isPlaying = false;
}

By the way, the more universal version of this tone generator is included in Accurate Tuner Pro.

Now the interesting part begins: To avoid any mistake in my code I’ve used the standard Silverlight Microphone Sample provided by Microsoft. It uses 500 ms long sound buffer by default. This is the record it produces on NoDo (recorded in the 7.0 Emulator of the 7.1 SDK):

Debug Wave recorded by NoDo 7.0 Emulator in SDK 7.1

I’ve carefully reviewed much longer part of the record and found no irregularities. This data is solid and can be safely used by Accurate Tuner for two purposes:

• Wave vizualization, which is phase-normalized for the stroboscopic efect usable for fine tuning.
• Fast Fourier Transformation algorithm, which analyses component frequencies for rough tuning.

Next, let’s run it on Mango:

Debug Wave recorded by Mango 7.1 Samsung Omnia 7 with 500 ms buffer

The growing amplitude is probably caused by the automatically adjusting microphone sensitivity of the phone and is not interesting for us now, it reaches some maximum amplitude and then stays constant. Another thing is more important – I haven’t found anything wrong with this signal too.

Is this the way Accurate Tuner records sound? Almost. 500 ms sound buffer is good enough for most recording purposes, but it is too long for tuners or sound visualizations, because you need fast response in these cases. The shortest buffer length allowed by WP 7.0 and 7.1 is 100 ms and Accurate Tuner uses this buffer length.

So let’s change the buffer length to 100 ms. The picture is wider now, so click on it to maximize it:

Debug Wave recorded by Mango 7.1 Samsung Omnia 7 with 100 ms buffer

Folks, this is clearly wrong!

Apparently, some sound data is lost!

This explains everything: While common sound visualizations used for estetic purposes only will still look as usual, a stroboscopic animation will be inevitably completely broken. FFT will provide more interesting and confusing results, as there are still big portions of correct waves here, but these portions are phase-shifted. The final results will be unstable and very inaccurate, but not completely wrong.

Which is exactly the behavior of the original Accurate Tuner Beta algorithms.

(I’m skipping the chart of the NoDo 100 ms buffer, but believe me the record is perfect and looks just exactly as the chart with the 500 ms buffer. I would also point out that the described issue is present in the WP 7.1 Emulator too, so it isn’t specific for Samsung phones.)

What to do with it??? FFT will work correctly, if only “safe part” of the buffer is used, for example 50 ms. But it will be extremely inaccurate. In fact, even 100 ms is not enough and several buffers must be joined for the analysis to be meaningful. Or I can change the buffer size to 500 ms, but the FTT results would be very delayed in such case and almost unusable. And the wave visualization delayed by 0.5 s will be just comical.

We will see next time.

To be continued.

Accurate Tuner Beta finally available in the Marketplace

My new app has found its way into the Windows Phone Marketplace. Accurate Tuner is a chromatic instrument tuner, which combines two classical pitch tuning methods: Needle and stroboscope.

The needle enables fast and comfortable rough tuning, while two optional stroboscopic animations facilitate fine tuning. Both methods are used at the same time, so you are not forced to choose just one of them.

Detected notes are shown on a musical staff with highlighted positions used by a selected musical instrument. Adjustable sensitivity may help in not so quiet environments.

With an integrated help system you won’t end up lost. Advanced settings take care of specific needs. It is simply worth a try. Download this free beta now!