Almost any application related to sound capture and audio communication, the microphone not only picks up the signal of interest, but also picks up noise that lowers the quality of the signal of interest for the far-end user. In the acoustic environment there are many sources of noise. There is ambient background noise, local interferers (local speakers) besides the speaker of interest, and acoustic echo feedback from the loudspeaker to microphone.
Each noise source has their individual solution. For ambient noise, it is common to use spectral subtraction, which relies on estimating the noise spectrum. For rejection or separation of local speakers, beamforming methods such as the generalized sidelobe canceller are used. The generalized sidelobe canceller is used to focus the main beam in direction of signal of interest, and to create null beams in the direction of interferes. And for acoustic echoes, a system identification adaptive filter is used to cancel the acoustic echoes.
Combining all three digital signal processing techniques in one system is not a trivial task. The order in which to implement these tools is an academic debate. It is common practice to perform the ambient noise reduction as a post-processing technique to cleanup what the AEC and beamformer leave behind. And generally it is not good practice to introduce noise reduction before adaptive linear filters, as it is possible for the spectral subtraction to introduce nonlinearities into the system. Therefore, the primary debate is whether the acoustic echo cancellation (AEC) or the beamforming should come first.
Due to the high computational complexity to implement, an AEC first solution is usually avoided. But implementing a beamformer first solution creates further difficulties for the AEC. To explain further the beamformer spatially filters out signals not from the received signal, thus reducing the power ratio of the acoustic echoes to the desired signal. Now, not only does the AEC have to model echo path but the beamforming vector as well. This condition makes estimating the echo path more difficult, and slows the convergence of AEC.
To make an educated decision on what tool to use first, one has to first look at the acoustic environment in which their system lies, and processing resources available to them. For example, if acoustic echoes are the most disruptive source of noise, then AEC should come first. This will allow AEC to quickly track and model the echo paths for good cancellation. If processing resources are limited and acoustic echoes is not the most disruptive source of the noise to communication, then beamforming first should be applied.