Blind source separation using FastICA in Scikit Learn - Coding

FastICA is the most popular method and the fastest algorithm to perform Independent Component Analysis. It can be used to separate all the individual signals from a mixed signal.

Independent Component Analysis(ICA) is a method where it performs a search for finding mutually independent non-Gaussian latent variables. Here, the components of the multivariate data are assumed to be linear combinations of them.

FastICA is of two types.

Deflation-based FastICA where the components are found in one by one manner.
Symmetric FastICA where the components are found simultaneously.

FastICA has some prominent features it can use any nonlinearity function and optimizes the extraction order in the deflation-based version.

Mathematical Approach

FastICA can be used to separate all the individual signals from a mixed signal. So let us assume a set of individual source signals is $f(s)$ where $f(s) = (f_1(s),...,f_n(s))^{T}$ . These signal can be mixed using a matrix $G = [g_{ij}] ∈ R^{m×n}$ . After mixing the signal it produces a mixed signal $x(s) = (x_1(s),......,x_m(s))^{T}$ .

Generally $n=m$ .
If $m>n$ , then the system of equations is overdetermined and the conventional linear method is used to unmix the signals.
If $n>m$ , then a non-linear method must be implemented because the system performs underdetermined.

Now FastICA algorithm is used to unmix these signals as there are some multidimensional signals that can be present on the mixed signals. Blind source separation can effectively unmix these signals with the help of the FastICA algorithm by determining an un-mixing matrix $U = [U_{ij}] ∈ R^{n×m}.$ Then it recovers the approximation of the original signals,

$y(s) = (y_1(s),...,y_n(s)) \\ y(s) = U\cdot x(s)$

We will see the steps of FastICA algorithms later on in this article.

Applications of FastICA

FastICA is used in various fields in real-life events like a mixed sound wave from multiple independent sources we can easily find out each signal wave along with its source by using FastICA. Also let if a room is full of people and there are a certain number of microphones, we can use FastICA to identify everyone from the mixed sound waves captures by microphones.

Blind Source Separation

When all the sources of the signals and also the mixing methodology is completely unknown, the Blind source separation technique comes into consideration. From the name, we can understand it separates signals ‘blindly’ i.e. maximum numbers are not known of the mixed signals but we need to separate them. In real life, there are many situations where it is necessary to separate the mixed signal into it’s each signal. Also, there is a high possibility that the mixed signal contains noises. Blind Source Separation is used to handle these types of situations where separation is important but most things are not known about its input signal.

Application of Blind Source Separation:

It is used in many domains like Audio source separation(speech separation and music industry), Medical tests(electroencephalography, Magnetic Resonance Imaging(MRI), electrocardiography, etc.), and communication applications.

FastICA Algorithm

FastICA consists of mainly three steps:-

1. Pre-whitening the data:

If the input data matrix is $X = (x_{ij})\; \epsilon \; R^{N\times M}$ then it should be centered and whitened before applying the FastICA algorithm on it.

Centering: Centering the data means subtracting the mean value from each variable, it will make the expected value for each row equal to 0. Suppose $X_{i}$ denotes the $i^{th}$ row having $M$ number of variables then $x_{ij}$ element will be equal to the $x_{ij} = x_{ij} - \frac{1}{M}\sum_{j}x_{ij}$ . where $i \;\epsilon \;(1,2,...,N)$ and $j \; \epsilon \; (1,2,...,M)$ .
Whitening: In whitening, we linearly transformed the centered data in such a way that they become uncorrelated and the variance between them will be equal to 1. It is done by Eigen Value Decomposition of Covariance Matrix of centered data X. i.e. $E\left \{ XX^{T}\right \} = EDE^{T}$ . Where E is the eigen vector matrix and D is the diagonal matrix of eigen values.

2. For single unit:

After that , the iterative algorithm finds the direction for the weight vector that maximizes a measure of non-Gaussian plane of the projection on the pre-whitened data matrix. In this case it randomizes initial weight vector then perform averaging over all column-vectors of matrix X. If converges then weight vector(w) value is calculated if not converged then it re-performs the averaging. For calculating non-Gaussianity FastICA uses a nonlinear nonquadratic function f(q) and it’s first derivative(g(q)) , second derivative(g'(q)).

Non-Gaussianity:- A Gaussian function measures the data is following normal distribution or not. If the mean or noise of dataset varies with time then there is a high chance that it triggers some errors as the variables become non-stationary i.e. the value of variables change with time. So, Non-Gaussianity is a corrective technique or distribution(Kurtosis) to reduce this error. In Gaussian variables this kurtosis value is zero and more high when it tending to follow non-gaussianity.

For general purpose uses the function will be –>

$f(q) = \log\cosh(q) \\ f'(q) = tanh(q) \\ f''(q) = 1-tanh^2(q)$

$f(q) = -e^{\frac{-u^2}{2}} \\ f'(q) = ue^{\frac{-u^2}{2}} \\ f''(q) = (1-u^2)e^{\frac{-u^2}{2}}$

3. For several unit:

Only one weight vector can be estimated by the single unit iterative algorithm which extracts only a single component. For estimating additional components that are mutually independent(i.e. a certain components performance will not be affected by other components occurrence) requires repeating the algorithm to obtain linearly independent projection vectors. In this case total number of desired components along with pre-whitened matrix is given as input the algorithm gives the output as Un-mixing matrix(each column projects onto independent component) and Independent components matrix. Pseudo code for this algorithm is given below–>

for p in 1 to C:

weight vector $w_p\leftarrow$ [Tex]         [/Tex] Random vector of length N

while $\text{[math]}$ changes

   $w_{p}\leftarrow\frac{1}{M}\ Xg(w_p^{T}X)^{T}\ - \frac{1}{M}\ g'(w_p^{T}X){1_M}{w_p}$

   $w_p\leftarrow\ w_p - \sum_{j=1}^{p-1}\ (w_p^{T}w_j)w_j$

$w_p\leftarrow\frac{w_p}{||w_p||}$

output $W\leftarrow\ [w_1,....,w_C]$

output $S\leftarrow\ W^{T}X$

Note: Here C= total number of desired components, X = pre-whitened matrix, W = unmixing matrix , M = independent components matrix.

Now we will see a example which depicts how FastICA is can be used for Blind source separation step by step–>

To estimate the sound sources effectively from given noisy measurements the Independent component analysis (ICA) is used. Let us assume that three different type of musical instrument is playing music simultaneously and three microphones recording the mixed signals. ICA is used here to recover the sources of the signals i.e. which signal is played by which instrument. Here we will also show that PCA fails to recover the instruments since the related signals reflect non-Gaussian processes. We will see the implementation step by step–>

Importing Python libraries and generating sample data

By using python libraries like NumPy, SciPy, Matplotlib and SKlearn we can perform complex computations easily and effectively. After that three types of signals(sinusoidal, square and saw-tooth) we will generate.

Python3

# importing libraries
import numpy as np
from scipy import signal
import matplotlib.pyplot as plt
from sklearn.decomposition import FastICA, PCA
# generating sample data
np.random.seed(20)
numberOfSamples = 10000
timeRange = np.linspace(0, 8, numberOfSamples)
 
# Sinusoidal signal
signal1 = np.sin(2 * timeRange)  
 
# Square signal
signal2 = np.sign(np.sin(3 * timeRange))  
 
# Saw-tooth signal
signal3 = signal.sawtooth(2 * np.pi * timeRange)  
signalSummation = np.c_[signal1, signal2, signal3]
 
# Adding some noise
signalSummation += 0.4 * \
    np.random.normal(size=signalSummation.shape)  
 
# Standardization of data
signalSummation /= signalSummation.std(axis=0)  
# Mixing the data
mixMatrix = np.array(
    [[1, 1, 1], [0.8, 2, 1.2], [1.6, 1.2, 2.4]])  
 
# Generate observations
obsvGenerate = np.dot(signalSummation, mixMatrix.T)  

Computing ICA

Now we will compute ICA model using FastICA and also as given earlier we will also compute PCA model for showing the comparison.

Python3

# Fitting ICA and PCA models
# Compute ICA
ica = FastICA(n_components=3, whiten="arbitrary-variance")
signalRecont = ica.fit_transform(obsvGenerate)  # Reconstruct signals
mixMatrixEst = ica.mixing_  # Get estimated mixing matrix
assert np.allclose(obsvGenerate, np.dot(
    signalRecont, mixMatrixEst.T) + ica.mean_)
# compute PCA
pca = PCA(n_components=3)
# Reconstruct signals based on orthogonal components
orthosignalrecont = pca.fit_transform(obsvGenerate)

Visualizing results and comparison in graphical format

Now we will plot the graph with our achieved values and can under stand the efficiency of ICA for blind source separation of signals as well as PCA has failed to do this.

Python3

# Plot results
plt.figure()
models = [obsvGenerate, signalSummation, signalRecont, orthosignalrecont]
names = ["Observations (mixed signal)",
         "True Sources", "ICA recovered signals", 
         "PCA recovered signals",
         ]
colors = ["yellow", "green", "cyan"]
 
for ii, (model, name) in enumerate(zip(models, names), 1):
    plt.subplot(4, 1, ii)
    plt.title(name)
    for sig, color in zip(model.T, colors):
        plt.plot(sig, color=color)
 
plt.tight_layout()
plt.show()

Output:

Blind source separation

From the graph, it is very much clear that ICA has separated all three signals efficiently but PCA can’t.

Reffered: https://www.geeksforgeeks.org

AI ML DS

Related
Machine Learning Model Evaluation
Derivative of the Sigmoid Function
Prior and Posterior Gaussian Process for Different kernels in Scikit Learn
Classifier Comparison in Scikit Learn
Implementing SVM and Kernel SVM with Python's Scikit-Learn

Type:	Geek
Category:	Coding
Sub Category:	Tutorial
Uploaded by:	Admin
Views:	13