因子分析Factor analysis

分手后的思念是犯贱 2022-07-28 01:50 249阅读 0赞

**简介：**本文主要介绍EM算法求解因子分析问题

## 因子分析Factor analysis ##

在文章  [EM算法][EM] 求解混合高斯模型时，通常假设拥有足够多的样本去构造这个混合高斯分布，即样本数量n要远大于样本维数d： ![gif.latex_n_5Cgg_20d][] 如果样本数量小于样本维数，那么协方差矩阵![gif.latex_5CSigma][] 是奇异矩阵,那么 ![gif.latex_5CSigma_5E_7B-1_7D][] 和![2_7D][]都无法计算。在因子分析中，一个d维的向量通常由一个k维向量生成，通常k远小于d。具体模型如公式1所示:

![gif.latex_5Cdpi_7B200_7D_20_5Ctiny_20x_3D_5CLambda_20z_5Cvarepsilon][] **公式1**

上式中d x k维矩阵 ![gif.latex_5CLambda][] 称为因子载荷矩阵factor loading matrix,k维向量z称为因子factors,d维向量![gif.latex_5Cvarepsilon][]是满足均值为0，对角协方差矩阵的高斯分布的噪声，那么根据公式1该模型写成概率形式如公式2所示：

![gif.latex_5Cdpi_7B200_7D_20_5Ctiny_20_5Cbegin_7Bmatrix_7D_20P_28z_29_3DN_280_2CI_29_5C_5C_20P_28_5Cvarepsilon_20_29_3DN_280_2C_5CPsi_20_29_5C_5C_20P_28x_5Cmid_20z_29_3DN_28_5CLambda_20z_2C_5CPsi_20_29_20_5Cend_7Bmatrix_7D][] **公式2**

该模型如图1所示：

![SouthEast][]

**图1** 因子分析生成模型

根据 [Pattern Recognition and Machine Learning][] 已知公式2可得x的边缘概率和已知z的条件概率如公式3和公式4所示(详见原书公式2.113-2.117):

![gif.latex_5Cdpi_7B200_7D_20_5Ctiny_20P_28x_29_3DN_280_2C_5CLambda_20_7B_5CLambda_7D_27_20_5CPsi_20_29][] **公式3**

![gif.latex_5Cdpi_7B200_7D_20_5Ctiny_20_5Cbegin_7Balign_7D_20P_28z_5Cmid_20x_29_20_26_3D_20N_28_28I_7B_5CLambda_20_7D_27_5CPsi_5E_7B-1_7D_20_5CLambda_20_29_5E_7B-1_7D_5CLambda_20_27_5CPsi_5E_7B-1_7Dx_2C_28I_7B_5CLambda_20_7D_27_5CPsi_5E_7B-1_7D_20_5CLambda_20_29_5E_7B-1_7D_29_5C_5C_20_26_3D_20N_28_7B_5CLambda_20_7D_27_28_5CLambda_7B_5CLambda_20_7D_27_5CPsi_29_5E_7B-1_7Dx_2CI-_7B_5CLambda_20_7D_27_28_5CLambda_7B_5CLambda_20_7D_27_5CPsi_29_5E_7B-1_7D_5CLambda_29_20_5Cend_7Balign_7D][] **公式4**

根据公式3采用[极大似然估计法maximum-likelihood estimation (MLE)][maximum-likelihood estimation _MLE] 可以的到似然函数如公式5所示：

![2_7D_7De_5E_7B-_5Cfrac_7B1_7D_7B2_7D_28x_7Bi_7D-_5Cmu_29_27_28_5CLambda_20_7B_5CLambda_20_7D_27_5CPsi_29_5E_7B-1_7D_28x_7Bi_7D-_5Cmu_29_7D][] **公式5**

直接对公式5求导并令其等于0很难得到结果，而EM算法可以很好的解决这个问题。

回顾 [EM算法][EM] (这里z为隐含变量)，迭代过程如下所示：

*  E-Step:

![gif.latex_q_28z_7Bi_7D_29_20_5Cleftarrow_20N_28_7B_5CLambda_20_7D_27_28_5CLambda_7B_5CLambda_20_7D_27_5CPsi_29_5E_7B-1_7Dx_7Bi_7D_2CI-_7B_5CLambda_20_7D_27_28_5CLambda_7B_5CLambda_20_7D_27_5CPsi_29_5E_7B-1_7D_5CLambda_29][]**公式6**

*  M-Step:

![gif.latex_5Cdpi_7B200_7D_20_5Ctiny_20_5CTheta_20_5C_7B_5CLambda_20_2C_5CPsi_20_5C_7D_5Cleftarrow_20_5Cunderset_7B_5CTheta_7D_7Barg_20max_7D_5Csum_7Bi_3D1_7D_5E_7Bn_7D_5Cint_7Bz_7Dq_28z_29ln_5C_2CP_28x_2Cz_5Cmid_20_5CTheta_29_5Cmathrm_7Bd_7Dz][]**公式7**

上述算法E-Step直接利用公式4的结论，下面详细分析M-Step。把要求的最大化概率推到导公式8所示，其中K是和参数无关的常量：

![gif.latex_5Cdpi_7B200_7D_20_5Ctiny_20_5Cbegin_7Balign_7D_20_5Csum_7Bi_3D1_7D_5E_7Bn_7D_5Cint_7Bz_7Dq_28z_29ln_5C_2CP_28x_2Cz_5Cmid_20_5CTheta_29_5Cmathrm_7Bd_7Dz_20_26_3D_20_5Csum_7Bi_3D1_7D_5E_7Bn_7D_5Cint_7Bz_7Dq_28z_29_5Bln_5C_2CP_28x_5Cmid_20z_3B_5CLambda_20_2C_5CPsi_20_29_ln_5C_2CP_28z_29_5D_5Cmathrm_7Bd_7Dz_5C_5C_20_26_3D_20_5Csum_7Bi_3D1_7D_5E_7Bn_7DE_7Bz_5Csim_20q_28z_29_7D_5Bln_5C_2CP_28x_5Cmid_20z_3B_5CLambda_20_2C_5CPsi_20_29_ln_5C_2CP_28z_29_5D_5C_5C_20_26_3D_20_5Csum_7Bi_3D1_7D_5E_7Bn_7DE_7Bz_5Csim_20q_28z_29_7D_5Bln_5C_2CP_28x_5Cmid_20z_3B_5CLambda_20_2C_5CPsi_20_29_5D_K_5C_5C_20_5Cend_7Balign_7D][] **公式8**

去掉常量，最终需要最大化的式子如公式9所示：

![2_7D_7De_5E_7B-_5Cfrac_7B1_7D_7B2_7D_28x-_5CLambda_20z_29_27_5CPsi_5E_7B-1_7D_28x-_5CLambda_20z_29_7D_5D_5C_5C_20_26_3D_20_5Csum_7Bi_3D1_7D_5E_7Bn_7DE_5B-_5Cfrac_7B1_7D_7B2_7Dln_5C_2C_5Cleft_20_7C_20_5CPsi_20_5Cright_20_7C-_5Cfrac_7Bn_7D_7B2_7Dln_5C_2C_282_5Cpi_29-_5Cfrac_7B1_7D_7B2_7D_28x-_5CLambda_20z_29_27_5CPsi_5E_7B-1_7D_28x-_5CLambda_20z_29_5D_5C_5C_20_5Cend_7Balign_7D][] **公式9**

*  求参数![gif.latex_5Cdpi_7B200_7D_20_5Ctiny_20_5CLambda][]

根据[矩阵迹Trace性质][Trace]![gif.latex_5Cdpi_7B200_7D_20_5Ctiny_20tr_5C_2Ca_3Da_2Ctr_5C_2CAB_3Dtr_5C_2CBA][] 和[矩阵求导的性质][Link 1]![gif.latex_5Cdpi_7B200_7D_20_5Ctiny_20_5Cpartial_20trABA_5E_7BT_7DC_5Csetminus_20_5Cpartial_20A_3DCAB_C_5E_7BT_7DAB][] ,公式9对![gif.latex_5Cdpi_7B200_7D_20_5Ctiny_20_5CLambda][] 求偏导如下：

![gif.latex_5Cdpi_7B200_7D_20_5Ctiny_20_5Cbegin_7Balign_7D_20_5Cbigtriangledown_7B_5CLambda_20_7D_20L_28_5CLambda_20_2C_5CPsi_29_20_26_3D_20_5Cbigtriangledown_7B_5CLambda_20_7D_5Csum_7Bi_3D1_7D_5E_7Bn_7DE_5B-_5Cfrac_7B1_7D_7B2_7Dln_5C_2C_5Cleft_20_7C_20_5CPsi_20_5Cright_20_7C-_5Cfrac_7Bn_7D_7B2_7Dln_5C_2C_282_5Cpi_29-_5Cfrac_7B1_7D_7B2_7D_28x-_5CLambda_20z_29_27_5CPsi_5E_7B-1_7D_28x-_5CLambda_20z_29_5D_5C_5C_20_26_3D_20_5Cbigtriangledown_7B_5CLambda_20_7D_5Csum_7Bi_3D1_7D_5E_7Bn_7DE_5B-_5Cfrac_7B1_7D_7B2_7D_28x-_5CLambda_20z_29_27_5CPsi_5E_7B-1_7D_28x-_5CLambda_20z_29_5D_5C_5C_20_26_3D_20_5Cbigtriangledown_7B_5CLambda_20_7D_5Csum_7Bi_3D1_7D_5E_7Bn_7DE_5B-_5Cfrac_7B1_7D_7B2_7Dx_27_5CPsi_5E_7B-1_7Dx_5Cfrac_7B1_7D_7B2_7Dx_27_5CPsi_5E_7B-1_7D_5CLambda_20z_5Cfrac_7B1_7D_7B2_7Dz_27_5CLambda_20_5CPsi_5E_7B-1_7Dx-_5Cfrac_7B1_7D_7B2_7Dz_27_5CLambda_27_5CPsi_5E_7B-1_7D_5CLambda_20z_5D_5C_5C_20_26_3D_20_7B_5Ccolor_7BBlue_7D_20_5Cbigtriangledown_7B_5CLambda_20_7D_5Csum_7Bi_3D1_7D_5E_7Bn_7DE_5Btr_28z_27_5CLambda_20_5CPsi_5E_7B-1_7Dx_29-_5Cfrac_7B1_7D_7B2_7Dtr_28z_27_5CLambda_27_5CPsi_5E_7B-1_7D_5CLambda_20z_29_5D_7D_5C_5C_20_26_3D_20_7B_5Ccolor_7BRed_7D_20_5Cbigtriangledown_7B_5CLambda_20_7D_5Csum_7Bi_3D1_7D_5E_7Bn_7DE_5Btr_28_5CLambda_20_5CPsi_5E_7B-1_7Dxz_27_29-_5Cfrac_7B1_7D_7B2_7Dtr_28_5CLambda_27_5CPsi_5E_7B-1_7D_5CLambda_20zz_27_29_5D_7D_5C_5C_20_26_3D_20_7B_5Ccolor_7BGreen_7D_20_5Csum_7Bi_3D1_7D_5E_7Bn_7DE_5B_5CPsi_5E_7B-1_7Dxz_27-_5CPsi_5E_7B-1_7D_5CLambda_20zz_27_5D_7D_20_5Cend_7Balign_7D][] **公式10**

带颜色部分利用了矩阵相关性质,令其等于0我们可以得到：

![gif.latex_5Cdpi_7B200_7D_20_5Ctiny_20_5Csum_7Bi_3D1_7D_5E_7Bn_7D_5CLambda_20E_7B_5Csim_20z_7D_5Bzz_27_5D_3D_5Csum_7Bi_3D1_7D_5E_7Bn_7DE_7B_5Csim_20z_7D_5Bxz_27_5D][]

整理可得 ![gif.latex_5Cdpi_7B200_7D_20_5Ctiny_20_5CLambda][] 如公式11所示：

![gif.latex_5Cdpi_7B200_7D_20_5Ctiny_20_5CLambda_3D_5Csum_7Bi_3D1_7D_5E_7Bn_7DxE_5Bz_27_5D_28_5Csum_7Bi_3D1_7D_5E_7Bn_7DE_5Bzz_27_5D_29_5E_7B-1_7D][] **公式11**

根据公式4，E\[z\]如公式12所示：

![gif.latex_5Cdpi_7B200_7D_20_5Ctiny_20E_5Bz_5D_3D_7B_5CLambda_20_7D_27_28_5CLambda_7B_5CLambda_20_7D_27_5CPsi_29_5E_7B-1_7Dx][] **公式12**

根据公式4，E\[zz’\]如公式13所示：

![gif.latex_5Cdpi_7B200_7D_20_5Ctiny_20_5Cbegin_7Balign_7D_20E_5Bzz_27_5D_20_26_3D_20cov_5Bz_5D_E_5Bz_5DE_5Bz_27_5D_5C_5C_20_26_3D_20I-_7B_5CLambda_20_7D_27_28_5CLambda_7B_5CLambda_20_7D_27_5CPsi_29_5E_7B-1_7D_5CLambda_E_5Bz_5DE_5Bz_27_5D_20_5Cend_7Balign_7D][] **公式13**

*  求参数![gif.latex_5Cdpi_7B200_7D_20_5Ctiny_20_5CPsi][]

公式9对 ![gif.latex_5Cdpi_7B200_7D_20_5Ctiny_20_5CPsi][] 求偏导如下：

![gif.latex_5Cbegin_7Balign_7D_20_5Cbigtriangledown_7B_5CPsi_5E_7B-1_7D_20_7D_20L_28_5CLambda_20_2C_5CPsi_20_29_20_26_3D_20_5Cbigtriangledown_7B_5CPsi_5E_7B-1_7D_20_7D_5Csum_7Bi_3D1_7D_5E_7Bn_7DE_5B-_5Cfrac_7B1_7D_7B2_7Dln_5C_2C_5Cleft_20_7C_20_5CPsi_20_5Cright_20_7C-_5Cfrac_7Bn_7D_7B2_7Dln_5C_2C_282_5Cpi_29-_5Cfrac_7B1_7D_7B2_7D_28x-_5CLambda_20z_29_27_5CPsi_5E_7B-1_7D_28x-_5CLambda_20z_29_5D_5C_5C_20_26_3D_20_5Csum_7Bi_3D1_7D_5E_7Bn_7D_28-_5Cfrac_7B1_7D_7B2_7D_5Cbigtriangledown_7B_5CPsi_5E_7B-1_7D_20_7Dln_5C_2C_5Cleft_20_7C_20_5CPsi_20_5Cright_20_7C-_5Cbigtriangledown_7B_5CPsi_5E_7B-1_7D_20_7DE_7B_5Csim_20z_7D_5Cfrac_7B1_7D_7B2_7D_28x-_5CLambda_20z_29_27_5CPsi_5E_7B-1_7D_28x-_5CLambda_20z_29_29_5C_5C_20_26_3D_20_5Csum_7Bi_3D1_7D_5E_7Bn_7D_28_5Cfrac_7B1_7D_7B2_7D_5CPsi-E_7B_5Csim_20z_7D_5Cfrac_7B1_7D_7B2_7D_28x-_5CLambda_20z_29_28x-_5CLambda_20z_29_27_29_20_5C_5C_20_26_3D_20_5Csum_7Bi_3D1_7D_5E_7Bn_7D_28_5Cfrac_7B1_7D_7B2_7D_5CPsi-_5Cfrac_7B1_7D_7B2_7Dxx_27_20_5Cfrac_7B1_7D_7B2_7DxE_5Bz_27_5D_5CLambda_5E_7B_27_7D_20_5Cfrac_7B1_7D_7B2_7D_5CLambda_20E_5Bz_5Dx_27-_5Cfrac_7B1_7D_7B2_7D_5CLambda_20E_5Bzz_27_5D_5CLambda_5E_7B_27_7D_29_20_5C_5C_20_26_3D_20_7B_5Ccolor_7BBlue_7D_20_5Csum_7Bi_3D1_7D_5E_7Bn_7D_28_5Cfrac_7B1_7D_7B2_7D_5CPsi-_5Cfrac_7B1_7D_7B2_7Dxx_27_5Cfrac_7B1_7D_7B2_7D_5CLambda_20E_5Bz_5Dx_27_29_7D_20_5Cend_7Balign_7D][] **公式14**

最后一步通过公式11代入得到，那么整理得到：

![gif.latex_5CPsi_3D_20_5Cfrac_7B1_7D_7Bn_7D_5Csum_7Bi_3D1_7D_5E_7Bn_7D_28xx_27-_5CLambda_20E_5Bz_5Dx_27_29][]

加上对角限制最终表达式如公式15所示：

![gif.latex_5CPsi_3D_20_5Cfrac_7B1_7D_7Bn_7D_5Cmathbf_7BDiag_7D_5Csum_7Bi_3D1_7D_5E_7Bn_7D_28xx_27-_5CLambda_20E_5Bz_5Dx_27_29][] **公式15**

EM算法运用于因子分析过程如下所示：

*  E-Step:

![gif.latex_E_5Bz_5D_5Cleftarrow_20_7B_5CLambda_20_7D_27_28_5CLambda_7B_5CLambda_20_7D_27_5CPsi_29_5E_7B-1_7Dx][]

![gif.latex_E_5Bzz_27_5D_20_5Cleftarrow_20I-_7B_5CLambda_20_7D_27_28_5CLambda_7B_5CLambda_20_7D_27_5CPsi_29_5E_7B-1_7D_5CLambda_E_5Bz_5DE_5Bz_27_5D][]

*  M-Step:

![gif.latex_5CLambda_5Cleftarrow_20_5Csum_7Bi_3D1_7D_5E_7Bn_7DxE_5Bz_27_5D_28_5Csum_7Bi_3D1_7D_5E_7Bn_7DE_5Bzz_27_5D_29_5E_7B-1_7D][]

![gif.latex_5CPsi_20_5Cleftarrow_20_5Cfrac_7B1_7D_7Bn_7D_5Cmathbf_7BDiag_7D_5Csum_7Bi_3D1_7D_5E_7Bn_7D_28xx_27-_5CLambda_20E_5Bz_5Dx_27_29][]

至此关于因子分析和EM算法推到全部结束。

## 参考文献 ##

1、 [CS229 Factor analysis][]

2、 [The EM Algorithm for Mixtures of Factor Analysis][]

3、 [The Expectation-Maximization (EM) Algorithm][The Expectation-Maximization _EM_ Algorithm]

4、 [Pattern Recognition and Machine Learning][]

[EM]: http://blog.csdn.net/ccrush/article/details/15614093
[gif.latex_n_5Cgg_20d]: http://latex.codecogs.com/gif.latex?n%5Cgg%20d
[gif.latex_5CSigma]: http://latex.codecogs.com/gif.latex?%5CSigma
[gif.latex_5CSigma_5E_7B-1_7D]: http://latex.codecogs.com/gif.latex?%5CSigma%5E%7B-1%7D
[2_7D]: /images/20220727/6f51cf9b2b9b42b5b0fbdb09c1a503c0.png
[gif.latex_5Cdpi_7B200_7D_20_5Ctiny_20x_3D_5CLambda_20z_5Cvarepsilon]: http://latex.codecogs.com/gif.latex?%5Cdpi%7B200%7D%20%5Ctiny%20x%3D%5CLambda%20z+%5Cvarepsilon
[gif.latex_5CLambda]: http://latex.codecogs.com/gif.latex?%5CLambda
[gif.latex_5Cvarepsilon]: /images/20220727/8e581409f7664d4995695dbb2f7208a5.png
[gif.latex_5Cdpi_7B200_7D_20_5Ctiny_20_5Cbegin_7Bmatrix_7D_20P_28z_29_3DN_280_2CI_29_5C_5C_20P_28_5Cvarepsilon_20_29_3DN_280_2C_5CPsi_20_29_5C_5C_20P_28x_5Cmid_20z_29_3DN_28_5CLambda_20z_2C_5CPsi_20_29_20_5Cend_7Bmatrix_7D]: /images/20220727/85076fd4e4bd4f8fbbcdb8b5f3b57045.png
[SouthEast]: /images/20220727/9c38cbf63ab44197b7e0c8fe93d702ce.png
[Pattern Recognition and Machine Learning]: http://research.microsoft.com/en-us/um/people/cmbishop/PRML/
[gif.latex_5Cdpi_7B200_7D_20_5Ctiny_20P_28x_29_3DN_280_2C_5CLambda_20_7B_5CLambda_7D_27_20_5CPsi_20_29]: http://latex.codecogs.com/gif.latex?%5Cdpi%7B200%7D%20%5Ctiny%20P%28x%29%3DN%280%2C%5CLambda%20%7B%5CLambda%7D%27%20+%5CPsi%20%29
[gif.latex_5Cdpi_7B200_7D_20_5Ctiny_20_5Cbegin_7Balign_7D_20P_28z_5Cmid_20x_29_20_26_3D_20N_28_28I_7B_5CLambda_20_7D_27_5CPsi_5E_7B-1_7D_20_5CLambda_20_29_5E_7B-1_7D_5CLambda_20_27_5CPsi_5E_7B-1_7Dx_2C_28I_7B_5CLambda_20_7D_27_5CPsi_5E_7B-1_7D_20_5CLambda_20_29_5E_7B-1_7D_29_5C_5C_20_26_3D_20N_28_7B_5CLambda_20_7D_27_28_5CLambda_7B_5CLambda_20_7D_27_5CPsi_29_5E_7B-1_7Dx_2CI-_7B_5CLambda_20_7D_27_28_5CLambda_7B_5CLambda_20_7D_27_5CPsi_29_5E_7B-1_7D_5CLambda_29_20_5Cend_7Balign_7D]: http://latex.codecogs.com/gif.latex?%5Cdpi%7B200%7D%20%5Ctiny%20%5Cbegin%7Balign*%7D%20P%28z%5Cmid%20x%29%20%26%3D%20N%28%28I+%7B%5CLambda%20%7D%27%5CPsi%5E%7B-1%7D%20%5CLambda%20%29%5E%7B-1%7D%5CLambda%20%27%5CPsi%5E%7B-1%7Dx%2C%28I+%7B%5CLambda%20%7D%27%5CPsi%5E%7B-1%7D%20%5CLambda%20%29%5E%7B-1%7D%29%5C%5C%20%26%3D%20N%28%7B%5CLambda%20%7D%27%28%5CLambda%7B%5CLambda%20%7D%27+%5CPsi%29%5E%7B-1%7Dx%2CI-%7B%5CLambda%20%7D%27%28%5CLambda%7B%5CLambda%20%7D%27+%5CPsi%29%5E%7B-1%7D%5CLambda%29%20%5Cend%7Balign*%7D
[maximum-likelihood estimation _MLE]: http://en.wikipedia.org/wiki/Maximum_likelihood
[2_7D_7De_5E_7B-_5Cfrac_7B1_7D_7B2_7D_28x_7Bi_7D-_5Cmu_29_27_28_5CLambda_20_7B_5CLambda_20_7D_27_5CPsi_29_5E_7B-1_7D_28x_7Bi_7D-_5Cmu_29_7D]: http://latex.codecogs.com/gif.latex?%5Cdpi%7B200%7D%20%5Ctiny%20L%28%5CLambda%20%2C%5CPsi%20%29%20%3D%20ln%5Cprod_%7Bi%3D1%7D%5E%7Bn%7D%5Cfrac%7B1%7D%7B%282%5Cpi%29%5E%7Bd/2%7D%5Cleft%20%7C%20%5CLambda%20%7B%5CLambda%20%7D%27+%5CPsi%20%5Cright%20%7C%5E%7B1/2%7D%7De%5E%7B-%5Cfrac%7B1%7D%7B2%7D%28x_%7Bi%7D-%5Cmu%29%27%28%5CLambda%20%7B%5CLambda%20%7D%27+%5CPsi%29%5E%7B-1%7D%28x_%7Bi%7D-%5Cmu%29%7D
[gif.latex_q_28z_7Bi_7D_29_20_5Cleftarrow_20N_28_7B_5CLambda_20_7D_27_28_5CLambda_7B_5CLambda_20_7D_27_5CPsi_29_5E_7B-1_7Dx_7Bi_7D_2CI-_7B_5CLambda_20_7D_27_28_5CLambda_7B_5CLambda_20_7D_27_5CPsi_29_5E_7B-1_7D_5CLambda_29]: http://latex.codecogs.com/gif.latex?q%28z_%7Bi%7D%29%20%5Cleftarrow%20N%28%7B%5CLambda%20%7D%27%28%5CLambda%7B%5CLambda%20%7D%27+%5CPsi%29%5E%7B-1%7Dx_%7Bi%7D%2CI-%7B%5CLambda%20%7D%27%28%5CLambda%7B%5CLambda%20%7D%27+%5CPsi%29%5E%7B-1%7D%5CLambda%29
[gif.latex_5Cdpi_7B200_7D_20_5Ctiny_20_5CTheta_20_5C_7B_5CLambda_20_2C_5CPsi_20_5C_7D_5Cleftarrow_20_5Cunderset_7B_5CTheta_7D_7Barg_20max_7D_5Csum_7Bi_3D1_7D_5E_7Bn_7D_5Cint_7Bz_7Dq_28z_29ln_5C_2CP_28x_2Cz_5Cmid_20_5CTheta_29_5Cmathrm_7Bd_7Dz]: /images/20220727/3423e83ec26c4d0cac30716648a58553.png
[gif.latex_5Cdpi_7B200_7D_20_5Ctiny_20_5Cbegin_7Balign_7D_20_5Csum_7Bi_3D1_7D_5E_7Bn_7D_5Cint_7Bz_7Dq_28z_29ln_5C_2CP_28x_2Cz_5Cmid_20_5CTheta_29_5Cmathrm_7Bd_7Dz_20_26_3D_20_5Csum_7Bi_3D1_7D_5E_7Bn_7D_5Cint_7Bz_7Dq_28z_29_5Bln_5C_2CP_28x_5Cmid_20z_3B_5CLambda_20_2C_5CPsi_20_29_ln_5C_2CP_28z_29_5D_5Cmathrm_7Bd_7Dz_5C_5C_20_26_3D_20_5Csum_7Bi_3D1_7D_5E_7Bn_7DE_7Bz_5Csim_20q_28z_29_7D_5Bln_5C_2CP_28x_5Cmid_20z_3B_5CLambda_20_2C_5CPsi_20_29_ln_5C_2CP_28z_29_5D_5C_5C_20_26_3D_20_5Csum_7Bi_3D1_7D_5E_7Bn_7DE_7Bz_5Csim_20q_28z_29_7D_5Bln_5C_2CP_28x_5Cmid_20z_3B_5CLambda_20_2C_5CPsi_20_29_5D_K_5C_5C_20_5Cend_7Balign_7D]: http://latex.codecogs.com/gif.latex?%5Cdpi%7B200%7D%20%5Ctiny%20%5Cbegin%7Balign*%7D%20%5Csum_%7Bi%3D1%7D%5E%7Bn%7D%5Cint_%7Bz%7Dq%28z%29ln%5C%2CP%28x%2Cz%5Cmid%20%5CTheta%29%5Cmathrm%7Bd%7Dz%20%26%3D%20%5Csum_%7Bi%3D1%7D%5E%7Bn%7D%5Cint_%7Bz%7Dq%28z%29%5Bln%5C%2CP%28x%5Cmid%20z%3B%5CLambda%20%2C%5CPsi%20%29+ln%5C%2CP%28z%29%5D%5Cmathrm%7Bd%7Dz%5C%5C%20%26%3D%20%5Csum_%7Bi%3D1%7D%5E%7Bn%7DE_%7Bz%5Csim%20q%28z%29%7D%5Bln%5C%2CP%28x%5Cmid%20z%3B%5CLambda%20%2C%5CPsi%20%29+ln%5C%2CP%28z%29%5D%5C%5C%20%26%3D%20%5Csum_%7Bi%3D1%7D%5E%7Bn%7DE_%7Bz%5Csim%20q%28z%29%7D%5Bln%5C%2CP%28x%5Cmid%20z%3B%5CLambda%20%2C%5CPsi%20%29%5D+K%5C%5C%20%5Cend%7Balign*%7D
[2_7D_7De_5E_7B-_5Cfrac_7B1_7D_7B2_7D_28x-_5CLambda_20z_29_27_5CPsi_5E_7B-1_7D_28x-_5CLambda_20z_29_7D_5D_5C_5C_20_26_3D_20_5Csum_7Bi_3D1_7D_5E_7Bn_7DE_5B-_5Cfrac_7B1_7D_7B2_7Dln_5C_2C_5Cleft_20_7C_20_5CPsi_20_5Cright_20_7C-_5Cfrac_7Bn_7D_7B2_7Dln_5C_2C_282_5Cpi_29-_5Cfrac_7B1_7D_7B2_7D_28x-_5CLambda_20z_29_27_5CPsi_5E_7B-1_7D_28x-_5CLambda_20z_29_5D_5C_5C_20_5Cend_7Balign_7D]: http://latex.codecogs.com/gif.latex?%5Cdpi%7B200%7D%20%5Ctiny%20%5Cbegin%7Balign*%7D%20%5Csum_%7Bi%3D1%7D%5E%7Bn%7DE%5Bln%5C%2CP%28x%5Cmid%20z%3B%5CLambda%20%2C%5CPsi%20%29%5D%20%26%3D%20%5Csum_%7Bi%3D1%7D%5E%7Bn%7DE%5Bln%5C%2C%5Cfrac%7B1%7D%7B%282%5Cpi%29%5E%7Bd/2%7D%5Cleft%7C%5CPsi%20%5Cright%20%7C%5E%7B1/2%7D%7De%5E%7B-%5Cfrac%7B1%7D%7B2%7D%28x-%5CLambda%20z%29%27%5CPsi%5E%7B-1%7D%28x-%5CLambda%20z%29%7D%5D%5C%5C%20%26%3D%20%5Csum_%7Bi%3D1%7D%5E%7Bn%7DE%5B-%5Cfrac%7B1%7D%7B2%7Dln%5C%2C%5Cleft%20%7C%20%5CPsi%20%5Cright%20%7C-%5Cfrac%7Bn%7D%7B2%7Dln%5C%2C%282%5Cpi%29-%5Cfrac%7B1%7D%7B2%7D%28x-%5CLambda%20z%29%27%5CPsi%5E%7B-1%7D%28x-%5CLambda%20z%29%5D%5C%5C%20%5Cend%7Balign*%7D
[gif.latex_5Cdpi_7B200_7D_20_5Ctiny_20_5CLambda]: /images/20220727/5385fe44d5bb46d4a333ec4fe9ecec58.png
[Trace]: http://en.wikipedia.org/wiki/Trace_%28linear_algebra%29
[gif.latex_5Cdpi_7B200_7D_20_5Ctiny_20tr_5C_2Ca_3Da_2Ctr_5C_2CAB_3Dtr_5C_2CBA]: http://latex.codecogs.com/gif.latex?%5Cdpi%7B200%7D%20%5Ctiny%20tr%5C%2Ca%3Da%2Ctr%5C%2CAB%3Dtr%5C%2CBA
[Link 1]: http://www.psi.toronto.edu/matrix/calculus.html
[gif.latex_5Cdpi_7B200_7D_20_5Ctiny_20_5Cpartial_20trABA_5E_7BT_7DC_5Csetminus_20_5Cpartial_20A_3DCAB_C_5E_7BT_7DAB]: http://latex.codecogs.com/gif.latex?%5Cdpi%7B200%7D%20%5Ctiny%20%5Cpartial%20trABA%5E%7BT%7DC%5Csetminus%20%5Cpartial%20A%3DCAB+C%5E%7BT%7DAB
[gif.latex_5Cdpi_7B200_7D_20_5Ctiny_20_5Cbegin_7Balign_7D_20_5Cbigtriangledown_7B_5CLambda_20_7D_20L_28_5CLambda_20_2C_5CPsi_29_20_26_3D_20_5Cbigtriangledown_7B_5CLambda_20_7D_5Csum_7Bi_3D1_7D_5E_7Bn_7DE_5B-_5Cfrac_7B1_7D_7B2_7Dln_5C_2C_5Cleft_20_7C_20_5CPsi_20_5Cright_20_7C-_5Cfrac_7Bn_7D_7B2_7Dln_5C_2C_282_5Cpi_29-_5Cfrac_7B1_7D_7B2_7D_28x-_5CLambda_20z_29_27_5CPsi_5E_7B-1_7D_28x-_5CLambda_20z_29_5D_5C_5C_20_26_3D_20_5Cbigtriangledown_7B_5CLambda_20_7D_5Csum_7Bi_3D1_7D_5E_7Bn_7DE_5B-_5Cfrac_7B1_7D_7B2_7D_28x-_5CLambda_20z_29_27_5CPsi_5E_7B-1_7D_28x-_5CLambda_20z_29_5D_5C_5C_20_26_3D_20_5Cbigtriangledown_7B_5CLambda_20_7D_5Csum_7Bi_3D1_7D_5E_7Bn_7DE_5B-_5Cfrac_7B1_7D_7B2_7Dx_27_5CPsi_5E_7B-1_7Dx_5Cfrac_7B1_7D_7B2_7Dx_27_5CPsi_5E_7B-1_7D_5CLambda_20z_5Cfrac_7B1_7D_7B2_7Dz_27_5CLambda_20_5CPsi_5E_7B-1_7Dx-_5Cfrac_7B1_7D_7B2_7Dz_27_5CLambda_27_5CPsi_5E_7B-1_7D_5CLambda_20z_5D_5C_5C_20_26_3D_20_7B_5Ccolor_7BBlue_7D_20_5Cbigtriangledown_7B_5CLambda_20_7D_5Csum_7Bi_3D1_7D_5E_7Bn_7DE_5Btr_28z_27_5CLambda_20_5CPsi_5E_7B-1_7Dx_29-_5Cfrac_7B1_7D_7B2_7Dtr_28z_27_5CLambda_27_5CPsi_5E_7B-1_7D_5CLambda_20z_29_5D_7D_5C_5C_20_26_3D_20_7B_5Ccolor_7BRed_7D_20_5Cbigtriangledown_7B_5CLambda_20_7D_5Csum_7Bi_3D1_7D_5E_7Bn_7DE_5Btr_28_5CLambda_20_5CPsi_5E_7B-1_7Dxz_27_29-_5Cfrac_7B1_7D_7B2_7Dtr_28_5CLambda_27_5CPsi_5E_7B-1_7D_5CLambda_20zz_27_29_5D_7D_5C_5C_20_26_3D_20_7B_5Ccolor_7BGreen_7D_20_5Csum_7Bi_3D1_7D_5E_7Bn_7DE_5B_5CPsi_5E_7B-1_7Dxz_27-_5CPsi_5E_7B-1_7D_5CLambda_20zz_27_5D_7D_20_5Cend_7Balign_7D]: http://latex.codecogs.com/gif.latex?%5Cdpi%7B200%7D%20%5Ctiny%20%5Cbegin%7Balign*%7D%20%5Cbigtriangledown_%7B%5CLambda%20%7D%20L%28%5CLambda%20%2C%5CPsi%29%20%26%3D%20%5Cbigtriangledown_%7B%5CLambda%20%7D%5Csum_%7Bi%3D1%7D%5E%7Bn%7DE%5B-%5Cfrac%7B1%7D%7B2%7Dln%5C%2C%5Cleft%20%7C%20%5CPsi%20%5Cright%20%7C-%5Cfrac%7Bn%7D%7B2%7Dln%5C%2C%282%5Cpi%29-%5Cfrac%7B1%7D%7B2%7D%28x-%5CLambda%20z%29%27%5CPsi%5E%7B-1%7D%28x-%5CLambda%20z%29%5D%5C%5C%20%26%3D%20%5Cbigtriangledown_%7B%5CLambda%20%7D%5Csum_%7Bi%3D1%7D%5E%7Bn%7DE%5B-%5Cfrac%7B1%7D%7B2%7D%28x-%5CLambda%20z%29%27%5CPsi%5E%7B-1%7D%28x-%5CLambda%20z%29%5D%5C%5C%20%26%3D%20%5Cbigtriangledown_%7B%5CLambda%20%7D%5Csum_%7Bi%3D1%7D%5E%7Bn%7DE%5B-%5Cfrac%7B1%7D%7B2%7Dx%27%5CPsi%5E%7B-1%7Dx+%5Cfrac%7B1%7D%7B2%7Dx%27%5CPsi%5E%7B-1%7D%5CLambda%20z+%5Cfrac%7B1%7D%7B2%7Dz%27%5CLambda%20%5CPsi%5E%7B-1%7Dx-%5Cfrac%7B1%7D%7B2%7Dz%27%5CLambda%27%5CPsi%5E%7B-1%7D%5CLambda%20z%5D%5C%5C%20%26%3D%20%7B%5Ccolor%7BBlue%7D%20%5Cbigtriangledown_%7B%5CLambda%20%7D%5Csum_%7Bi%3D1%7D%5E%7Bn%7DE%5Btr%28z%27%5CLambda%20%5CPsi%5E%7B-1%7Dx%29-%5Cfrac%7B1%7D%7B2%7Dtr%28z%27%5CLambda%27%5CPsi%5E%7B-1%7D%5CLambda%20z%29%5D%7D%5C%5C%20%26%3D%20%7B%5Ccolor%7BRed%7D%20%5Cbigtriangledown_%7B%5CLambda%20%7D%5Csum_%7Bi%3D1%7D%5E%7Bn%7DE%5Btr%28%5CLambda%20%5CPsi%5E%7B-1%7Dxz%27%29-%5Cfrac%7B1%7D%7B2%7Dtr%28%5CLambda%27%5CPsi%5E%7B-1%7D%5CLambda%20zz%27%29%5D%7D%5C%5C%20%26%3D%20%7B%5Ccolor%7BGreen%7D%20%5Csum_%7Bi%3D1%7D%5E%7Bn%7DE%5B%5CPsi%5E%7B-1%7Dxz%27-%5CPsi%5E%7B-1%7D%5CLambda%20zz%27%5D%7D%20%5Cend%7Balign*%7D
[gif.latex_5Cdpi_7B200_7D_20_5Ctiny_20_5Csum_7Bi_3D1_7D_5E_7Bn_7D_5CLambda_20E_7B_5Csim_20z_7D_5Bzz_27_5D_3D_5Csum_7Bi_3D1_7D_5E_7Bn_7DE_7B_5Csim_20z_7D_5Bxz_27_5D]: /images/20220727/3f6db1e8ca8e4307b4cf4d1b69964b1f.png
[gif.latex_5Cdpi_7B200_7D_20_5Ctiny_20_5CLambda_3D_5Csum_7Bi_3D1_7D_5E_7Bn_7DxE_5Bz_27_5D_28_5Csum_7Bi_3D1_7D_5E_7Bn_7DE_5Bzz_27_5D_29_5E_7B-1_7D]: /images/20220727/5385fe44d5bb46d4a333ec4fe9ecec58.png%3D%5Csum_%7Bi%3D1%7D%5E%7Bn%7DxE%5Bz%27%5D%28%5Csum_%7Bi%3D1%7D%5E%7Bn%7DE%5Bzz%27%5D%29%5E%7B-1%7D
[gif.latex_5Cdpi_7B200_7D_20_5Ctiny_20E_5Bz_5D_3D_7B_5CLambda_20_7D_27_28_5CLambda_7B_5CLambda_20_7D_27_5CPsi_29_5E_7B-1_7Dx]: http://latex.codecogs.com/gif.latex?%5Cdpi%7B200%7D%20%5Ctiny%20E%5Bz%5D%3D%7B%5CLambda%20%7D%27%28%5CLambda%7B%5CLambda%20%7D%27+%5CPsi%29%5E%7B-1%7Dx
[gif.latex_5Cdpi_7B200_7D_20_5Ctiny_20_5Cbegin_7Balign_7D_20E_5Bzz_27_5D_20_26_3D_20cov_5Bz_5D_E_5Bz_5DE_5Bz_27_5D_5C_5C_20_26_3D_20I-_7B_5CLambda_20_7D_27_28_5CLambda_7B_5CLambda_20_7D_27_5CPsi_29_5E_7B-1_7D_5CLambda_E_5Bz_5DE_5Bz_27_5D_20_5Cend_7Balign_7D]: http://latex.codecogs.com/gif.latex?%5Cdpi%7B200%7D%20%5Ctiny%20%5Cbegin%7Balign*%7D%20E%5Bzz%27%5D%20%26%3D%20cov%5Bz%5D+E%5Bz%5DE%5Bz%27%5D%5C%5C%20%26%3D%20I-%7B%5CLambda%20%7D%27%28%5CLambda%7B%5CLambda%20%7D%27+%5CPsi%29%5E%7B-1%7D%5CLambda+E%5Bz%5DE%5Bz%27%5D%20%5Cend%7Balign*%7D
[gif.latex_5Cdpi_7B200_7D_20_5Ctiny_20_5CPsi]: /images/20220727/3135d1c36ae94209a0a91fde4f999899.png
[gif.latex_5Cbegin_7Balign_7D_20_5Cbigtriangledown_7B_5CPsi_5E_7B-1_7D_20_7D_20L_28_5CLambda_20_2C_5CPsi_20_29_20_26_3D_20_5Cbigtriangledown_7B_5CPsi_5E_7B-1_7D_20_7D_5Csum_7Bi_3D1_7D_5E_7Bn_7DE_5B-_5Cfrac_7B1_7D_7B2_7Dln_5C_2C_5Cleft_20_7C_20_5CPsi_20_5Cright_20_7C-_5Cfrac_7Bn_7D_7B2_7Dln_5C_2C_282_5Cpi_29-_5Cfrac_7B1_7D_7B2_7D_28x-_5CLambda_20z_29_27_5CPsi_5E_7B-1_7D_28x-_5CLambda_20z_29_5D_5C_5C_20_26_3D_20_5Csum_7Bi_3D1_7D_5E_7Bn_7D_28-_5Cfrac_7B1_7D_7B2_7D_5Cbigtriangledown_7B_5CPsi_5E_7B-1_7D_20_7Dln_5C_2C_5Cleft_20_7C_20_5CPsi_20_5Cright_20_7C-_5Cbigtriangledown_7B_5CPsi_5E_7B-1_7D_20_7DE_7B_5Csim_20z_7D_5Cfrac_7B1_7D_7B2_7D_28x-_5CLambda_20z_29_27_5CPsi_5E_7B-1_7D_28x-_5CLambda_20z_29_29_5C_5C_20_26_3D_20_5Csum_7Bi_3D1_7D_5E_7Bn_7D_28_5Cfrac_7B1_7D_7B2_7D_5CPsi-E_7B_5Csim_20z_7D_5Cfrac_7B1_7D_7B2_7D_28x-_5CLambda_20z_29_28x-_5CLambda_20z_29_27_29_20_5C_5C_20_26_3D_20_5Csum_7Bi_3D1_7D_5E_7Bn_7D_28_5Cfrac_7B1_7D_7B2_7D_5CPsi-_5Cfrac_7B1_7D_7B2_7Dxx_27_20_5Cfrac_7B1_7D_7B2_7DxE_5Bz_27_5D_5CLambda_5E_7B_27_7D_20_5Cfrac_7B1_7D_7B2_7D_5CLambda_20E_5Bz_5Dx_27-_5Cfrac_7B1_7D_7B2_7D_5CLambda_20E_5Bzz_27_5D_5CLambda_5E_7B_27_7D_29_20_5C_5C_20_26_3D_20_7B_5Ccolor_7BBlue_7D_20_5Csum_7Bi_3D1_7D_5E_7Bn_7D_28_5Cfrac_7B1_7D_7B2_7D_5CPsi-_5Cfrac_7B1_7D_7B2_7Dxx_27_5Cfrac_7B1_7D_7B2_7D_5CLambda_20E_5Bz_5Dx_27_29_7D_20_5Cend_7Balign_7D]: http://latex.codecogs.com/gif.latex?%5Cbegin%7Balign*%7D%20%5Cbigtriangledown_%7B%5CPsi%5E%7B-1%7D%20%7D%20L%28%5CLambda%20%2C%5CPsi%20%29%20%26%3D%20%5Cbigtriangledown_%7B%5CPsi%5E%7B-1%7D%20%7D%5Csum_%7Bi%3D1%7D%5E%7Bn%7DE%5B-%5Cfrac%7B1%7D%7B2%7Dln%5C%2C%5Cleft%20%7C%20%5CPsi%20%5Cright%20%7C-%5Cfrac%7Bn%7D%7B2%7Dln%5C%2C%282%5Cpi%29-%5Cfrac%7B1%7D%7B2%7D%28x-%5CLambda%20z%29%27%5CPsi%5E%7B-1%7D%28x-%5CLambda%20z%29%5D%5C%5C%20%26%3D%20%5Csum_%7Bi%3D1%7D%5E%7Bn%7D%28-%5Cfrac%7B1%7D%7B2%7D%5Cbigtriangledown_%7B%5CPsi%5E%7B-1%7D%20%7Dln%5C%2C%5Cleft%20%7C%20%5CPsi%20%5Cright%20%7C-%5Cbigtriangledown_%7B%5CPsi%5E%7B-1%7D%20%7DE_%7B%5Csim%20z%7D%5Cfrac%7B1%7D%7B2%7D%28x-%5CLambda%20z%29%27%5CPsi%5E%7B-1%7D%28x-%5CLambda%20z%29%29%5C%5C%20%26%3D%20%5Csum_%7Bi%3D1%7D%5E%7Bn%7D%28%5Cfrac%7B1%7D%7B2%7D%5CPsi-E_%7B%5Csim%20z%7D%5Cfrac%7B1%7D%7B2%7D%28x-%5CLambda%20z%29%28x-%5CLambda%20z%29%27%29%20%5C%5C%20%26%3D%20%5Csum_%7Bi%3D1%7D%5E%7Bn%7D%28%5Cfrac%7B1%7D%7B2%7D%5CPsi-%5Cfrac%7B1%7D%7B2%7Dxx%27+%20%5Cfrac%7B1%7D%7B2%7DxE%5Bz%27%5D%5CLambda%5E%7B%27%7D+%20%5Cfrac%7B1%7D%7B2%7D%5CLambda%20E%5Bz%5Dx%27-%5Cfrac%7B1%7D%7B2%7D%5CLambda%20E%5Bzz%27%5D%5CLambda%5E%7B%27%7D%29%20%5C%5C%20%26%3D%20%7B%5Ccolor%7BBlue%7D%20%5Csum_%7Bi%3D1%7D%5E%7Bn%7D%28%5Cfrac%7B1%7D%7B2%7D%5CPsi-%5Cfrac%7B1%7D%7B2%7Dxx%27+%5Cfrac%7B1%7D%7B2%7D%5CLambda%20E%5Bz%5Dx%27%29%7D%20%5Cend%7Balign*%7D
[gif.latex_5CPsi_3D_20_5Cfrac_7B1_7D_7Bn_7D_5Csum_7Bi_3D1_7D_5E_7Bn_7D_28xx_27-_5CLambda_20E_5Bz_5Dx_27_29]: /images/20220727/f75edfca1887473d8d2c86a776431bdc.png
[gif.latex_5CPsi_3D_20_5Cfrac_7B1_7D_7Bn_7D_5Cmathbf_7BDiag_7D_5Csum_7Bi_3D1_7D_5E_7Bn_7D_28xx_27-_5CLambda_20E_5Bz_5Dx_27_29]: /images/20220727/1ceb51d47bcb49d88fefe25315d4be94.png
[gif.latex_E_5Bz_5D_5Cleftarrow_20_7B_5CLambda_20_7D_27_28_5CLambda_7B_5CLambda_20_7D_27_5CPsi_29_5E_7B-1_7Dx]: http://latex.codecogs.com/gif.latex?E%5Bz%5D%5Cleftarrow%20%7B%5CLambda%20%7D%27%28%5CLambda%7B%5CLambda%20%7D%27+%5CPsi%29%5E%7B-1%7Dx
[gif.latex_E_5Bzz_27_5D_20_5Cleftarrow_20I-_7B_5CLambda_20_7D_27_28_5CLambda_7B_5CLambda_20_7D_27_5CPsi_29_5E_7B-1_7D_5CLambda_E_5Bz_5DE_5Bz_27_5D]: http://latex.codecogs.com/gif.latex?E%5Bzz%27%5D%20%5Cleftarrow%20I-%7B%5CLambda%20%7D%27%28%5CLambda%7B%5CLambda%20%7D%27+%5CPsi%29%5E%7B-1%7D%5CLambda+E%5Bz%5DE%5Bz%27%5D
[gif.latex_5CLambda_5Cleftarrow_20_5Csum_7Bi_3D1_7D_5E_7Bn_7DxE_5Bz_27_5D_28_5Csum_7Bi_3D1_7D_5E_7Bn_7DE_5Bzz_27_5D_29_5E_7B-1_7D]: /images/20220727/8bd598d7085e4e5ca14c46d56dd1ef7e.png
[gif.latex_5CPsi_20_5Cleftarrow_20_5Cfrac_7B1_7D_7Bn_7D_5Cmathbf_7BDiag_7D_5Csum_7Bi_3D1_7D_5E_7Bn_7D_28xx_27-_5CLambda_20E_5Bz_5Dx_27_29]: /images/20220727/3859e263602a48a1928eb3a0aa63d0c9.png
[CS229 Factor analysis]: http://cs229.stanford.edu/notes/cs229-notes9.pdf
[The EM Algorithm for Mixtures of Factor Analysis]: http://mlg.eng.cam.ac.uk/zoubin/papers/tr-96-1.pdf
[The Expectation-Maximization _EM_ Algorithm]: http://www0.cs.ucl.ac.uk/staff/S.Prince/Resources/EMAlgorithm.pdf