机器学习-machine learning 课程--期末考试思路整理

淡淡的烟草味﹌ 2022-03-29 03:18 252阅读 0赞

对于两个问题我想了一些，写了一部分，大概有两千字左右，希望大家对于两个问题提供不同的思路，一起交流学习，为以后的学弟学妹留下一些参考的想法。其中贝叶斯公式与SVM公式我不贴图了。

针对第二题，集成学习方法在一定程度上来说，是很重要的，emmmm，其他的我就不说了

**Chapter 1 :** **Introduction**

In chapter 1, teacher teaches me a lot of basic concept about machine learning which will be used in future class, such as learners, learning paradigms and resources of machine learning. The most important concepts are training set , test set ,supervised learning and unsupervised learning.  In the next chapters, supervised learning consists of decision tree, KNN and naive Bayesian Model. On contrast, random forest, PCA and K-means clustering belong to unsupervise learning.  We use machine learning to solve the task such as classification, regression and clustering.

**Chapter 2 : main concept of machine learning**

Chapter 2 focus on main concepts of machine learning auch as classification, regression, clustering analysis and anomaly detection. Teacher also introduces an instance of machine learning: fish classification which judge the species of fish is sea bass or salmon. To make a decision about fish species, firstly, we use sensor to sensing and preprocess the data we collect. In the next, we should define the feature and extra them. In the end, we could classify the fish and compare the different classifier's performance.

After the instance, we learn the basic concept of machine learning.According to the data with label, the learning method can be divided into two categories: supervised learning and unsupervised learning. In the chapter 2, it also teaches supervised learning ,unsupervised learning ,semi-supervised learning and weakly supervised learning.

It introduces some items such as generalization error, training error, underfitting and overfitting. When we get the data set, we will split the data into training set and test set, and sometimes we need validation set. Error of the learner on the training called training error and on testing called generalization error. Training error is too large cause underfitting and generalization error is too large will cause overfitting. We should perform the model parameter and reduce the generalization error. After getting the trained model,  we could use hold-out, cross validation and bootstrapping to test generalization ability. And using precision, AUC and ROC to decribe the performance measurement of the model. The most significant thing is that trained model not only focus on the training sample, but also care about improving the generalization ability to adapt the new sample.

**Chapter 3 : Bayesian model**

Chapter 3 introduces Bayesian classifier which make a decision on the basis of probability.The formular of Bayesain model is as follows:... To minimize the general risk, we should calculate a label which could minimize general risk of every sample. To achieve this goal, we start to build a generative model, which based on the joint probaility distribution. Then we get the Bayesian optimal calssifier and bayes risk. In the final, we use Maximum Likelihood Estimation to estimate parameters of the Bayesian model.

Naive-Bayesian model treat every attribute mutally independent and semi-Bayesian model consider the part independent information of the attribute. While in the Bayesian network, sttributes are dependent with each other.

EM algorithm estimate the latent variable which devide into two step : E-step and M-step.

**Chapter 4 : Feature Selection and Feature Extraction**

Chapter 4 introduces feature selection and feature evaluation. Both of them ensure the related feature are involved in our task. Feature selection includes subset search and subset evaluation. Subset search can use forward search, backward search and bidirectional search. The methods of subset search can be divide into three categories: filter, wrapper and embedding.

Feature evaluation transform the original feature into a new feature set by constructing the combination of features. The methods of feature evaluation are linear method and nonlinear method. Linear method includes PCA and LDA. On contrast, nonlinear method consists of KPCA, MDS and LLE.

**Chapter 5 : Decision Tree**

Chapter 5 introduces decision tree which classify the sample from the root moving through the branches to the leaf node. The algorithm will use best attribute as a node and spilt the remaining sample into the next branch and node. Repeat this process untill all the instances reach the same target  attribute value. We can use information gain, gain ratio and Gini index to get the optimal partition attribute.In this algorithm, we use ID3 to calculate and generate the decision tree. When all instances reach it's node, it may cause overfitting. And it's better for us to use pruning which consists of pre-pruning and post pruning on the basis of information gain.

Decision tree can process the continuous value and missing data. In this chapter it also introduces multivariate decision tree. We can use bi-partition to process the continuous vlaue such as C4.5 and take advantage of missing data in decision tree.Multivariate tree can solve many complex problem. Decision tree can generate understandable rules, but it will generate error propagation and perform poorly when it meet many classes.

**Chapter 6 : Support Vector Machine**

Chapter 6 introduces support vector machine which use hyperplane to classify the categories. This chapter mainly introduces the concept of SVM, kernel function, KKT, hard margin, soft amrgin and support vector regression. The instance (xi,yi) which satisfy wxi+yi=+-1 called support vector.

SVM aims to find a hyperplane that has maximum margin. To solve the linearly inseparable problem, we map the sample into higher dimension place by using kernel function k(x,y)=a(xi)a(xj). To avoid overfitting and data with noise, we use soft margin to deal with these problem. Regularization allows error of SVM on some samples.

**Chapter 7 : K Nearest Neighbors**

Chapter 7 : K Nearest Neighbors algorithm is considered a lazy learning algorithm that classifies an unseen smaple based on similarity with its neighbors. This chapter introduces the defination of KNN, distance measures, digit recongnition and how to choose K. For classification, it outpts a discrete variable and a continuous variable for regression. The algorithm requires distance function and parameter K. The distance function aims to compute the similarities between example and the parameter k is number of nearest neighbors. Edclidean distance and Manhatten distance are commonly used. And we can choose proper K through cross-validation to get better generalization ability.

**Chapter 8 : Artificial neural network and deep learning**

Chapter 8 mainly introduces artificial neural network and deep learning. ANN aims to generate a function which rely on lots of unknow input. It consists of three layer: input layer, hidden layer and output layer. Input layer accepts outside parameter and input them into the ANN. Hidden layer and output layer processes the data. Output layer outputs the result of the model. Perceptron is the simplest ANN model without hidden layer. Firstly, every neurons in the ANN have a weight and we use BP algorithm to adjust them. Comparing the different between the estimation value and the real value, Error back propagation will adjust the weight of neurons to get a better generalization ability.

Deep learning is a branch of machine learning and it has deep layer that will performance better. Compare to older machine learning, deep learning show better ability in big data and deep learning will improve it's generalization ability by increasing the amount of data set.

**Chapter 9  : Semi-supervised learning**

Chapter 9 introduces semi-supervised learning which could use labeled data and data without label and them improve the classification performance . It consists of semi-supervised SVM,semi-supervised clustering ,disagreeed-based method, generative model and graph-based semi-supervised learning.

Semi-supervised SVM aims to find a hyperplane to classify the labeled data.

Semi-supervised clustering uses additional monitoring information to reach clustering.

**Chapter 10 : Ensemble learning**

Chapter 10 introduces ensemble learning which combine different classifier to deal with the same problem. There are two different categories: serial method such as Boosting and parallel method such as Bagging. Adaboost is commonly used in the task. Fusion method includes averaging, voting and learning. Diversity measure includes disagreement measure and correlation coefficient.

**Chapter 11 : Linear Regression**

Regression model output value is continuous. Regression estimates the relationship between input values and output value, and establishes a mathematical model in order to  accurately predict the output value of new sample.Regression is a supervised learning question. Basic linear regression includes ridge regression lasso regression.

Problem 2

Set a classifier for classify the identity of boys and girls in the School of Software should consider whether the data with labels.

What I thought I could use different classifier to estimate the gender of samples. According to their different generalization ability, it's better to use ensemble learning to  combine the result of different classifier. Assuming the data with label or without label, I will use supervised learning or unsupervised learning to claculate this problem.

**(1)Select characteristics of sample: **

Set X=\{a1, a2, a3, a4, a5, a6\},represent a sample. At the same time, a1,a2,a3 and a4 are attribute of sample represent the average length of hair, height, weight, bust, waistline and hipline.

The label are y=\{1,0\}.When y equals 1, it means the sample is a boy. On the contrast, it means the sample is a girl.

**(2)Data collection and data preprocessing:  **

We can apply to use the database of students in school or we could measure student's data by ourselves by asking for their information.

For example, we chose a vector A (24, 168, 45, 80, 0), it means the sample is a girl whose height is 168 cm, weight is 45kg, waistline is 80 cm and the longest length of hair is 24 cm. The previous feature vector is 4-dimensional vector, it change into 5-dimensional after we add the label attribute.

In the end, we get a list of data U=\{(x1,y1),(x2,y2),......(xn,yn)\};

N is the number of samples, where Xi indicates i-th student's attribute and yi indicates the class label of the i-th student.

**(3)Divide the data set into training set and test set:**

Here we use 10-fold cross validation :divide the data set D into 10 mutually exclusive subsets of equal size D= D1+D2+...+D10. There are no same sample between Di and Dj.Then we select randomly 9 sets for the training set and the remaining set as the test set.

**(4)Model Selection**

We use four classifier to the implement of the task for classify the identity of boy and girl in the School of Software, and using averaging method to get the final result:

A.Bayesian classifier

B.SVM model

C.Decision Tree

D.Neural Network

E.ensemble learning

**A.Bayesian Classifier**

Naive Bayesian classifier bases on the prior probability of a certain object, using Bayesian formula to calculate the posterior probability, and find the class with the maximum posterior probability. In the formular, P(c) is the prior probaility. P(c|x) is the posterior probaility. P(c1|x) is the posterior probaility of boys and P(c2|x) is posterior probaility of girls. Assuming the data following Gussian distribution, so we could use Maximum Likelihood Estimation to calculate the parameter of model and estimate the gender of sample.

**B.SVM Model**

SVM model depends on kernel function. The mostly used kernel function is RBF kernel. RBF kernel shows great generalization performance among the small data set and large data set ,low dimension and high dimension. So we choose the RBF kernel to the implement of this task.

We use RBF kernel to calculate the best parameters C and g ,which C is penalty factor ,means tolerance to the error , and g is radius of the kernel function.Then we use best parameters C and g to train the training set to get the SVM model.

Put the test sample into the trained SVM model and we could get the classification results of the test sample,the accuracy and ROC curve of the classifier.

**C.Decision Tree **

At first, we use information gain or Gini index to calculate the important of different attributes:X=\{a1, a2, a3, a4, a5, a6\}. Then we could use ID3 or C4.5 to build the decision tree. At last, we use pruning to avoid overfitting.

**D.Neural network**

Firstly, the input vector is X=\{a1, a2, a3, a4, a5, a6\}, we can build neural network which contains 2 hidden layer, then we use softmax process the result that from hidden layer. The activity function we use sigmoid function. In the end, we could get the final result from output layer. We set the only two result of the output layer: 0(means girl) and 1(means boy). According to the result that we estimate, we use error back propagation to adjust the weight of each neuron.

**(5)Performance measure**

At last. we get all the result of the different classifier and we could compare with each other by using precision, P-R cruve, AUC and ROC. And setting a weight of each other to improve the accuracy.

第三章：贝叶斯模型

第三章主要讲了贝叶斯分布，贝叶斯分布是集预告率来做出下一步决策。贝叶斯的公式如下所示：f(x)。贝叶斯要使得风险最小化，就要求一个标签使得每个样本的风险都是最小化。当我们在求后验概率是，要先建立生产模型，生产模型是基于联合概率分布构建的。我们求出贝叶斯最优分类与贝叶斯风险后，使用最大似然估计，来求得模型的参数。

朴素贝叶斯只考虑属性之间是相互独立的，半贝叶斯考虑独立属性的部分信息，在贝叶斯网络中，诸多属性都是相互关联的。

EM算法是用来预测潜在变量的，分为两步走，E步与M步。

第四章：特征选择与特征提取

特征选择与特征提取保证了保留相关特征，以便于任务运算。特征选择包括子集搜索与子集评估。子集搜索包括前向搜索，后向搜索与双向搜索。子集评估包括距离，独立性与持续性等。子集搜索有三种手段：过滤器，包装器与嵌入。

特征提取是将原始特征通过构造特征组合来重新组建新的特征集合。包含线性与非线性两种方法。线性方法包括PCA,LDA。非线性方法包括LLE,MDS与KPCA。

第五章 ： 决策树

第五章决策树将实例按照从根节点到叶子结点进行分类的算法。该算法思想是：先选出最佳属性，将剩余实例按照改属性划分，在不断选取剩余的最佳属性，重复该过程，直到所有节点被归类分置。我们可以使用信息增益，增益率，基尼指数来选择最佳属性。在此我们使用ID3来迭代的产生决策树。当所有节点被分类后，容易产生过拟合，这时候需要进行预剪枝与后剪枝，两种方法都是基于信息增益。

决策树针对连续值与空缺值有自己的处理方式，同时本章还讲了多变量决策树。针对连续值就使用二分法，C4.5。使用决策树可以好好利用空缺值。多变量决策树可以处理多种复杂问题。决策树具有良好的可解释性是一种很好的分类算法。但是当类的种类过多时，容易产生误差传播，决策树效果就不会变得很好。

第六章 ： 支持向量机

第六章支持向量机主要讲根据超平面来对于类别进行分类，我们要获取具有最大软边界的超平面。这一张主要讲了SVM的概念，核函数，KKT，对偶问题与支持向量回归。满足条件的wxi+yi=+-1的样本（xi，yi）被称为支持向量。

当SVM处理不可分问题时，首先将案例使用核函数映射到高位平面，此时目标数据就可以被分解。当我们遇到噪声数据时或者过拟合时，可以使用软边界来处理。正则化可以允许SVM在某些样本犯错误。

第七章 ： K近邻

第七章KNN讲了使用被动学习算法，基于相似度，让不可见得样本进行分类。这一张主要讲了KNN概念，距离函数与参数K。距离函数用来说明样本之间的相似度。参数k代表样本点的最相似的点个数。当用于分类时，输出值是离散的。当用于回归时，输出值是连续值。欧几里得距离与曼哈顿距离是最常用的距离。我们常用交叉验证来获取合适的K值。

第八章 ： 神经网络

第八章神经网络讲述了依据大量未知输入所构建的函数，工作方式类似于人类大脑。本章主要讲述了ANN的基本概念，激活函数，BP算法。神经网络由三层组成：输入层，隐含层与输出层。输入层负责接受外部的参数，隐含层与输出层负责处理信息，输出层输出结果。感知机是最简单的ANN的模型，不需要隐含层的加入。我们在神经网络运行时，会给与每个神经元赋予权重。之后使用BP算法，将预测值与输出值得差值得到误差函数，根据梯度下降来动态调整权重分布，以获得更好的泛化能力。

深度学习是机器学习中的一种，具有更深的网络结构。与普通机器学习算法相比较，更适合处理大数据，会随着数据量的上升，表现也会随之提升。

第九章 ： 半监督学习

第九章半监督学习是利用打标数据或者未打标数据，来进行半监督学习。常用的方法有：半监督的图学习，半监督的SVM，半监督的聚类与生成模型。

半监督的SVM就是利用打标数据来进行分类。

半监督聚类使用额外监控信息来实现聚类。

第十章 ： 集成学习

第十章集成学习主要讲了针对同一问题，采用不同的学习模型方法来共同学习预测。常用的方法有：线性方法与非线性方法。线性方法包括boosting，非线性方法包括bagging。Ada是非常著名的boosting方法。

目前有三种融合策略，平均法，投票法，学习法。

多样性检测一般使用不合度量与相关系数。

第十一章 ： 线性回归

第十一章介绍了线性回归的基本定义与几种回归方式。回归模型的返回值是连续的。回归模型是建立一种输入至于输出的关系，并且用起来预测新样本的输出值。回归学习是监督学习。典型的线性回归学习包括罗素回归与岭回归。

感谢打赏，手动狗头；)

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0xFRTE4MjU0MjkwNzM2_size_16_color_FFFFFF_t_70][]         ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0xFRTE4MjU0MjkwNzM2_size_16_color_FFFFFF_t_70 1][]

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0xFRTE4MjU0MjkwNzM2_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20190109234556809.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0xFRTE4MjU0MjkwNzM2,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0xFRTE4MjU0MjkwNzM2_size_16_color_FFFFFF_t_70 1]: /images/20220329/84a61529ab334ca6b336a78cd3dd1c5f.png