Apache Spark 3.0 机器学习库（MLlib）指南

ゝ一纸荒年。 2022-12-04 07:50 249阅读 0赞

学习地址：[https://spark.apache.org/docs/latest/ml-guide.html][https_spark.apache.org_docs_latest_ml-guide.html]

# 学习目录 #

### **[MLlib：主要指南][https_spark.apache.org_docs_latest_ml-guide.html]** ###

*  [基本统计][Link 1]
 *  [数据源][Link 2]
 *  [流水线][Link 3]
 *  [提取，转换和选择特征][Link 4]
 *  [分类与回归][Link 5]
 *  [聚类][Link 6]
 *  [协同过滤][Link 7]
 *  [频繁模式挖掘][Link 8]
 *  [模型选择和调整][Link 9]
 *  [进阶主题][Link 10]

### **[MLlib：基于RDD的API指南][MLlib_RDD_API]** ###

*  [资料类型][Link 11]
 *  [基本统计][Link 12]
 *  [分类与回归][Link 13]
 *  [协同过滤][Link 14]
 *  [聚类][Link 15]
 *  [降维][Link 16]
 *  [特征提取和转换][Link 17]
 *  [频繁模式挖掘][Link 18]
 *  [评估指标][Link 19]
 *  [PMML模型导出][PMML]
 *  [优化（开发人员）][Link 20]

# **机器学习库（MLlib）指南** #

MLlib是Spark的机器学习（ML）库。其目标是使实用的机器学习可扩展且容易。在较高级别，它提供了以下工具：

*  ML算法：常见的学习算法，例如分类，回归，聚类和协作过滤
 *  特征化：特征提取，变换，降维和选择
 *  管道：用于构建，评估和调整ML管道的工具
 *  持久性：保存和加载算法，模型和管道
 *  实用程序：线性代数，统计信息，数据处理等

# **声明：基于DataFrame的API是主要API** #

**基于MLlib RDD的API现在处于维护模式。**

从Spark 2.0开始，软件包中基于[RDD][]的API `spark.mllib`已进入维护模式。现在，用于Spark的主要机器学习API是[软件包中][Link 21]基于[DataFrame][Link 21]的API `spark.ml`。

*这意味着什么？*

*  MLlib仍将`spark.mllib`通过错误修复来支持基于RDD的API 。
 *  MLlib不会向基于RDD的API添加新功能。
 *  在Spark 2.x发行版中，MLlib将向基于DataFrames的API添加功能，以与基于RDD的API达到功能奇偶。

*为什么MLlib切换到基于DataFrame的API？*

*  与RDD相比，DataFrames提供了更加用户友好的API。DataFrames的许多好处包括Spark数据源，SQL / DataFrame查询，Tungsten和Catalyst优化以及跨语言的统一API。
 *  用于MLlib的基于DataFrame的API为ML算法和多种语言提供了统一的API。
 *  DataFrame有助于实际的ML管道，特别是功能转换。有关详细信息，请参见[管道指南][Link 3]。

*什么是“ Spark ML”？*

*  “ Spark ML”不是正式名称，但有时用于指代基于MLlib DataFrame的API。这主要归因于`org.apache.spark.ml`基于DataFrame的API使用的Scala软件包名称，以及我们最初用来强调管道概念的“ Spark ML Pipelines”一词。

*MLlib是否已弃用？*

*  不会。MLlib同时包含基于RDD的API和基于DataFrame的API。基于RDD的API现在处于维护模式。但是既不弃用API，也不弃用MLlib。

# **依存关系** #

MLlib使用线性代数程序包[Breeze][]，该程序依赖于 [netlib-java][]进行优化的数值处理。如果本机库[1][]在运行时不可用，您将看到警告消息，而将使用纯JVM实现。

由于运行时专有二进制文件存在许可问题，因此`netlib-java`默认情况下，我们不包含的本机代理。要配置`netlib-java`/ Breeze使用系统优化的二进制文件，请包含 `com.github.fommil.netlib:all:1.1.2`（或使用构建Spark `-Pnetlib-lgpl`）作为项目的依赖项，并阅读[netlib-java][]文档以获取平台的其他安装说明。

最受欢迎的本机BLAS（例如[Intel MKL][]，[OpenBLAS][]）可以在单个操作中使用多个线程，这可能与Spark的执行模型冲突。

将这些BLAS实现配置为使用单个线程进行操作实际上可以提高性能（请参阅[SPARK-21305][]）。通常，最好将此配置与每个Spark任务配置为使用的内核数匹配，默认情况下为1，通常保留为1。

请参考以下资源，以了解如何配置这些BLAS实现使用的线程数：[Intel MKL][Intel MKL 1]或[Intel oneMKL][]和[OpenBLAS][OpenBLAS 1]。请注意，如果未在系统中正确配置nativeBLAS，则将使用java实现（f2jBLAS）作为后备选项。

要在Python中使用MLlib，您将需要[NumPy][] 1.4版或更高版本。

# **3.0中的亮点** #

以下列表突出显示了`3.0` Spark版本中添加到MLlib的一些新功能和增强功能：

*  多列支持已添加到`Binarizer`（[SPARK-23578][]），`StringIndexer`（[SPARK-11215][]），`StopWordsRemover`（[SPARK-29808][]）和PySpark `QuantileDiscretizer`（[SPARK-22796][]）。
 *  添加了基于树的特征转换（[SPARK-13677][SPARK-13_677]）。
 *  添加了两个新的评估器`MultilabelClassificationEvaluator`（[SPARK-16692][]）和`RankingEvaluator`（[SPARK-28045][]）。
 *  在`DecisionTreeClassifier/Regressor`（[SPARK-19591][]），`RandomForestClassifier/Regressor`（[SPARK-9478][]），`GBTClassifier/Regressor`（[SPARK-9612][]）， `MulticlassClassificationEvaluator`（[SPARK-24101][]），`RegressionEvaluator`（[SPARK-24102][]），`BinaryClassificationEvaluator`（[SPARK-24103][]），`BisectingKMeans`（[SPARK-30351][]），`KMeans`（[SPARK -29967][]）和`GaussianMixture`（[SPARK-30102][]）。
 *  `PowerIterationClustering`添加了R API （[SPARK-19827][]）。
 *  添加了Spark ML侦听器，用于跟踪ML管道状态（[SPARK-23674][]）。
 *  适合验证集已添加到Python的Gradient Boosted Trees（[SPARK-24333][]）中。
 *  [`RobustScaler`][RobustScaler]添加了变压器（[SPARK-28399][]）。
 *  [`Factorization Machines`][Factorization Machines]添加了分类器和回归器（[SPARK-29224][]）。
 *  添加了高斯朴素贝叶斯分类器（[SPARK-16872][]）和互补朴素贝叶斯分类器（[SPARK-29942][]）。
 *  Scala和Python（[SPARK-28958][]）之间的ML函数奇偶校验。
 *  `predictRaw`在所有分类模型中公开。`predictProbability`在除`LinearSVCModel` （[SPARK-30358][]）之外的所有分类模型中公开。

# **迁移指南** #

现在，迁移指南已存档[在此页面上][Link 22]。

1.  要了解有关系统优化的本机的好处和背景的更多信息，您可能希望观看Sam Halliday的ScalaX关于[Scala中的高性能线性代数的][Scala]演讲。 [↩][Link 23]

# **MLlib：基于RDD的API** #

本页记录了MLlib指南中有关基于RDD的API（`spark.mllib`程序包）的各个部分。请参阅基于DataFrame的API（程序包）的[MLlib主指南][https_spark.apache.org_docs_latest_ml-guide.html]，该[指南][https_spark.apache.org_docs_latest_ml-guide.html]`spark.ml`现在是MLlib的主要API。

*  [资料类型][Link 11]
 *  [基本统计][Link 12]
    
     *  [摘要统计][Link 24]
     *  [相关性][Link 25]
     *  [分层抽样][Link 26]
     *  [假设检验][Link 27]
     *  [流重要性测试][Link 28]
     *  [随机数据生成][Link 29]
 *  [分类与回归][Link 13]
    
     *  [线性模型（SVM，逻辑回归，线性回归）][SVM]
     *  [朴素的贝叶斯][Link 30]
     *  [决策树][Link 31]
     *  [树木合奏（随机森林和梯度生长的树木）][Link 32]
     *  [等渗回归][Link 33]
 *  [协同过滤][Link 14]
    
     *  [交替最小二乘法（ALS）][ALS]
 *  [聚类][Link 15]
    
     *  [k均值][k]
     *  [高斯混合][Link 34]
     *  [功率迭代聚类（PIC）][PIC]
     *  [潜在狄利克雷分配（LDA）][LDA]
     *  [均等k均值][k 1]
     *  [流式均值][Link 35]
 *  [降维][Link 16]
    
     *  [奇异值分解（SVD）][SVD]
     *  [主成分分析（PCA）][PCA]
 *  [特征提取和转换][Link 17]
 *  [频繁模式挖掘][Link 18]
    
     *  [FP增长][FP]
     *  [关联规则][Link 36]
     *  [前缀跨度][Link 37]
 *  [评估指标][Link 19]
 *  [PMML模型导出][PMML]
 *  [优化（开发人员）][Link 20]
    
     *  [随机梯度下降][Link 38]
     *  [有限内存BFGS（L-BFGS）][BFGS_L-BFGS]

[https_spark.apache.org_docs_latest_ml-guide.html]: https://spark.apache.org/docs/latest/ml-guide.html
[Link 1]: https://spark.apache.org/docs/latest/ml-statistics.html
[Link 2]: https://spark.apache.org/docs/latest/ml-datasource.html
[Link 3]: https://spark.apache.org/docs/latest/ml-pipeline.html
[Link 4]: https://spark.apache.org/docs/latest/ml-features.html
[Link 5]: https://spark.apache.org/docs/latest/ml-classification-regression.html
[Link 6]: https://spark.apache.org/docs/latest/ml-clustering.html
[Link 7]: https://spark.apache.org/docs/latest/ml-collaborative-filtering.html
[Link 8]: https://spark.apache.org/docs/latest/ml-frequent-pattern-mining.html
[Link 9]: https://spark.apache.org/docs/latest/ml-tuning.html
[Link 10]: https://spark.apache.org/docs/latest/ml-advanced.html
[MLlib_RDD_API]: https://spark.apache.org/docs/latest/mllib-guide.html
[Link 11]: https://spark.apache.org/docs/latest/mllib-data-types.html
[Link 12]: https://spark.apache.org/docs/latest/mllib-statistics.html
[Link 13]: https://spark.apache.org/docs/latest/mllib-classification-regression.html
[Link 14]: https://spark.apache.org/docs/latest/mllib-collaborative-filtering.html
[Link 15]: https://spark.apache.org/docs/latest/mllib-clustering.html
[Link 16]: https://spark.apache.org/docs/latest/mllib-dimensionality-reduction.html
[Link 17]: https://spark.apache.org/docs/latest/mllib-feature-extraction.html
[Link 18]: https://spark.apache.org/docs/latest/mllib-frequent-pattern-mining.html
[Link 19]: https://spark.apache.org/docs/latest/mllib-evaluation-metrics.html
[PMML]: https://spark.apache.org/docs/latest/mllib-pmml-model-export.html
[Link 20]: https://spark.apache.org/docs/latest/mllib-optimization.html
[RDD]: https://spark.apache.org/docs/latest/rdd-programming-guide.html#resilient-distributed-datasets-rdds
[Link 21]: https://spark.apache.org/docs/latest/sql-programming-guide.html
[Breeze]: http://www.scalanlp.org/
[netlib-java]: https://github.com/fommil/netlib-java
[1]: https://spark.apache.org/docs/latest/ml-guide.html#fn:1
[Intel MKL]: https://software.intel.com/en-us/mkl
[OpenBLAS]: http://www.openblas.net/
[SPARK-21305]: https://issues.apache.org/jira/browse/SPARK-21305
[Intel MKL 1]: https://software.intel.com/en-us/articles/recommended-settings-for-calling-intel-mkl-routines-from-multi-threaded-applications
[Intel oneMKL]: https://software.intel.com/en-us/onemkl-linux-developer-guide-improving-performance-with-threading
[OpenBLAS 1]: https://github.com/xianyi/OpenBLAS/wiki/faq#multi-threaded
[NumPy]: http://www.numpy.org/
[SPARK-23578]: https://issues.apache.org/jira/browse/SPARK-23578
[SPARK-11215]: https://issues.apache.org/jira/browse/SPARK-11215
[SPARK-29808]: https://issues.apache.org/jira/browse/SPARK-29808
[SPARK-22796]: https://issues.apache.org/jira/browse/SPARK-22796
[SPARK-13_677]: https://issues.apache.org/jira/browse/SPARK-13677
[SPARK-16692]: https://issues.apache.org/jira/browse/SPARK-16692
[SPARK-28045]: https://issues.apache.org/jira/browse/SPARK-28045
[SPARK-19591]: https://issues.apache.org/jira/browse/SPARK-19591
[SPARK-9478]: https://issues.apache.org/jira/browse/SPARK-9478
[SPARK-9612]: https://issues.apache.org/jira/browse/SPARK-9612
[SPARK-24101]: https://issues.apache.org/jira/browse/SPARK-24101
[SPARK-24102]: https://issues.apache.org/jira/browse/SPARK-24102
[SPARK-24103]: https://issues.apache.org/jira/browse/SPARK-24103
[SPARK-30351]: https://issues.apache.org/jira/browse/SPARK-30351
[SPARK -29967]: https://issues.apache.org/jira/browse/SPARK-29967
[SPARK-30102]: https://issues.apache.org/jira/browse/SPARK-30102
[SPARK-19827]: https://issues.apache.org/jira/browse/SPARK-19827
[SPARK-23674]: https://issues.apache.org/jira/browse/SPARK-23674
[SPARK-24333]: https://issues.apache.org/jira/browse/SPARK-24333
[RobustScaler]: https://spark.apache.org/docs/latest/ml-features.html#robustscaler
[SPARK-28399]: https://issues.apache.org/jira/browse/SPARK-28399
[Factorization Machines]: https://spark.apache.org/docs/latest/ml-classification-regression.html#factorization-machines
[SPARK-29224]: https://issues.apache.org/jira/browse/SPARK-29224
[SPARK-16872]: https://issues.apache.org/jira/browse/SPARK-16872
[SPARK-29942]: https://issues.apache.org/jira/browse/SPARK-29942
[SPARK-28958]: https://issues.apache.org/jira/browse/SPARK-28958
[SPARK-30358]: https://issues.apache.org/jira/browse/SPARK-30358
[Link 22]: https://spark.apache.org/docs/latest/ml-migration-guide.html
[Scala]: http://fommil.github.io/scalax14/#/
[Link 23]: https://spark.apache.org/docs/latest/ml-guide.html#fnref:1
[Link 24]: https://spark.apache.org/docs/latest/mllib-statistics.html#summary-statistics
[Link 25]: https://spark.apache.org/docs/latest/mllib-statistics.html#correlations
[Link 26]: https://spark.apache.org/docs/latest/mllib-statistics.html#stratified-sampling
[Link 27]: https://spark.apache.org/docs/latest/mllib-statistics.html#hypothesis-testing
[Link 28]: https://spark.apache.org/docs/latest/mllib-statistics.html#streaming-significance-testing
[Link 29]: https://spark.apache.org/docs/latest/mllib-statistics.html#random-data-generation
[SVM]: https://spark.apache.org/docs/latest/mllib-linear-methods.html
[Link 30]: https://spark.apache.org/docs/latest/mllib-naive-bayes.html
[Link 31]: https://spark.apache.org/docs/latest/mllib-decision-tree.html
[Link 32]: https://spark.apache.org/docs/latest/mllib-ensembles.html
[Link 33]: https://spark.apache.org/docs/latest/mllib-isotonic-regression.html
[ALS]: https://spark.apache.org/docs/latest/mllib-collaborative-filtering.html#collaborative-filtering
[k]: https://spark.apache.org/docs/latest/mllib-clustering.html#k-means
[Link 34]: https://spark.apache.org/docs/latest/mllib-clustering.html#gaussian-mixture
[PIC]: https://spark.apache.org/docs/latest/mllib-clustering.html#power-iteration-clustering-pic
[LDA]: https://spark.apache.org/docs/latest/mllib-clustering.html#latent-dirichlet-allocation-lda
[k 1]: https://spark.apache.org/docs/latest/mllib-clustering.html#bisecting-kmeans
[Link 35]: https://spark.apache.org/docs/latest/mllib-clustering.html#streaming-k-means
[SVD]: https://spark.apache.org/docs/latest/mllib-dimensionality-reduction.html#singular-value-decomposition-svd
[PCA]: https://spark.apache.org/docs/latest/mllib-dimensionality-reduction.html#principal-component-analysis-pca
[FP]: https://spark.apache.org/docs/latest/mllib-frequent-pattern-mining.html#fp-growth
[Link 36]: https://spark.apache.org/docs/latest/mllib-frequent-pattern-mining.html#association-rules
[Link 37]: https://spark.apache.org/docs/latest/mllib-frequent-pattern-mining.html#prefix-span
[Link 38]: https://spark.apache.org/docs/latest/mllib-optimization.html#stochastic-gradient-descent-sgd
[BFGS_L-BFGS]: https://spark.apache.org/docs/latest/mllib-optimization.html#limited-memory-bfgs-l-bfgs