Spark核心类：SQLContext和DataFrame

分手后的思念是犯贱 2022-07-14 02:43 205阅读 0赞

http://[blog.csdn.net/pipisorry/article/details/53320669][blog.csdn.net_pipisorry_article_details_53320669]

## pyspark.sql.SQLContext ##

Main entry point for DataFrame and SQL functionality.

\[[pyspark.sql.SQLContext][]\][  
][Link 1]

[皮皮blog][Link 1]

# pyspark.sql.DataFrame #

A distributed collection of data grouped into named columns.

## spark df和pandas df    ##

spark df的操作基本和pandas df操作一样的\[[Pandas小记][Pandas](6) \]

### 相互转换 ###

从pandas\_df转换：

spark\_df = SQLContext.createDataFrame(pandas\_df)

sc = SparkContext(master='local[8]', appName='kmeans')
    sql_ctx = SQLContext(sc)
    lldf_rdd = sql_ctx.createDataFrame(lldf)

另外，createDataFrame支持从list转换spark\_df，其中list元素可以为tuple，dict，rdd

从spark\_df转换：

pandas\_df = spark\_df.toPandas()

toPandas()

Returns the contents of this [DataFrame][] as Pandas pandas.DataFrame.

Note that this method should only be used if the resulting Pandas’s DataFrame is expectedto be small, as all the data is loaded into the driver’s memory.

This is only available if Pandas is installed and available.

>>> df.toPandas()  
       age   name
    0    2  Alice
    1    5    Bob

\[[Spark与Pandas中DataFrame对比（详细）][Spark_Pandas_DataFrame]\]

## spark df方法 ##

rdd

Returns the content as an [pyspark.RDD][] of [Row][].

rollup(\*cols)

Create a multi-dimensional rollup for the current [DataFrame][] usingthe specified columns, so we can run aggregation on them.

>>> df.rollup("name", df.age).count().orderBy("name", "age").show()
    +-----+----+-----+
    | name| age|count|
    +-----+----+-----+
    | null|null|    2|
    |Alice|null|    1|
    |Alice|   2|    1|
    |  Bob|null|    1|
    |  Bob|   5|    1|
    +-----+----+-----+

select(\*cols)

Projects a set of expressions and returns a new [DataFrame][].

<table>     
 <tbody> 
  <tr> 
   <th>Parameters:</th> 
   <td>cols – list of column names (string) or expressions (<a href="http://spark.apache.org/docs/latest/api/python/pyspark.sql.html#pyspark.sql.Column" title="pyspark.sql.Column" rel="nofollow">Column</a>).If one of the column names is ‘*’, that column is expanded to include all columnsin the current DataFrame.</td> 
  </tr> 
 </tbody> 
</table>

>>> df.select('*').collect()
    [Row(age=2, name=u'Alice'), Row(age=5, name=u'Bob')]
    >>> df.select('name', 'age').collect()
    [Row(name=u'Alice', age=2), Row(name=u'Bob', age=5)]
    >>> df.select(df.name, (df.age + 10).alias('age')).collect()
    [Row(name=u'Alice', age=12), Row(name=u'Bob', age=15)]

selectExpr(\*expr)

Projects a set of SQL expressions and returns a new [DataFrame][].

This is a variant of [select()][select] that accepts SQL expressions.

>>> df.selectExpr("age * 2", "abs(age)").collect()
    [Row((age * 2)=4, abs(age)=2), Row((age * 2)=10, abs(age)=5)]

toDF(\*cols)

Returns a new class:DataFrame that with new specified column names

<table>     
 <tbody> 
  <tr> 
   <th>Parameters:</th> 
   <td>cols – list of new column names (string)</td> 
  </tr> 
 </tbody> 
</table>

>>> df.toDF('f1', 'f2').collect()
    [Row(f1=2, f2=u'Alice'), Row(f1=5, f2=u'Bob')]

persist(storageLevel=StorageLevel(False, True, False, False, 1)) [¶][Link 2]

Sets the storage level to persist its values across operationsafter the first time it is computed. This can only be used to assigna new storage level if the RDD does not have a storage level set yet.If no storage level is specified defaults to (MEMORY\_ONLY).

\[[pyspark.sql.DataFrame][DataFrame]\]

from: [http://blog.csdn.net/pipisorry/article/details/53320669][blog.csdn.net_pipisorry_article_details_53320669]

ref:

[blog.csdn.net_pipisorry_article_details_53320669]: http://blog.csdn.net/pipisorry/article/details/53320669
[pyspark.sql.SQLContext]: http://spark.apache.org/docs/latest/api/python/pyspark.sql.html#pyspark.sql.SQLContext
[Link 1]: http://blog.csdn.net/pipisorry
[Pandas]: http://blog.csdn.net/pipisorry/article/category/5916621
[DataFrame]: http://spark.apache.org/docs/latest/api/python/pyspark.sql.html#pyspark.sql.DataFrame
[Spark_Pandas_DataFrame]: http://www.lining0806.com/spark%e4%b8%8epandas%e4%b8%addataframe%e5%af%b9%e6%af%94/
[pyspark.RDD]: http://spark.apache.org/docs/latest/api/python/pyspark.html#pyspark.RDD
[Row]: http://spark.apache.org/docs/latest/api/python/pyspark.sql.html#pyspark.sql.Row
[select]: http://spark.apache.org/docs/latest/api/python/pyspark.sql.html#pyspark.sql.DataFrame.select
[Link 2]: http://spark.apache.org/docs/latest/api/python/pyspark.sql.html#pyspark.sql.DataFrame.persist