HDPCD-Java-复习笔记（19）

秒速五厘米 2022-06-07 10:47 185阅读 0赞

**Hive**

***Apache******Hive***maintains metadata information in a **metastore** to generate tables. A Hive table consists of：

<table> 
 <tbody> 
 <tr> 
 <td> ·&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; A schema stored in the metastore </td> 
 </tr> 
 <tr> 
 <td> ·&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Data stored on HDFS </td> 
 </tr> 
 </tbody> 
</table>

**HiveQL**

Hive converts HiveQL commands into MapReduce jobs。

**Hive and Pig**

**Pig** is designed to **move and restructure data**, while **Hive** is built to**analyze data**.For most use cases:

Pig -- Is a good choice for ETL jobs, where unstructured data is reformatted so that it is easier to define a structure to it.

Hive -- Is a good choice when you want to query data that has a certain known structure.

**Comparing Hive to SQL**

<table> 
 <thead> 
 <tr> 
 <td style="background:#DDDDDD"> SQL Datatypes </td> 
 <td style="background:#DDDDDD"> SQL Semantics </td> 
 </tr> 
 </thead> 
 <tbody> 
 <tr> 
 <td> INT </td> 
 <td> SELECT,&nbsp;LOAD,&nbsp;INSERT&nbsp;from query </td> 
 </tr> 
 <tr> 
 <td> TINYINT/SMALLINT/BIGINT </td> 
 <td> Expressions in&nbsp;WHERE&nbsp;and&nbsp;HAVING </td> 
 </tr> 
 <tr> 
 <td> BOOLEAN </td> 
 <td> GROUP&nbsp;BY,&nbsp;ORDER&nbsp;BY,&nbsp;SORT&nbsp;BY </td> 
 </tr> 
 <tr> 
 <td> FLOAT </td> 
 <td> CLUSTER&nbsp;BY,&nbsp;DISTRIBUTE&nbsp;BY </td> 
 </tr> 
 <tr> 
 <td> DOUBLE </td> 
 <td> Sub-queries in&nbsp;FROM&nbsp;clause </td> 
 </tr> 
 <tr> 
 <td> STRING </td> 
 <td> ROLLUP </td> 
 </tr> 
 <tr> 
 <td> BINARY </td> 
 <td> CUBE </td> 
 </tr> 
 <tr> 
 <td> TIMESTAMP </td> 
 <td> UNION </td> 
 </tr> 
 <tr> 
 <td> ARRAY,&nbsp;MAP,&nbsp;STRUCT,&nbsp;UNION </td> 
 <td> LEFT,&nbsp;RIGHT&nbsp;and&nbsp;FULL&nbsp;INNER/OUTER&nbsp;JOIN </td> 
 </tr> 
 <tr> 
 <td> DECIMAL </td> 
 <td> CROSS&nbsp;JOIN,&nbsp;LEFT&nbsp;SEMI&nbsp;JOIN </td> 
 </tr> 
 <tr> 
 <td> CHAR </td> 
 <td> Windowing functions (OVER,&nbsp;RANK, etc.) </td> 
 </tr> 
 <tr> 
 <td> VARCHAR </td> 
 <td> Sub-queries for&nbsp;IN/NOT&nbsp;IN,&nbsp;HAVING </td> 
 </tr> 
 <tr> 
 <td> DATE </td> 
 <td> EXISTS&nbsp;/&nbsp;NOT&nbsp;EXISTS </td> 
 </tr> 
 <tr> 
 <td> INTERSECT,&nbsp;EXCEPT </td> 
 </tr> 
 </tbody> 
</table>

**Hive Architecture**

**![Center][]**

<table> 
 <tbody> 
 <tr> 
 <td> Issuing Commands </td> 
 <td> Using the Hive CLI, a Web interface, or a Hive&nbsp;JDBC/ODBC&nbsp;client, a Hive query is submitted to the HiveServer </td> 
 </tr> 
 <tr> 
 <td> Hive Query Plan </td> 
 <td> The Hive query is compiled, optimized and planned as a MapReduce job </td> 
 </tr> 
 <tr> 
 <td> MapReduce Job Executes </td> 
 <td> The corresponding MapReduce job is executed on the Hadoop cluster </td> 
 </tr> 
 </tbody> 
</table>

**HiveQL**

Hive queries are written using the HiveQL language, a SQL-like scripting language that simplifies the creation of MapReduce jobs. With HiveQL, data analysts can focus on**answering questions about the data**, and let the Hive framework**convert the HiveQL into a MapReduce job**.

**Hive User-Defined Functions**

Hive has **three different types** of **User-Defined Functions**:

<table style="font-weight:bold"> 
 <tbody> 
 <tr> 
 <td> UDF </td> 
 <td> A single row is input, and a single row is output </td> 
 </tr> 
 <tr> 
 <td> UDAF (User-Defined Aggregate Function) </td> 
 <td> Multiple rows are input, and a single row is output </td> 
 </tr> 
 <tr> 
 <td> UDTF (User-Defined Table-generating Function) </td> 
 <td> A single row is input, and multiple rows (i.e. a table) are output </td> 
 </tr> 
 </tbody> 
</table>

The Hive API contains parent classes for writing each type of User-Defined Function: the **UDF**class for UDF functions, the **UDAF** class for UDAF functions, and the **GenericUDTF** class for writing UDTF functions.

**Writing a Hive UDF**

Extends the org.apache.hadoop.hive.ql.exec.UDF class

package hiveudfs;
    
    import org.apache.hadoop.hive.ql.exec.UDF;
    import org.apache.hadoop.io.DoubleWritable;
    
    
    public class ComputeShipping extends UDF {
        public static int originZip = 11344;
        public static double multiplier = 0.00045;
        DoubleWritable shippingAmt = new DoubleWritable();
    
        public DoubleWritable evaluate(int zip, double weight) {
            long distance = Math.abs(originZip - zip);
            double amt = (distance * multiplier) + weight;
            shippingAmt.set(amt);
    
            return shippingAmt;
        }
    }

**Invoking a Hive UDF**

To invoke a UDF from within a Hive script:

<table> 
 <tbody> 
 <tr> 
 <td> ·&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Register the JAR file that contains the&nbsp;UDF&nbsp;class, and </td> 
 </tr> 
 <tr> 
 <td> ·&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Define an alias for the function using the&nbsp;CREATE TEMPORARY FUNCTION&nbsp;command. </td> 
 </tr> 
 </tbody> 
</table>

*  ADD JAR /myapp/lib/myhiveudfs.jar;
 *  CREATE TEMPORARY FUNCTION ComputeShipping
 *  AS 'hiveudfs.ComputeShipping';
 *  FROM orders
 *  SELECT address,
 *  description,
 *  ComputeShipping(zip, weight);

**Overview of GenericUDF**

The GenericUDF class provides more features and benefits over UDF, including:

<table> 
 <tbody> 
 <tr> 
 <td> ·&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; The arguments passed in to a GenericUDF can be complex types, including non-writable types like&nbsp;struct,&nbsp;map&nbsp;and&nbsp;array. </td> 
 </tr> 
 <tr> 
 <td> ·&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; The return value can also be a complex type. </td> 
 </tr> 
 <tr> 
 <td> ·&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; A variable length of arguments can be passed in. </td> 
 </tr> 
 <tr> 
 <td> ·&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; A GenericUDF can perform operations that a UDF cannot support. </td> 
 </tr> 
 <tr> 
 <td> ·&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Better performance, due to lazy evaluation andshort-circuiting. </td> 
 </tr> 
 </tbody> 
</table>

**The GenericUDF class declares three abstract methods:**

<table> 
 <tbody> 
 <tr> 
 <td> ObjectInspector initialize(ObjectInspector [] arguments) </td> 
 <td> The&nbsp;ObjectInspector&nbsp;(OI) instances represent thearguments for the function. The&nbsp;initialize&nbsp;method is invoked once per GenericUDF instance and allows you tovalidate the arguments passed in. </td> 
 </tr> 
 <tr> 
 <td> Object&nbsp; evaluate(GenericUDF.DeferredObject [] arguments) </td> 
 <td> Similar to UDF, this method gets passed in the arguments and returns the result of the function call. </td> 
 </tr> 
 <tr> 
 <td> String&nbsp; getDisplayString(String [] children) </td> 
 <td> Return a string that gets displayed from the explain command. </td> 
 </tr> 
 </tbody> 
</table>

**Example of a GenericUDF**

class ComplexUDFExample extends GenericUDF {
 ListObjectInspector listOI;
 StringObjectInspector elementOI;
 
 public String getDisplayString(String[] arg0) {
 return "arrayContainsExample()";
 }
 
 public ObjectInspector initialize(ObjectInspector[] arguments) {
 if (arguments.length != 2) {
 throw new UDFArgumentLengthException(
 "method takes 2 arguments: List<T>, T");
 }
 
 // Verify we received the right object types.
 ObjectInspector a = arguments[0];
 ObjectInspector b = arguments[1];
 
 if (!(a instanceof ListObjectInspector) ||
 !(b instanceof StringObjectInspector)) {
 throw new UDFArgumentException(
 "first argument must be a list / array, second argument must be a string");
 }
 
 this.listOI = (ListObjectInspector) a;
 this.elementOI = (StringObjectInspector) b;
 
 // Verify that the list contains strings
 if (!(listOI.getListElementObjectInspector() instanceof StringObjectInspector)) {
 throw new UDFArgumentException(
 "first argument must be a list of strings");
 }
 
 // the return type of our function is a boolean
 return PrimitiveObjectInspectorFactory.javaBooleanObjectInspector;
 }
 
 public Object evaluate(DeferredObject[] arguments) {
 // get the list and string from the deferred objects
 // using the object inspectors
 List<String> list = (List<String>) this.listOI.getList(arguments[0].get());
 String arg = elementOI.getPrimitiveJavaObject(arguments[1].get());
 
 // check for nulls
 if ((list == null) || (arg == null)) {
 return null;
 }
 
 // see if our list contains the value we need
 for (String s : list) {
 if (arg.equals(s)) {
 return new Boolean(true);
 }
 }
 
 return new Boolean(false);
 }
 }

**Overview of HCatalog**

HCatalog,a metadata and table management system, helps enableschema on read in Hadoop. HCatalog has the following features:

<table> 
 <tbody> 
 <tr> 
 <td> ·&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Makes the Hive metastore available to users of other toolson Hadoop. </td> 
 </tr> 
 <tr> 
 <td> ·&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Provides connectors for Map Reduce and Pig so that users of those tools can read data from and write data to Hive’s warehouse. </td> 
 </tr> 
 <tr> 
 <td> ·&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Allows users to share data and metadata across Hive, Pig, and MapReduce. </td> 
 </tr> 
 <tr> 
 <td> ·&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Provides a relational view, through SQL-like language (HiveQL), to data within Hadoop. </td> 
 </tr> 
 <tr> 
 <td> ·&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Allows users to write their applications without being concerned how or where the data is stored. </td> 
 </tr> 
 <tr> 
 <td> ·&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Insulates users from schema and storage format changes. </td> 
 </tr> 
 </tbody> 
</table>

**HCatalog in the Ecosystem**

**![Center 1][]**

HCatalog provides tableabstraction, which abstracts some of the details about data like:

<table> 
 <tbody> 
 <tr> 
 <td> ·&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; How the data is stored. </td> 
 </tr> 
 <tr> 
 <td> ·&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Where the data resides on the filesystem </td> 
 </tr> 
 <tr> 
 <td> ·&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; What format that data is in </td> 
 </tr> 
 <tr> 
 <td> ·&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; What the schema is of the data </td> 
 </tr> 
 </tbody> 
</table>

**HCatInputFormat and HCatOutputFormat**

For Java MapReduce applications, the HCatInputFormat and HCatOutputFormat classes can be used to read and write data using HCatalog schemas and data types.

Here is an **example** of a Mapper that uses HCatInputFormat:

public static class Map extends Mapper<WritableComparable, HCatRecord, Text, IntWritable> {
 String name;
 int age;
 double gpa;
 
 protected void map(WritableComparable key, HCatRecord value, Context context)
 throws IOException, InterruptedException {
 name = (String) value.get(0);
 age = (Integer) value.get(1);
 gpa = (Double) value.get(2);
 context.write(new Text(name), new IntWritable(age));
 }
 }

The Job configuration looks like:

String principalID = System.getProperty(HCatConstants.HCAT_METASTORE_PRINCIPAL);
    if (principalID != null)
    	 conf.set(HCatConstants.HCAT_METASTORE_PRINCIPAL, principalID);
    Job job = Job.getInstance (conf, "SimpleRead");
    HCatInputFormat.setInput(job, InputJobInfo.create(dbName, tableName, null));
    job.setInputFormatClass(HCatInputFormat.class);

[Center]: /images/20220607/15e0f352f881489bbf52e3b098459f55.png
[Center 1]: /images/20220607/081a1e1cdc0b4202ad35f9c8c2e8899b.png