大数据的一些基本命令汇总

野性酷女 2022-05-22 09:39 186阅读 0赞

**一，集群启动命令**

CDH 启动:  
/opt/cm-5.1.3/etc/init.d/cloudera-scm-server start

/opt/cm-5.1.3/etc/init.d/cloudera-scm-agent start

CDH 关闭:  
/opt/cm-5.1.3/etc/init.d/cloudera-scm-server stop

/opt/cm-5.1.3/etc/init.d/cloudera-scm-agent stop

启动mysql  
/usr/local/mysql/bin/mysqld\_safe --user=mysql &  （默认是启动的）  
登录   
/usr/local/mysql/bin/mysql --user=root -p123

1，全部节点启动懂zookeeper   
zkServer.sh start

2，在node1启动 hadoop   
start-dfs.sh    
3，在node1启动yarn  
start-yarn.sh  
4,node 3 node4启动   
yarn-daemon.sh start resourcemanager

//查看端口是否被占用  
netstat -anp |grep 9998  
启动：

zkServer.sh start   //启动zk

//启动kafka  
./bin/kafka-server-start.sh -daemon ./config/server.properties

//后台启动kafka-manager   
nohup bin/kafka-manager -Dconfig.file=conf/application.conf -Dhttp.port=9998 &

start-dfs.sh   
start-yarn.sh  
yarn-daemon.sh start resourcemanager

//启动spark  
/home/spark-2.2.0-bin-hadoop2.6/sbin/start-all.sh

启动Hbase  
/home/hbase-1.2.6/bin/start-hbase.sh

关闭：

yarn-daemon.sh stop resourcemanager  
stop-yarn.sh  
stop-dfs.sh     
zkServer.sh stop

//关闭spark  
./sbin/stop-all.sh

//关闭kafka  
./bin/kafka-server-stop.sh

关闭Hbase  
/home/hbase-1.2.6/bin/stop-hbase.sh

HDFS Web页面：  
50070

yarn Web页面：  
8088

beeline -u jdbchive2://localhost:10000 /

spark:  
7077  
export SPARK\_MASTER\_IP=node1   
export SPARK\_MASTER\_PORT=7077    
export SPARK\_WORKER\_CORES=1     
export SPARK\_WORKER\_INSTANCES=1    
export SPARK\_WORKER\_MEMORY=512m

./bin/spark-submit --class org.apache.spark.examples.SparkPi --master local lib/spark-examples-1.0.0-hadoop2.2.0.jar   
./bin/spark-submit --class org.apache.spark.examples.SparkPi --master spark://node1:7077 ./lib/spark-examples-1.6.1-hadoop2.6.0.jar  100

./bin/spark-submit --class org.apache.spark.examples.SparkPi --master local ./lib/spark-examples-1.6.1-hadoop2.6.0.jar  100

**二，端口排查**

**1，一个进程有多个端口，一个端口只能被一个进程占用**

**lsof -i tcp:9092  -- 查看9092端口被占用的是哪些程序**

**查看端口是否被占用：  
lsof -i:8082**

**2，显示网络相关信息：  
sudo netstat -anp|grep 8080**

**有数据代表被占用**

**3,Linux中怎么通过PID号找到对应的进程名及所在目录方法  
ps -aux |grep -v grep|grep 28990**

**4，通过进程名字 查看进程信息  
    ps -ef | grep 'nginx'  
      
      
5, jps -m | grep 'nginx'**

**三，kafka的一些命令**

查看topic某分区偏移量最大（小）值  
bin/kafka-run-class.sh kafka.tools.GetOffsetShell --topic hive-mdatabase-hostsltable  --time -1 --broker-list node86:9092 --partitions 0

bin/kafka-console-producer.sh --broker-list node2:9091,node3:9091 --topic fcy\_test  
bin/kafka-console-producer.sh --broker-list node4:9092,node5:9092,node6:9092 --topic dianyou\_wxgz\_test

bin/kafka-console-consumer.sh --bootstrap-server node4:9092,node5:9092,node6:9092 --topic dianyou\_wxgz\_test2 --from-beginning

\#\#\#展示命令\#\#\#   
/zywa/kafka/kafka\_2.11-1.1.0/bin/kafka-topics.sh --list --zookeeper node1:2181,node2:2181,node3:2181

\------------------------------------------------------------------------------------------------------------------------------------------------------  
\#\#\#创建topic\#\#\#  
/zywa/kafka/kafka\_2.11-1.1.0/bin/kafka-topics.sh --create --zookeeper node1:2181,node2:2181,node3:2181 --replication-factor 2 --partitions 1 --topic  wx\_test2

/zywa/kafka/kafka\_2.11-1.1.0/bin/kafka-topics.sh --create --zookeeper node1:2181,node2:2181,node3:2181 --replication-factor 2 --partitions 2 --topic  dianyou\_topic\_test  
\------------------------------------------------------------------------------------------------------------------------------------------------------

生产者命令：

/zywa/kafka/kafka\_2.11-1.1.0/bin/kafka-console-producer.sh --broker-list node2:9091,node3:9091  --topic dianyou\_filter\_test  
/zywa/kafka/kafka\_2.11-1.1.0/bin/kafka-console-producer.sh --broker-list node2:9091,node3:9091  --topic dianyou\_topic\_test

/zywa/kafka/kafka\_2.11-1.1.0/bin/kafka-console-consumer.sh --bootstrap-server node4:9092,node5:9092,node6:9092 --topic dianyou\_wxgz\_test2

/zywa/kafka/kafka\_2.11-1.1.0/bin/kafka-console-consumer.sh --bootstrap-server node4:9092,node5:9092,node6:9092 --topic dianyou\_filter

/zywa/kafka/kafka\_2.11-0.10.2.0/bin/kafka-console-producer.sh --broker-list node2.hadoop:9092 --topic aa  
/zywa/kafka/kafka\_2.11-0.10.2.0/bin/kafka-console-consumer.sh --zookeeper node2.hadoop:2181 --topic aa  
/zywa/kafka/kafka\_2.11-0.10.2.0/bin/kafka-console-consumer.sh --zookeeper node2.hadoop:2181 --topic aa

四，