【Hadoop】HDFS操作文档指令集

电玩女神 2024-04-17 11:26 45阅读 0赞

#### HDFS常用指令 ####

--------------------

`几乎与shell命令一致`

*  常用指令

hdfs dfs -ls /
    hdfs dfs -put
    hdfs dfs -copyFromLocal
    hdfs dfs -moveFromLocal
    hdfs dfs -cat
    hdfs dfs -text
    hdfs dfs -get
    hdfs dfs -mkdir 
    hdfs dfs -mv  移动/改名
    hdfs dfs -getmerge
    hdfs dfs -rm
    hdfs dfs -rmdir
    hdfs dfs -rm -r

*  案例
    
    1.  创建文件夹

[imooc@imooc ~]$ hdfs dfs -ls /user/imooc
    drwxr-xr-x   - imooc supergroup          0 2023-02-19 01:54 /user/imooc/.sparkStaging
    drwxr-xr-x   - imooc supergroup          0 2023-02-17 20:12 /user/imooc/df.csv
    drwxr-xr-x   - imooc supergroup          0 2023-02-19 01:27 /user/imooc/df.parquet
    -rw-r--r--   1 imooc supergroup        223 2023-02-19 01:51 /user/imooc/job.yaml
    -rw-r--r--   1 imooc supergroup        177 2023-02-19 01:39 /user/imooc/qdb.yaml
    [imooc@imooc ~]$ hdfs dfs -mkdir /user/imooc/test_tmp
    [imooc@imooc ~]$ hdfs dfs -ls /user/imooc
    drwxr-xr-x   - imooc supergroup          0 2023-02-19 01:54 /user/imooc/.sparkStaging
    drwxr-xr-x   - imooc supergroup          0 2023-02-17 20:12 /user/imooc/df.csv
    drwxr-xr-x   - imooc supergroup          0 2023-02-19 01:27 /user/imooc/df.parquet
    -rw-r--r--   1 imooc supergroup        223 2023-02-19 01:51 /user/imooc/job.yaml
    -rw-r--r--   1 imooc supergroup        177 2023-02-19 01:39 /user/imooc/qdb.yaml
    drwxr-xr-x   - imooc supergroup          0 2023-02-19 23:02 /user/imooc/test_tmp

1.  查看目录下的文件

[imooc@imooc ~]$ hdfs dfs -ls
    drwxr-xr-x   - imooc supergroup          0 2023-02-19 01:54 .sparkStaging
    drwxr-xr-x   - imooc supergroup          0 2023-02-17 20:12 df.csv
    drwxr-xr-x   - imooc supergroup          0 2023-02-19 01:27 df.parquet
    -rw-r--r--   1 imooc supergroup        223 2023-02-19 01:51 job.yaml
    -rw-r--r--   1 imooc supergroup        177 2023-02-19 01:39 qdb.yaml

1.  查看文件夹总大小

[imooc@imooc ~]$ hdfs dfs -du -h /user
    2023-02-19 23:04:18,279 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
    390    390    /user/hive
    2.4 G  2.4 G  /user/imooc

1.  查看文件内容

[imooc@imooc ~]$ hdfs dfs -cat /user/imooc/job.yaml
    metrics:
      - /user/imooc/qdb.yaml
    inputs:
    output:
      jdbc:
        connectionUrl: "jdbc:mysql://192.168.72.134:3306/sjjc"
        user: "root"
        password: "*******"
        driver: "com.mysql.jdbc.Driver"
    explain: true
    showQuery: true

1.  移动文件

[imooc@imooc ~]$ hadoop fs -ls /user/imooc
    Found 6 items
    drwxr-xr-x   - imooc supergroup          0 2023-02-19 01:54 /user/imooc/.sparkStaging
    drwxr-xr-x   - imooc supergroup          0 2023-02-17 20:12 /user/imooc/df.csv
    drwxr-xr-x   - imooc supergroup          0 2023-02-19 01:27 /user/imooc/df.parquet
    -rw-r--r--   1 imooc supergroup        223 2023-02-19 01:51 /user/imooc/job.yaml
    -rw-r--r--   1 imooc supergroup        177 2023-02-19 01:39 /user/imooc/qdb.yaml
    drwxr-xr-x   - imooc supergroup          0 2023-02-19 23:02 /user/imooc/test_tmp
    [imooc@imooc ~]$ hadoop fs -mv /user/imooc/test_tmp /user
    [imooc@imooc ~]$ hadoop fs -ls /user
    Found 3 items
    drwxr-xr-x   - imooc supergroup          0 2022-07-23 21:06 /user/hive
    drwxr-xr-x   - imooc supergroup          0 2023-02-19 23:11 /user/imooc
    drwxr-xr-x   - imooc supergroup          0 2023-02-19 23:02 /user/test_tmp

1.  删除文件

[imooc@imooc ~]$ hadoop fs -rm -r /user/test_tmp
    Deleted /user/test_tmp
    [imooc@imooc ~]$ hadoop fs -ls /user
    Found 2 items
    drwxr-xr-x   - imooc supergroup          0 2022-07-23 21:06 /user/hive
    drwxr-xr-x   - imooc supergroup          0 2023-02-19 23:11 /user/imooc

1.  上传到hdfs系统

[imooc@imooc ~]$ hdfs dfs -put qdb.txt /user/imooc
    [imooc@imooc ~]$ hadoop fs -ls /user/imooc
    Found 6 items
    drwxr-xr-x   - imooc supergroup          0 2023-02-19 01:54 /user/imooc/.sparkStaging
    drwxr-xr-x   - imooc supergroup          0 2023-02-17 20:12 /user/imooc/df.csv
    drwxr-xr-x   - imooc supergroup          0 2023-02-19 01:27 /user/imooc/df.parquet
    -rw-r--r--   1 imooc supergroup        223 2023-02-19 01:51 /user/imooc/job.yaml
    -rw-r--r--   1 imooc supergroup        384 2023-02-19 23:13 /user/imooc/qdb.txt
    -rw-r--r--   1 imooc supergroup        177 2023-02-19 01:39 /user/imooc/qdb.yaml

1.  下载到本地

[imooc@imooc ~]$ hdfs dfs -get /user/imooc/qdb.txt /tmp
    [imooc@imooc ~]$ ll /tmp |grep qdb
    -rw-r--r--. 1 imooc imooc 384 2月  19 23:14 qdb.txt

--------------------