troubleshooting hadoop hdfs

  • web interface

    • namenode: port 50070

      • lists datanodes, status, stats, and file browser

    • datanodes: port 50075

      • file browser and access to logs

  • command line tools

    • +hadoop fs -hel+p

              [-cat [-ignoreCrc] <src> ...]
              [-chgrp [-R] GROUP PATH...]
              [-chmod [-R] <MODE[,MODE]... | OCTALMODE> PATH...]
              [-chown [-R] [OWNER][:[GROUP]] PATH...]
              [-copyFromLocal <localsrc> ... <dst>]
              [-copyToLocal [-ignoreCrc] [-crc] <src> ... <localdst>]
              [-count [-q] <path> ...]
              [-cp <src> ... <dst>]
              [-df [-h] [<path> ...]]
              [-du [-s] [-h] <path> ...]
              [-expunge]
              [-get [-ignoreCrc] [-crc] <src> ... <localdst>]
              [-getmerge [-nl] <src> <localdst>]
              [-help [cmd ...]]
              [-ls [-d] [-h] [-R] [<path> ...]]
              [-mkdir [-p] <path> ...]
              [-moveFromLocal <localsrc> ... <dst>]
              [-moveToLocal <src> <localdst>]
              [-mv <src> ... <dst>]
              [-put <localsrc> ... <dst>]
              [-rm [-f] [-r|-R] [-skipTrash] <src> ...]
              [-rmdir [--ignore-fail-on-non-empty] <dir> ...]
              [-setrep [-R] [-w] <rep> <path/file> ...]
              [-stat [format] <path> ...]
              [-tail [-f] <file>]
              [-test -[ezd] <path>]
              [-text [-ignoreCrc] <src> ...]
              [-touchz <path> ...]
              [-usage [cmd ...]]
  • hadoop hdfs -help

        [-report]
        [-safemode <enter | leave | get | wait>]
        [-saveNamespace]
        [-restoreFailedStorage true|false|check]
        [-refreshNodes]
        [-setQuota <quota> <dirname>...<dirname>]
        [-clrQuota <dirname>...<dirname>]
        [-setSpaceQuota <quota> <dirname>...<dirname>]
        [-clrSpaceQuota <dirname>...<dirname>]
        [-refreshServiceAcl]
        [-refreshUserToGroupsMappings]
        [refreshSuperUserGroupsConfiguration]
        [-printTopology]
        [-refreshNamenodes datanodehost:port]
        [-deleteBlockPool datanodehost:port blockpoolId [force]]
        [-setBalancerBandwidth <bandwidth>]
        [-fetchImage <local directory>]
        [-help [cmd]]
  • hadoop fsck - check and repair filesystem