Hadoop Commands –  A Comprehensive Guide for Big Data Enthusiasts

In the data management field, Hadoop is known as a giant for managing massive data. It consists of two key parts: HDFS, which is like a special filing system for storing all that data across many computers, and MapReduce, which helps process this data efficiently. Together, they allow for handling huge tasks by breaking them into smaller, manageable pieces. This makes it much easier and faster to work with really big datasets.

In this blog, we will explore these commands in detail, covering everything from managing files in HDFS to executing MapReduce jobs and managing the Hadoop command.

What are Hadoop’s Commands?

Hadoop commands are like tools for handling files in Hadoop, a system for managing big data. They help with tasks like organizing files, running special tasks (MapReduce), and managing Hadoop itself. You can use these commands to do things like see a list of files, make new folders, and transfer files. It’s handy for working with lots of data.

Get a confirmed ₹35,000 total stipend on your first internship with our data analyst course with placement.

HDFS Commands

HDFS is the distributed file system at the core of Hadoop commands. Let’s delve into the essential HDFS commands for file and directory operations.

File and Directory Operations 

Hadoop provides a range of commands for efficient management of files and directories within the Hadoop Distributed File System (HDFS). Here are some essential commands for interacting with files and directories in HDFS:

• ls- This Hadoop command lets users view detailed info on files and folders stored in the Hadoop Distributed File System (HDFS). This info covers things like sizes, permissions, and timestamps. It’s super handy for analyzing and moving around data in HDFS.

For example, to list the contents of the “/data” directory, we would run `hdfs dfs -ls /data`.

• mkdir – This Hadoop command helps create new folders/directories. When you give it a specific path, like `/data/output`, it forms a new folder called “output” inside an existing one, say “/data.” It does that because it is important for organizing data in HDFS, and lets you arrange it in a neat and nested way.

For example, let’s say you want to create a directory called “output” within the “/data” directory in HDFS. You would run this command hdfs dfs -mkdir /data/outputs, something like this:

import subprocess

# Define the HDFS directory path
directory_path = "/data/output"

# Create the directory using the hdfs dfs -mkdir command
command = ["hdfs", "dfs", "-mkdir", directory_path]

# Execute the command
subprocess.run(command)

• copyFromLocal – This is a helpful Hadoop command that moves files from your computer to Hadoop’s distributed file system, known as HDFS. If the destination folder doesn’t exist in HDFS, it creates one. This way, you can easily manage your files within HDFS.

For example, to copy a file named “file.txt” from the local file system to HDFS at “/data/input”, we would execute `hdfs dfs -copyFromLocal file.txt /data/input`. Here’s how you will do it:

import subprocess

# Define the HDFS file path and local file path
hdfs_file_path = '/data/output/part-00000'
local_file_path = 'result.txt'

# Build the command to copy the file from HDFS to local file system
command = ['hdfs', 'dfs', '-copyToLocal', hdfs_file_path, local_file_path]

# Execute the command
subprocess.call(command)

• rm- This command is used for deleting files or folders. It not only removes the entry but also frees up storage space by deleting the associated data. This tool proves to be quite handy in maintaining the organization and efficiency of your HDFS.

For example, to delete a file named “file.txt” located at “/data/input”, we would run:

hdfs dfs -rm /data/input/file.txt

• mv- This Hadoop command allows you to move files or folders from one place to another in the Hadoop Distributed File System (HDFS). It’s like rearranging files on your computer but for big data. This command helps to reorganize data, and it’s done through Hadoop’s command line (CLI), which helps in managing and manipulating data stored in HDFS.

For example, `hdfs dfs -mv /data/input/file.txt /data/output` moves the file “file.txt” from “/data/input” to “/data/output”. Here’s how:

import subprocess

def transfer_file_or_directory(source_path, destination_path):
    command = f"hadoop fs -mv {source_path} {destination_path}"
    process = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    stdout, stderr = process.communicate()
    
    if process.returncode != 0:
        error_message = stderr.decode("utf-8").strip()
        raise Exception(f"Error occurred while transferring: {error_message}")

# Example usage
source = "/data/input/file.txt"
destination = "/data/output"
transfer_file_or_directory(source, destination)

File Manipulation

Hadoop provides several commands for efficient file manipulation within the Hadoop Distributed File System (HDFS). Here are some of the commonly used commands for file manipulation

• cat– It is a handy Hadoop command for shuffling files in Hadoop’s storage system, HDFS. Picture it as tidying up files on your computer. Just type this in the command line, and you can smoothly handle and structure your data kept in HDFS.

For example, `hdfs dfs -cat /data/input/file.txt` displays the contents of “file.txt”. 

import subprocess

def display_hdfs_file(file_path):
    command = ['hdfs', 'dfs', '-cat', file_path]
    try:
        output = subprocess.check_output(command)
        print(output.decode('utf-8'))
    except subprocess.CalledProcessError as e:
        print(f"Error: {e}")

# Example usage
display_hdfs_file('/data/input/file.txt')

• get– This command lets you fetch a file from the Hadoop Distributed File System (HDFS) and easily store it on the system. You just need to specify the file’s location in HDFS as `<hdfs_path>` and where you want to save it locally as `<local_path>`. 

For example, `hdfs dfs -get /data/output/part-00000 result.txt` downloads “part-00000” from HDFS and saves it as “result.txt” locally. something like this

import subprocess

def get_file_from_hdfs(hdfs_path, local_path):
    command = ['hdfs', 'dfs', '-get', hdfs_path, local_path]
    try:
        subprocess.check_output(command)
        print(f"File '{hdfs_path}' downloaded and saved as '{local_path}'")
    except subprocess.CalledProcessError as e:
        print(f"Error downloading file: {e}")

# Example usage
hdfs_path = '/data/output/part-00000'
local_path = 'result.txt'
get_file_from_hdfs(hdfs_path, local_path)

• put– This command in Hadoop lets you transfer a file from your computer to the Hadoop Distributed File System (HDFS). Just replace `<local_path>` with the file’s location on your computer and `<hdfs_path>` with where you want it in HDFS. It’s like copying a file to a special storage space. 

For example, `hdfs dfs -put data.csv /data/input` uploads a file named “data.csv” from the local filesystem to “/data/input” in HDFS. Like this: 

hdfs dfs -put /path/to/local_file.csv /destination/directory/in/hdfs

• appendToFile– A command of Hadoop that helps you add data from a local file to an HDFS one without replacing/overwriting the existing data. It’s like adding a new chapter to a book without erasing the old ones. This keeps the original content safe and sound.

For example, `hdfs dfs -appendToFile newdata.csv /data/input/data.csv` appends the contents of “newdata.csv” to the existing file “data.csv” in HDFS. 

import subprocess

def append_to_hdfs_file(local_file_path, hdfs_file_path):
    command = ['hdfs', 'dfs', '-appendToFile', local_file_path, hdfs_file_path]
    subprocess.run(command, check=True)

# Example usage
local_file = 'newdata.csv'
hdfs_file = '/data/input/data.csv'

append_to_hdfs_file(local_file, hdfs_file)

File Permissions

Managing file and directory permissions is a crucial aspect of data security and access control in the Hadoop Distributed File System (HDFS). It provides some commands to modify permissions and ownership. Let’s look at them

• chmod– This command helps to control user actions with a file or folder in HDFS. The `<mode>` part decides the new permissions, while `<path>` points to the place you’re changing. It decides who can read, write, or execute the file for the owner, group, and others.

For example, `hdfs dfs -chmod 755 /data/input/file.txt` sets the permissions of “file.txt” to read, write, and execute for the owner and read and execute for others.

#!/bin/bash

file_path="/data/input/file.txt"
permissions="755"

chmod "$permissions" "$file_path"

• chown-  This command in Hadoop lets you change who owns a file or folder in the Hadoop Distributed File System (HDFS). You just need to specify the `<owner>` you want to assign, along with the `<path>` of the file or folder. It’s handy for shifting ownership or regulating access in HDFS.

For example, `hdfs dfs -chown user1 /data/input/file.txt` changes the owner of “file.txt” to “user1”.

#!/bin/bash

owner="user1"
path="/data/input/file.txt"

hdfs dfs -chown $owner $path

echo "Ownership of $path changed to $owner"

• chgrp– This is one of those Hadoop commands that is used to change the group ownership of a file or directory. It enables users to assign a new group and path, simplifying the management of access rights for data in HDFS. This ensures that the correct groups have the necessary permissions to handle the data.

For example, `hdfs dfs -chgrp group1 /data/input/file.txt` changes the group ownership of “file.txt” to “group1”. 

#!/bin/bash

# Define the group and path variables
group="group1"
path="/data/input/file.txt"

# Change the group ownership using the hdfs dfs -chgrp command
hdfs dfs -chgrp $group $path

# Verify the changes by printing the file information
hdfs dfs -ls $path

MapReduce Job Execution Commands

Hadoop MapReduce facilitates the distributed processing of extensive datasets over a cluster. Let’s explore the essential Hadoop commands for executing MapReduce jobs.

`Hadoop jar <jar_file> <main_class> <input_path> <output_path>`

• The `Hadoop jar` command is a tool that helps you to manage who can access and control files in HDFS. By using it with specific <group> and <path>, you can ensure that the right people or groups have the proper permissions to work with the data securely and effectively.

For example, `hadoop jar myjob.jar com. example.MyJob /data/input /data/output` runs the MapReduce job defined in “myjob.jar” on the input data at “/data/input” and stores the output in “/data/output”.

#!/bin/bash

# Specify the group and path variables
group="mygroup"
inputPath="/data/input"
outputPath="/data/output"

# Change group ownership
hadoop fs -chgrp -R $group $inputPath

# Run MapReduce job
hadoop jar myjob.jar com.example.MyJob $inputPath $outputPath

Hadoop fs -text <output_path>`

• The `-text` option in Hadoop commands allows viewing contents of MapReduce job output files stored in HDFS. It’s used post-job completion to read and display files directly from the terminal. Users specify `<output_path>` to examine and analyze the outcomes of their MapReduce tasks effectively.

For example, `hadoop fs -text /data/output/part-00000` displays the contents of the “part-00000” file in the “/data/output” directory.

# Assuming you have Hadoop installed and configured properly

# View the contents of the "part-00000" file in the "/data/output" directory

hadoop fs -text /data/output/part-00000

`Hadoop job -kill <job_id>`

• This command allows administrators and users to forcefully stop a running MapReduce task. This helps free up resources and stop further processing. By providing the job ID, Hadoop accurately identifies and terminates the task, preventing additional processing. This command is crucial for maintaining control over the task execution process. 

For example, `Hadoop job -kill job_1234567890_0001` terminates the MapReduce job with the ID::

hadoop job -kill job_1234567890_0001

Hadoop Cluster Administration Commands

The successful management and administration of Hadoop commands depend on the utilization of distinctive commands. Let us see some crucial commands that are essential for the smooth functioning and control of a Hadoop cluster.

`Hadoop dfsadmin -report`

This command allows important details about a Hadoop system. It reveals which parts are actively working and which aren’t, shows how much storage is available, and gives stats on how the storage is being used. This information helps in keeping an eye on the system’s health, managing resources, and solving problems effectively.

`hadoop fsck <path>`

It’s a command used to check the condition of files and folders in the Hadoop Distributed File System (HDFS). It checks for problems in the data and information about the specified location, it highlights the problems like missing blocks or other problems that might affect data reliability. 

For example, `hadoop fsck /data/input` checks the health of files and directories in the “/data/input” path.

`Hadoop dfsadmin -safe mode enter/leave`

This command is like a safety net for your data. It puts your system in a safe mode during maintenance, preventing certain changes to files. This ensures that important tasks, like upgrades and repairs, can be done without accidentally messing up your data. It’s like a cautious traffic signal for your files. 

`Hadoop balancer`

It is like a traffic cop for data in a Hadoop system. It makes sure that information is spread out evenly across all the computers. By doing this, it helps the system run smoothly and efficiently. It’s like making sure everyone gets a fair share of the work. 

`yarn node -list`

This command is important for managing nodes in a YARN-managed Hadoop system. It provides essential information like IDs, states, and available resources. This data is vital for optimal performance, resource distribution, and keeping the system healthy. The Node Manager REST API is the key to accessing this valuable information.

Learn more about it through an online SQL course to get the knowledge and skills necessary when working with data in Hadoop.

Conclusion

In this blog, we’ve looked into essential Hadoop commands that play a crucial role in managing files, running MapReduce jobs, and keeping an eye on clusters. Whether you’re a data whiz, a curious explorer of big data, or somewhere in between, mastering these commands is like having a secret key to unlock Hadoop’s true potential for your projects. Now, it’s your turn, share your thoughts and feedback in the comments section below.