Hadoop V1 Install - Pre-req Setup

In this blog we are going to have Pre-req  done for Hadoop V1 Cluster
 
The Pre-req are pretty much same for Hadoop V1 and Hadoop V2 we will see what changes in the later blogs.



Below is my /etc/hosts for HDP V1 Cluster. The names are pretty apparent talking about their roles and functionality. nn,snn are namenode and secondary namenode and d1n-d4n are datanodes. 

Our Controller Node is namenode or nn or namenode.cluster.com
We start by  putting below in the /etc/hosts file of namenode


127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
#::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.10.51 namenode.cluster.com      namenode nn
192.168.10.53 snamenode.cluster.com     snamenode snn
192.168.10.54 d1node.cluster.com        d1node      d1n
192.168.10.55 d2node.cluster.com        d2node      d2n
192.168.10.58 d3node.cluster.com        d3node      d3n
192.168.10.57 d4node.cluster.com        d4node    d4n




Step 1  - Download Hadoop V1 (1.2.1)

wget http://www-us.apache.org/dist/hadoop/common/hadoop-1.2.1/hadoop-1.2.1.tar.gz
--2018-02-20 06:22:53--  http://www-us.apache.org/dist/hadoop/common/hadoop-1.2.1/hadoop-1.2.1.tar.gz
Connecting to 172.20.24.50:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: 63851630 (61M) [application/x-gzip]
Saving to: ‘hadoop-1.2.1.tar.gz’
100%[==========================================================================>] 63,851,630   369KB/s   in 3m 6s

2018-02-20 06:25:59 (335 KB/s) - ‘hadoop-1.2.1.tar.gz’ saved [63851630/63851630]


Step 2
Setup ssh equivalency between root user of all nodes (You can delete this later if you want)
This will help us to set-up everything as root user.

[As root]
./sshUserSetup.sh  -user root  -hosts "nn snn d1n d2n d3n d4n" -noPromptPassphrase -confirm -advanced

Give the passwords once prompted - End of the script you should have passwordless connectivity between root users on all nodes.

The Script sshUserSetup is something you will be knowing if you are an OracleDBA
If not you can do this manually or provide your email below, i'll send it over.
All what script does is create ssh equivalency between all the nodes given.

Step 3
[As root - Managed Hosts file creation]
Create hosts file , each host in the file is in a line in each line
 
cd /tmp
chmod 777 hosts
cat hosts
nn
snn
d1n
d2n
d3n
d4n

Step 4
[As root - Hosts File Updation]
##for i in $(cat hosts) ; do scp /etc/hosts ${i}:/etc/hosts; done

Step 5
[As root - Group Creation]
## for i in $(cat hosts) ; do echo "ssh ${i} groupadd -g 1000 hadoop" >> /tmp/useradd; done ; bash /tmp/useradd ; rm -f /tmp/useradd

Step 6
[As root - User Creation]
## for i in $(cat hosts) ; do echo "ssh ${i} useradd -u 1002  -g hadoop hduser" >> /tmp/useradd; done ; bash /tmp/useradd ; rm -f /tmp/useradd 
for i in $(cat hosts) ; do echo "ssh ${i} useradd -u 1003  -g hadoop mapred" >> /tmp/useradd; done ; bash /tmp/useradd ; rm -f /tmp/useradd

Step 7
[As root - hduser and mapred user password change ]
Change hduser to mapred for second execution.
Create Script as below and run it

#!/bin/bash
for server in `cat hosts`; do
echo $server;
ssh ${server} 'passwd hduser <<EOF
hadoop
hadoop
EOF';
done

Step 8
[Hadoop User Equivalency] [As hduser and mapred -  on namenode]
This will create ssh equivalency for hduser and mapred
#./sshUserSetup.sh  -user hduser  -hosts "nn snn d1n d2n d3n d4n" -noPromptPassphrase -confirm -advanced
#./sshUserSetup.sh  -user mapred  -hosts "nn snn d1n d2n d3n d4n" -noPromptPassphrase -confirm -advanced
 

Step 9
[As root - Java Installation]
#for i in $(cat hosts) ; do echo "scp jdk-8u152-linux-x64.rpm ${i}:/tmp &" >> /tmp/sendjdk.bash ; done
Paste and run contents of the file

Step 10
[As root - Install Java] 
# for i in $(cat hosts) ; do  ssh ${i}  rpm -Uvh /tmp/jdk-8u152-linux-x64.rpm  ; done;

Step 11
[AS root - Verify Changes done till now]
Java
# for i in $(cat hosts) ; do ssh ${i}   java -version  ;  done;



[Other Automated Helper Scripts]
[Partition Creation]
# for i in $(cat hosts) ; do ssh ${i} 'echo -e "n\np\n1\n\n\nw" | fdisk /dev/sdc'  ; done;
# for i in $(cat hosts) ; do ssh ${i} 'echo -e "n\np\n1\n\n\nt\n8e\nw" | fdisk /dev/sdd'  ; done;

[For deleting the partitions if required]
# for i in $(cat hosts) ; do ssh ${i} 'echo -e "d\nw" | fdisk /dev/sdd'  ; done;

[File System formatting]
# for i in $(cat hosts) ; do ssh ${i} mkfs.ext4 /dev/sdc1   ; done;

[Directory Creation]
# for i in $(cat hosts) ; do ssh ${i} mkdir /opt/HDV1; done;

[Ownership Change]
# for i in $(cat hosts) ; do ssh ${i} chmod 777 /opt  ; done;
# for i in $(cat hosts) ; do ssh ${i} chown hduser:hadoop  /opt/hadoop  ; done;

[FSTAB Entry Addition]
# for i in $(cat hosts) ; do ssh ${i} echo "/dev/sdc1       /opt/HDPV1 ext4 defaults 1 2 >> /etc/fstab" ; done

[Mounting the file system]
#for i in $(cat hosts) ; do ssh ${i} mount /opt/HDPV1 ; done


[VG and FS Extension]
for i in $(cat hosts) ; do ssh ${i} pvcreate /dev/sdd1  ; done;
for i in $(cat hosts) ; do ssh ${i} vgextend rootvg /dev/sdd1 ; done;
for i in $(cat hosts) ; do ssh ${i} lvextend /dev/mapper/rootvg-root_lv -L 15G  ; done;
for i in $(cat hosts) ; do ssh ${i} resize2fs /dev/mapper/rootvg-root_lv  ; done;
for i in $(cat hosts) ; do ssh ${i} lvextend /dev/mapper/rootvg-tmp_lv -L 8G  ; done;
for i in $(cat hosts) ; do ssh ${i} resize2fs /dev/mapper/rootvg-tmp_lv  ; done;