Setup Hadoop Cluster

Hadoop Cluster Setup in CentOS6

Requirement:

1.    Have java 1.6.x installed.

2.    Have ssh installed.

Installation & Configuratio[MUST be a root user]

1.    Download hadoop rpm file from apache hadoop official website.

2.    Install hadoop:

rpm –i hadoop_version.rpm

3.    Edit the file /etc/hosts on the servers:

192.168.1.40   master

192.168.1.41   slave1

192.168.1.42   slave2

4.    We must configure password less login from name node(master) to all data nodes (slave1 and slave2), on all servers do the following:

Ø  Command :ssh-keygen –tdsa

Ø  Keep press ENTER button until the id_dsa.pub file is generated.

We have 3 .pub files; one is on master, and others on the two slaves.

Copy the contents of those three .pub files to the authorized_keys file.

All servers authorized_keys file should have the same content.

5.    Open the file /etc/hadoop/hadoop-env.sh, and set the $JAVA_HOME: 

export JAVA_HOME=/usr/java/jdk1.6.0_38.

6.    Open the file /etc/hadoop/core-site.xml, add the following properties. This file is to configure the name node store information:

7.    Open the file /etc/hadoop/hdfs-site.xml and add the following properties:

8.    Open the file /etc/hadoop/mapred-site.xml, add the following properties. This file is to configure the host and port of the MapReduce jobtracker in the name node of the hadoop setup:

9.    Open the file /etc/hadoop/masters, add the namenode name: [NAMENODE SERVER ONLY]

master

10. Open the file /etc/hadoop/slaves, add all the datanodes names:[NAMENODE SERVER ONLY]

/* in case you want the namenode to also store data(i.e namenode also behave like a datanode) this can be mentioned in the salves file.*/

master

slave1

slave2

11. Modify files permissions.

Once Hadoop is installed, start-all.sh, stop-all.sh and several other files would be generated under /usr/sbin/, we must change all of those files permission:

           # sudo chmod a+x  file_name

Notice: Step 9 ,10 and 11 only for master server, the slaves should do nothing about those steps.

Start and Stop Hadoop Cluster (doing on hippo server)

1.    Formatting the namenode:

           # hadoop namenode –format

2.    Starting the Hadoop Cluster

           # start-all.sh

Run JPS command on master server:

          # jps 

            922 JobTracker

815 SecondaryNameNode

1062 TaskTracker

521 NameNode

1136 Jps

Run JPS command on slaves:

          # jps  

7407 DataNode

7521 TaskTracker

7583 Jps

3.    Checking the status of Hadoop Cluster:

(1)  Type the command :

# hadoop dfsadmin –report

            (2) Browse the web interface for the NameNode (master server) and the JobTracker:

·      NameNode – http://192.168.1.40:50070/

·      JobTracker – http://192.168.1.40:50030/

4.    Process a sample to test Hadoop Cluster (wordcount example):

(1)  Create a directory in master server

# mkdir input

          
(2) Create two test files under the ‘input’ directory and add the following text into the files

               echo "Hello haifzhan" >> text1.txt

               echo "Hello hadoop" >> text2.txt

               echo "Hello hadoop again" >> text3.txt

(3) Copy the two test files from master server to Hadoop’s HDFS

                  Under the ‘input’ directory:

                 # hadoop dfs -put ./   input

            (4)  Now you can check the files on Hadoop’s HDFS

                 # hadoop dfs -ls input/*

-rw-r--r--   2 root supergroup         15 2013-04-01 15:03 /user/root/input/text1.txt

-rw-r--r--   2 root supergroup         13 2013-04-01 15:03 /user/root/input/text2.txt

-rw-r--r--   2 root supergroup         19 2013-04-01 15:03 /user/root/input/text3.txt   

(5)  Run the MapReduce job

                 # hadoop jar /usr/share/hadoop/hadoop-example-1.0.3.jar wordcount input output

            (6)  Check the result

                 # hadoop dfs -cat output/part-r-00000

                        Hello  3

again  1

hadoop            2

haifzhan         1       

5.  Stopping the Hadoop Cluster

               # stop-all.sh

Other useful resources:

1. The logfiles locate in: /var/log/hadoop/root

2. Useful websites:

http://ankitasblogger.blogspot.ca/2011/01/hadoop-cluster-setup.html

http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-single-node-cluster/#running-a-mapreduce-job

Error Solving:

1. Datanode: No route to host (start but then shut down automatically for a while)

           close the firewalls on both master and slaves machines

           # service iptables stop

2. Namenode: How to exit the safemode

           # hadoop dfsadmin -safemode leave

3. How to start datanode or tasktracker independently

            # hadoop-daemon.sh start datanode/tasktracker

4. How to check the current java version and the path of your local machine

            # echo $JAVA_HOME

5.  proccess information unavailable

remove all files under /tmp , reformate namenode and restart all servers.

Perf4j for TPS(transactions per second)

How to use Perf4j with Slf4j ?


I used to use the following code for counting TPS

long start = System.currentTimeMillis();

long timeTook = System.currentTimeMillis() - start;
System.out.println("Time took(milliseconds):  " + timeTook);
System.out.println("Successful output #: " + counter);
System.out.println("Average TPS: " + counter * 1000 / timeTook);

Actually it is not a good way to record TPS.

Now I will introduce a new way to counting TPS, it is Perf4j, it can be used both for log4j and slf4j,
log4j is the way which the perf4 offical website introduced.Now I will introduce how to use Perf4j with slf4j.
Please take a glance at my github repo for perf4j.

It is very easy to use, convenient to use, hope it helps.

Install GuestAddition for Fedora 18/19/20

Install guest additions for Fedora

I encountered this problem:

Unable to mount the CD/DVD image /Applications/VirtualBox.app/Contents/MacOS/VBoxGuestAdditions.iso on the machine fedora18. Would you like to force mounting of this medium?

Could not mount the media/drive '/Applications/VirtualBox.app/Contents/MacOS/VBoxGuestAdditions.iso' (VERR_PDM_MEDIA_LOCKED).

How did I solve this problem?

Host: Mac 10.7.5
VM: Fedora 18


1.
change to the root user using "su"
2. 
yum install kernel* gcc -y
3.
cd /run/media/haifzhan/VBOXADDITIONS_xxx
4.
./VBoxLinuxAdditions.sh

Done!

Jenkins Installation w/ Git Support in CentOS 6

Jenkins Installation w/ Git Support in CentOS 6

Installation

First, we need to set up the Jenkins repository for CentOS.

# sudo wget –O /etc/yum.repos.d/jenkins.repo http://pkg.jenkins-ci.org/redhat/jenkins.repo

# sudo rpm - -import http://pkg.jenkins-ci.org/redhat/jenkins-ci.org.key

# sudo yum install –y jenkins

Make sure git is installed and check the path:

# which git

Output: /usr/bin/git

If no output, do a:

# sudo yum install git

Start/Stop Service

Start Jenkins service by issuing:

# sudo service jenkins start

Enable Jenkins to start at boot by:

# sudo chkconfig jenkins on

Logging in to Jenkins Web UI and Install Git and Github Plugins

Access jenkins web interface by pointing your browser to:

http://localmachine:8080

Where “localmachine” is the ip address or hostname of the machine you installed jenkins on.

Next, click on “Manage Jenkins” on the left hand column, and on the next menu, “Manage Plugins”.

Click the “Available” tab and check the boxes next to “Git Plugin” and “Github Plugin”, then click “Download now and install after restart” at the bottom of the page.

Once the plugins are installed, check the box that says “Restart Jenkins when installation is complete and no jobs are running”.  Jenkins will now restart and the plugins will be installed.

Configure Security

Click on “Manage Jenkins” and on the next menu, “Configure Global Security”.

Check “Enable security”, and under “Security Realm”, click the radio button next to “Jenkins’s own user database”.  Under “Authorization”, click on “Matrix-based security”.

Add a user named “administrator” (or whatever you like) and once the user appears, check *all* the boxes under *all* the categories.

Click “Save” in the bottom left, and you will be taken to a sign up page.  Enter “administrator” under the username field, and fill in the rest of the fields.  You will then be logged in as administrator and have full control over Jenkins.

Giving Jenkins user Access to Github

First, we need to temporarily give the Jenkins user shell access so we can generate SSH keys in order to access github.

# sudo vim /etc/passwd

Change the following line from:

jenkins:x:496:492:Jenkins Continuous Build server:/var/lib/jenkins:/bin/false

To:

jenkins:x:496:492:Jenkins Continuous Build server:/var/lib/jenkins:/bin/bash

Save and close the file.

Then switch to jenkins user:

#sudo su - jenkins

At the bash shell, issue:

# ssh-keygen

Hit enter for all the options.

Once the keys are generated and you’re back at the bash prompt, type “exit”.

Next, we have to rename the key files by doing the following:

# sudo cp /var/lib/jenkins/.ssh/id_rsa.pub /var/lib/jenkins/.ssh/authorized_keys

Now we need to copy the key.

# sudo cat /var/lib/jenkins/.ssh/authorized_keys

Copy the entire output and add to the SSH Keys section of your github user profile, giving it a name such as “Jenkins ssh key”.

Lastly, we have to connect to the github repo that we’re going to use in order to accept github into Jenkins’ known_hosts file.  Do this by issuing:

# sudo su – jenkins

# /usr/bin/git ls-remote -h git@github.com:starscriber/MessageBroker.git HEAD

When a message comes up asking you to continue, type “yes”.

If it is successful, you should go back to the bash prompt after a few seconds.  You can now type “exit”.

Now edit the /etc/passwd file again, and change the jenkins user’s shell back to /bin/false.

Adding Github repo to Jenkins Web UI

Click “Manage Jenkins” and then “Configure System”.

Click on “Git installations” and change “Path to Git executable” to full git path from step 1: /usr/bin/git

Make sure “Manually manage hook URLs” at the bottom of the page is checked.

Click on “Save”.

Next, select “New Job”.

Put something under “Job name” ie: MyCompany_MyApplication, and select “Build a free-style software project”, and click OK.

On the next screen, enter the “GitHub project” URL, ie: https://github.com/mycompany/MyApplication/

Under “Source Code Management”, select “Git” and enter the SSH repository URL, ie: git@github.com:mycompany/MyApplication.git

Under “Build Triggers”, check the box next to “Build when a change is pushed to GitHub”.

Save.

Test Everything

Go back to Jenkins main page, and you should see a list of your Jobs.  If you hover your mouse on the Job Name, ie: Starscriber MessageBroker, you should get a dropdown list.  Click on “Build Now” and if successful, everything is working properly. 

fix .gitignore does not work problem

http://www.randallkent.com/development/gitignore-not-working

All the following information is from Randall Kent's blog, it is really helpful, that is why I put it here:

"I’ve run a problem several times when .gitignore doesn’t appear to be working. The file I want to ignore is specified in .gitignore, but it always comes up as an unstaged change  . I always end up searching the internet for the resolution, because I can never seem to remember it. Well today, I’m making a blog post about it, so it will be easier for me to find and hopefully help others with the same problem!

What I have discovered is that at some point in time, I mistakenly added the files that I now want to ignore to my repository. For me, this is typically a .dll in a .NET project that’s recreated on every build anyway. Since git ‘knows’ about the file, it can’t ignore it. So the resolution is simple, remove everything from git’s index and add it back which can be done with the commands below…"

git rm -r --cached .

git add .

git commit -m "fixing .gitignore"

CentOS(installed in VirtualBox) connects to internet automatically

CentOS(installed in VirtualBox) connects to internet automatically

1. Settings(VirutalBox):
choose Network, attached to NAT, check Cable connected, and then press OK.

2. CentOS
There's a network icon on the top right CentOS screen, click it, then click VPN Connections.
You will find System eto0, after that click Edit, and then check Connect automatically.

now it should work.

Logging to file with slf4j & How to use RollingFileAppender

The example can be downloaded from:

1. How to Use slf4j logger.  [you can find maven dependency in pom.xml file]

2. How to use logback, [you can find it in logback.xml file]

3. How to use logback Rolling Policy. [ you can find in logback.xml file]


https://github.com/haifzhan/LoggingToFile.git