عکس arminZolfaghari
Simply up Hadoop-filesystem + yarn using vagrant, then run Mapreduce Code on Hadoop.Python
موضوع‌ها
۰
فورک‌ها
۰
ستاره‌ها
۴
تاریخ ایجاد
۱۵ خرداد ۱۴۰۱
آخرین بروزرسانی
۲ ماه قبل

Introduction

vagrant is a useful tool for managing vm's

this project is a simple hadoop setup with vagrant

Install

At first install vagrant

Then install Virtualbox

sudo apt install virtualbox

How to run

just run this command's

sudo mkdir -p /etc/vbox/
echo "* 10.0.0.0/8 192.168.0.0/16"|sudo tee /etc/vbox/networks.conf
vagrant up

Note:

for Apple Silicon run this commands

docker_branch use docker as a provider instead of virtualbox

git checkout docker_provider
vagrant up

WebUI

Note:

if you are using docker your webUI address will be changed because in docker version project use port_forwarding

Some Useful Vagrant commands:

  • delete vm's with their files

    vagrant destroy
    
  • shutdown vm's

    vagrant halt
    
  • ssh to a specific vm

    vagrant ssh <YOUR-VM-NAME>
    

    for example:

      vagrant ssh hadoop-master
      vagrant ssh hadoop-worker1
    

    In windows before vagrant ssh, run this command in terminal:

      set VAGRANT_PREFER_SYSTEM_BIN=0
    

Some Useful Hadoop commands (in master or worker vm's):

  • ls in hdfs
    hdfs dfs -ls /your/path/to
    
  • put file from your directory to hdfs
    hdfs dfs -put  /path/to/your-file /path/to/hdfs
    

    Optional

    run this command to set hosts ```bash sudo sed -i '/hadoop/d' /etc/hosts echo "

10.20.30.11 hadoop-master 10.20.30.21 hadoop-worker1 10.20.30.22 hadoop-worker2 10.20.30.23 hadoop-worker3 " | sudo tee -a /etc/hosts


# References
+ [hadoop](https://hadoop.apache.org/docs/stable/)
+ [hadoop-core-configs](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/core-default.xml)
+ [hadoop-hdfs-configs](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml)
+ [hadoop-yarn-configs](https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-common/yarn-default.xml)