Hadoop on Pi – The Network

8Mar - by Simon - 0 - In Projects

To build a small Pi-based Hadoop cluster, a simple network is needed. As I am using Pi3 units then I can use WiFi, but I also wanted the flexibility to switch between hard-wired and wireless networking in the Hadoop Pi network without too much hassle. 

WiFi

The wireless design is simple – WiFi was setup when installing Raspbian, with the wpa-supplicant configuration of the Pi3 set to lock onto my local access point, which provided internet access (see here for guidance on how to set up WiFi on a Pi). My AP was configured to serve DHCP addresses up to .100 – leaving the remainder of the range free for fixed addresses to be manually assigned to devices. This I did with a file created in /etc/network/interfaces.d/ directory of each Pi. This contained the following wlan0 entry; noting that the default interfaces file was modified to remove its ‘standard’ wlan0 entry and to look in the interfaces.d directory for more configuration.

/etc/network/interfaces becomes very simple, just including the interfaces.d directory:

# interfaces(5) file used by ifup(8) and ifdown(8)

# Include files from /etc/network/interfaces.d:
source-directory /etc/network/interfaces.d

and in /etc/network.interfaces.d/ there is a file called fixed-cluster containing:

# interfaces file - can be used by ifup and ifdown commands
# Copy into /interfaces.d/ directory to be picked up at boot

auto lo
     iface lo inet loopback

iface eth0 inet dhcp

auto wlan0
     allow-hotplug wlan0

iface wlan0 inet static
     address 192.168.15.130
     netmask 255.255.255.0
     gateway 192.168.15.1
     wpa-conf /etc/wpa_supplicant/wpa_supplicant.conf

Each fixed-cluster file is modified to give the correct address for each particular Pi as in the following simple design:

So, each of the 4 Pi devices is set up with static addresses .130 through .133. The hosts file is modified to include all the servers as follows:

192.168.15.130           data-master
192.168.15.131           data-slave01
192.168.15.132           data-slave02
192.168.15.133           data-slave03

Simple stuff. That just left each hostname file to be modified on each Pi to reflect its name in the cluster. Note that you can also disable WiFi power saving – the interface will sleep with no traffic; whilst it ‘wakes’ on activity, this can introduce up to a 40 second delay in response. Whilst I have not experienced any timeout issues on the final Hadoop cluster, it is probably worth doing this. Simple command to be entered from the command line on each of the Pi devices in the cluster:

sudo iw dev wlan0 set power_save off

Quick Checklist:

  1. Modify /etc/hostname
  2. Modify /etc/hosts to include all hosts in the cluster with their static addresses
  3. Modify /etc/network/interfaces to refer to the interfaces.d directory
  4. Create a file in /etc/network/interfaces.d/ directory with the local static address assigned to the wlan0 interface (see the fixed-cluster example above)
  5. Optionally disable WiFi power saving on the WLAN device (improves response times by preventing the WiFi interface from sleeping)
  6. Reboot to activate the Hadoop Pi network and make the new hostnames and addresses accessible

 

Fixed network

Much the same approach but with the wlan0 interface set to use DHCP (or disabled if preferred) and the eth0 interface now set with the same fixed address.

This can be done using a file saved in your home directory which can then be copied into /etc/network/interfaces.d/ directory (overwriting the one there). A reboot then switches the Pi3 over to a static address on the eth0 fixed interface.

No change to /etc/network/interfaces – it remains:

# interfaces(5) file used by ifup(8) and ifdown(8)

# Include files from /etc/network/interfaces.d:
source-directory /etc/network/interfaces.d

and in /etc/network.interfaces.d/ there is a file called fixed-cluster now containing:

# interfaces file - can be used by ifup and ifdown commands
# Copy into /interfaces.d/ directory to be picked up at boot

auto lo
     iface lo inet loopback

iface eth0 inet static
     address 192.168.15.130
     netmask 255.255.255.0
     gateway 192.168.15.1

auto wlan0
     allow-hotplug wlan0

iface wlan0 inet dhcp
     wpa-conf /etc/wpa_supplicant/wpa_supplicant.conf

To wire-up this fixed network, I found a neat TPLink Ethernet switch on Amazon which uses a 5V power supply and hence can plug into a spare slot on any USB power supply. This completes the compact ethernet arrangement of the 4 Pi servers in the cluster (see related articles on the hardware build for more information and pictures).

Don’t forget that, to retain internet access, your switch/hub needs to be cabled back to your internet router too.

Quick checklist:

  1. Cable-up the cluster, ideally via an Ethernet switch, to your internet access point / router
  2. Modify /etc/hostname (if not already done form the WiFi setup)
  3. Modify /etc/hosts to include all hosts in the cluster with their static addresses (again, not needed if you did this for the WiFi version)
  4. Modify /etc/network/interfaces to refer to the interfaces.d directory (and again, yes, should already be done if you built the WiFi cluster on Pi3 devices above).
  5. Create a file in /etc/network/interfaces.d/ directory with the local static address assigned to the eth0 interface (see the fixed-cluster example above). Don’t forget to modify the static address for each device to be unique and matching its entry in the hosts

[or – if you already have a file for the wlan0 solution – then create another one in your home directory for eth0. You can then overwrite the file in interfaces.d with either the wlan0 or eth0 version as you wish – enabling you to switch the network between Ethernet and WiFi as needed – just reboot after copying (or use ifdown and ifup commands – you can even build a script which does this – switching between Ethernet and wifi addresses as required)].

Leave a Reply

Your email address will not be published. Required fields are marked *