Raspberry Pi Cluster Node – 15 A more complex webserver

This tutorial focuses on improving the webserver to display information about the slaves connected to the master using python Bottle.

Refactoring the Master Script

To start with the changes, I am going to focus on the master script. This is going to move the changes into a new class.

class RpiMaster:

     def __init__(self, socket_bind_ip, socket_port):
        self.socket_bind_ip = socket_bind_ip
        self.socket_port = socket_port
        self.connected_clients = {}

This class will hold some information about the connected clients, in addition to the bind IP address and port number.

The main method in this class will be start() which will be called to begin setting up the sockets and allow slaves to connect.

 def start(self):
        logger.info("Starting script...")

        listening_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        listening_socket.bind((self.socket_bind_ip, self.socket_port))

        listening_socket.listen(10)  # listen to 10 connects
        while True:
            (clientsocket, address) = listening_socket.accept()
            logger.info("Got client at {address}".format(address=address))

            rpi_client = RpiClusterClient(self, clientsocket, address)
            self.connected_clients[rpi_client.uuid] = rpi_client
            rpi_client.start()

This takes much of the code from the old basic_master file and wraps it in the class. Some changes here are pulling a UUID created from the RpiClusterClient and storing this in an dictionary for easy access.

By combing this data into an object some helper methods can be created to help access the data and remove it when a client disconnects.

 def remove_client(self, rpi_client):
        del self.connected_clients[rpi_client.uuid]

def get_slave_details(self):
    slave_details = {}
    for uuid in self.connected_clients:
        slave_details[uuid] = {
            "uuid": uuid,
            "address": str(self.connected_clients[uuid].address[0]) + ":" + str(self.connected_clients[uuid].address[1]),

         }

     return slave_details

Here the two helper functions are added to this class. The first remove_client removes the client from the connected_clients dictionary. This is called once the client has disconnected.

The second is used to get some basic information about all currently connected clients. This will be used by the slaves to request information about the other connected clients.

Changing the RpiClusterClient to assign UUIDs

Instead of using a random int that may be liable to collisions the RpiClusterClient is going to be changed to assign UUID’s to each connecting slave.

This changes the constructor to the following code.

    def __init__(self, master, clientsocket, address):
        threading.Thread.__init__(self)
        self.uuid = uuid.uuid4().hex
        self.master = master
        self.clientsocket = clientsocket
        self.address = address
        self.node_specifications = None

In addition to randomly creating a uuid, this accepts the RpiMaster object so it can get information from the master.

The node_specifications variable is set to None and will be filled in once the slave connects. This then changes the code that receives the computer details to instead of printing it out, also save it.

elif message['type'] == 'computer_details':
    self.node_specifications = message['payload']
    logger.info("Received Computer specifications: " + json.dumps(self.node_specifications))

In the body of the message parsing code I have also added two more pieces of information the slave can request.

elif message['payload'] == 'uuid':
    send_message(self.clientsocket, create_payload(self.uuid, "uuid"))
elif message['payload'] == 'slave_details':
    slave_details = self.master.get_slave_details()
    send_message(self.clientsocket, create_payload(slave_details, "slave_details"))

Here the slave can relieve its currently assigned UUID and the details of all slaves connected to the master. The slave will be changed so that once it has connected it will request the UUID to save.

The slave details call will be used by the webserver to receive all information about the slaves connected.

The final change to the RpiClusterClient is to remove itself from the master once it disconnects. This uses the newly created remove_client method.

except DisconnectionException as e:
    logger.info("Got disconnection exception with message: " + e.message)
    logger.info("Shutting down slave connection handler")
    self.master.remove_client(self)

By doing this the list of connected slaves will always be up to date.

Changes to the RpiBasicSlaveThread

The only change needed to the RpiBasicSlaveThread class is to change its random ID to the UUID the master assigns.

The constructor changes from creating the client_number to instead creating a placeholder for the UUID. This will later be requested from the master in the perform_action method.

This makes the start of the perform action method the following.

logger.info("Sending an initial hello to master")
send_message(self.sock, create_payload("uuid", "info"))
message = get_message(self.sock)
self.uuid = message['payload']
logger.info("My assigned UUID is " + self.uuid)

Once connected, the first call of the basic slave will be to request the UUID. This is then stored in the slave thread.

Changing the RpiWebserverSlaveThread to retrieve slave details

Now we are able to request slave details from the master we need to change the RpiWebserverSlaveThread to request these details. This changes the class to hold the following information.

class RpiWebserverSlaveThread(RpiBasicSlaveThread):

    current_master_details = None
    current_slave_details = None
    webserver_data_updated = None

    def perform_action(self):
        logger.info("Now sending a keepalive to the master")
        send_message(self.sock, create_payload("I am still alive, client: {num}".format(num=self.uuid)))
        send_message(self.sock, create_payload("computer_details", "info"))
        master_details = get_message(self.sock)
        send_message(self.sock, create_payload("slave_details", "info"))
        slave_details = get_message(self.sock)

        RpiWebserverSlaveThread.current_master_details = master_details['payload']
        RpiWebserverSlaveThread.current_slave_details = slave_details['payload']
        RpiWebserverSlaveThread.webserver_data_updated = datetime.datetime.now()
        time.sleep(5)

This will request both the master and slave details every 5 seconds and store it as a class variable. This information is then accessible in the Bottle webserver and can be served by the webserver.

Changing the Bottle Webserver to serve slave details

The final changes needed are to change the Bottle webserver so that it serves the slave details. This involves changing the route as below.

@route('/')
def index():
    return template("templates/ClusterHomepage.html",
                    masterinfo=json.dumps(RpiWebserverSlaveThread.current_master_details, indent=4, sort_keys=True),
                    slaveinfo=json.dumps(RpiWebserverSlaveThread.current_slave_details, indent=4, sort_keys=True)
                    )

Once the route has changed the final tweak that needs to be made is to add this information to the html file.

<h2>Master Information</h2>
<pre>{{masterinfo}}</pre>

<h2>Slave Information</h2>
<pre>{{slaveinfo}}</pre>

Summary of improving the Webserver information

In this tutorial the cluster is improved so that each slave is assigned a unique UUID. In addition the python Bottle webserver now displays information about the slaves.

In the future this will be used to view the cluster as it processes tasks.

Now the basics of the cluster is in place we can look at starting to define jobs that each node can run.

The full code is available on Github, any comments or questions can be raised there as issues or posted below.