Raspberry Pi Cluster Node – 06 Sending Slave Details to the Master

This post builds on my previous posts in the Raspberry Pi Cluster series by changing how the slave sends messages. From now on the slave will send useful information about the system to the master. The master will also be configured to receive messages of many types.

Changing the message format

To start with I have coded it so that sending JSON messages between the nodes would print out the message. Now I am going to change it so that it can send different types of messages.

To do this I am first going to change the format I use to send messages. To do this I am going to modify create_msg_payload function to be more useful.

def create_payload(payload, payload_type="message"):
    return json.dumps({"type": payload_type, "payload": payload}) + MESSAGE_SEPARATOR

Here I have renamed the function and added a new parameter to the function, payload_type. This is currently set to default to the string message.

The format I am using to send data has been changed to a more generic one with two keys, type and payload. This allows me to send a simple string or a more complex object if required. The master will handle each payload differently based on the string in the type field.

Now I have a flexible system able to send any data and payload I require. I will talk about how the master can then handle each different type later.

Getting useful system information

To make the slave a little more interesting I am going to add a new type of payload, the machine_details payload. This is going to include a number of details about the slave to pass to the master.

In the future I am going to use this information to judge what slaves should perform what tasks. This is important as certain tasks may require specific machine requirements, such as a minimum amount of system RAM.

I am going to send the hostname, CPU percentage used, RAM, CPU type, and the number of CPU cores. To get this data I am going to use a number of different python libraries in the slave script. Below I import the required python libraries to send the data I want to the master.

import socket
import psutil
import platform
import multiprocessing

Now I have these libraries I can create my object to send to the master.

machine_details = {
    'hostname': socket.gethostname(),
    'cpu_percent_used': psutil.cpu_percent(1),
    'ram': psutil.virtual_memory().total,
    'cpu': platform.processor(),
    'cpu_cores': multiprocessing.cpu_count()
}

Above I have created the machine_details object that I will send across. I will detail what I am using for each section and what it returns below.

• hostname – The socket library provides a very helpful method called gethostname. This returns the current hostname of the machine. This will be used later to assign a name to each slave.
• cpu_percent_used – The psutil module is a cross-platform module that deals with getting machine specific information. Here I am using the cpu_percent function telling it to wait 1 second averaging the CPU usage over that time period. This can be used to give a rough guide on how loaded the system is. This will prove useful when I am looking to balance the CPU load across a number of nodes.
• ram – psutil is again used to return the amount of RAM the system has available.
• cpu – Here I am using the platform library to return a string representing the type of CPU the machine has. This is a platform independent module but does not return any specific format. This is important to note as it depends what information the operating system can obtain about the CPU. Becuase of this reason I am going to use this as a display value only.
• cpu_cores – The multiprocessing library allows us to easily get the number of CPU’s the system has. Again this will be useful in the future to give a rough estimate of how parallel each machine can run tasks.

Now I have my object I can then send it to the master using the create_payload function as below.

sock.send(create_payload(machine_details, 'machine_details'))

I use the key machine_details as the type so the master can handle this differently. Now I am going to change how the master handles the messages.

Handling different message types on the master

Now I am sending different types of messages I need to handle this on the master in some way. Since the type field determines the format of the payload we just need to handle it based on this.

message = get_message(clientsocket)
if message:
    if message['type'] == 'message':
        logger.info("Received message: " + message['payload'])
    elif message['type'] == 'machine_details':
        logger.info("Machine specifications: " + json.dumps(message['payload']))
    else:
        logger.warning("Unknown payload type {type}, payload content {content}".format(
            type=message['type'], content=json.dumps(message['payload'])
        ))

In the above example I am using a simple if statement chain to handle the different types of payloads. I am using the same code as before to handle any messages received. If there is a type that I have not handled in the code, it will print out an error declaring it doesn’t understand the message.

If I receive the message type machine_details I am json encoding and printing out the data received. In the future I will store this machine data and use it to allocate tasks.

As further message types are added I will add more if statements here. In the future I will add a more generic solution for handling message types.

Summary

Now I have modified the original data sending functions so it now uses a more generic format. This allows the master to read what messages are being received and determine what action it needs to take.

I have a versatile system for receiving messages from a node and processing them so I will look into creating a two-way dialogue between the master and slave in the future.

The full code is available on Github, any comments or qduestions can be raised there as issues or posted below.