Raspberry Pi Cluster Node – 01 Logging Liveness

This post describes how to make a simple python script that logs the node is alive every 10 seconds.

Why we are going to log each node is alive

As discussed in the previous post on Distributed Computing on the Raspberry Pi Cluster there will be many slaves and a single master in the first stage.

The image reproduced opposite shows the process each slave will go through. They will repeatedly ask for work, then perform the action given to them.

If there is no work then they will idle for a period of time and then ask the master again for work.

This cycle will form the basis of running distributed work on the cluster. Any live node will be able to come online and request work. Once the work is done it will report back to the master with the result.

Then the loop will continue again requesting more work.

 What we are going to use

This first script will create the event loop to check if there is work to do and add logging to keep track of what the node is doing.

I am going to use the standard python logger to keep track of the work the node is doing. I will be configuring this to log data to both the console and a file. This will mean that I can view what the process is doing by observing the running console and ensure this data is saved to a file.

Another really useful feature that the python logger has is that it is inherently thread safe. For the moment we are only using a single thread so this won’t matter. However in the future I plan to move towards making this applicable multi-threaded and having logging that handles this will be very useful.

Setting up the python logger

The python logging module can be simply imported using import logging. Once we have imported this we have access to all the logging features needed for our Raspberry Pi Cluster.

To start with I am going to set up a formatter for the logs.

logFormatter = logging.Formatter(
    "%(asctime)s [%(threadName)-12.12s] [%(levelname)-5.5s]  %(message)s")

This formatter will be used to define the format of the logs produced. This includes all the important information including the time, thread, level and the message logged. As discussed above while the current program is not multi-threaded this will be in the future so it will be helpful to keep track of which thread is writing to the log.

Now that I have a formatter I can start to get the base logger object. This can be obtained using the below code.

logger = logging.getLogger()
logger.setLevel(logging.DEBUG)

Here I am getting the base logger and setting the level of messages that will be logged. By setting it to DEBUG I ensure that all messages will be logged. This base level set is inherited by all child loggers and output handlers.

To actually save print out or save the log I need to attach handlers to the log. The most basic type of handler is the console handler. This works by printing all the information out to the console. I create one using the following code.

consoleHandler = logging.StreamHandler()
consoleHandler.setFormatter(logFormatter)
consoleHandler.setLevel(logging.INFO)
logger.addHandler(consoleHandler)

Once the handler has been created with logging.StreamHandler() I attach the formatter I created earlier. For the console output I have also set the logging level to INFO. This means only messages of INFO and higher will be printed to the console. The final but most important line attaches the handler to the logger we created earlier. This will receive the messages sent to the logger and will print them to the console.

In addition to logging to the console I want to keep a store of all messages in case I need to review what has been occurring. I am able to do this with the following code by creating a file handler.

fileHandler = logging.FileHandler("mainlogger.log")
fileHandler.setFormatter(logFormatter)
logger.addHandler(fileHandler)

Here as I am creating a logger to save to a file I supply a filename mainlogger.log. This will be the file that is written to each time the logger runs. By default each time this code runs the file will have log data appended to it. This means if the script is stopped and rerun old log data won’t be lost. Again like above we use our custom log formatter to format our logs in a specific way.

Now we have created our logger and various handlers to display the data we are ready to use it.

Using the Python Logger

The logger object has a number of methods to allow us to log a message and data. The basic log function is

logger.log(level, message, args)

Here you log a specific message at a specific error level. This error level is an integer representing the severity of the message. The message along with any additional arguments will be used by your formatter to display and store the message. With log you have to specify a specific error level, the basic logging levels, as taken from the python site are:

This means that a logger set to a default level of DEBUG will log any messages set at level 10 and above. So that you don’t have to remember these values you can use the helper methods

logger.debug(message, args)
logger.info(message, args)
logger.warning(message, args)
logger.error(message, args)
logger.critical(message, args)

These work identically to logger.log() except they have preset the log level to the respective integer (based on the above table). For ease of use I will be using the specific logger functions defined above.

Creating the basis of a Raspberry Pi Cluster worker

Now I have the logger set up I can create a really basic Raspberry Pi Cluster worker. For this example it won’t perform any jobs but this structure will be used in the future.

I will define a function that will eventually be used to find work to perform. In this case I will just use my logger to log that there is no work to be done.

def find_work_to_do():
    logger.info("Node slave still alive")
    logger.info("Looking for work to run")
    return None

This function also returns None so that the calling function knows that there is nothing to do. The next and most crucial piece of code is the worker loop.

while True:
    work = find_work_to_do()
    if work is None:
        logger.info("Found no jobs to perform, going to sleep again")
        time.sleep(10)
    else:
        pass # we will do work in the future but not at the moment

Here we run forever checking to see if there is any work. If there is no work found, as determined by getting None back from the find_work_to_do function then it will log this and sleep for 10 seconds.

For the moment we will ignore the case that work has been found as this is just to set up a basic work loop.

Summary of work towards the Raspberry Pi Cluster

Today I have gone over the basics of the python logger and set up the initial code for our cluster project. In addition, I have created the template function that will be used to get work for our nodes. The next lessons will focus on creating a basic system to talk to a master node and send messages.

The full code for this first example is available on the Github Repository for this project.