Raspberry Pi Cluster Node – 02 Packaging common functionality

This post builds on the first step to create a Raspberry Pi Cluster node to package the common functionality that will be shared between master and clients.

Packaging common functionality

With the Raspberry Pi Cluster project there will be a number of things all nodes will do. This can lead to a large amount of duplicated code. To resolve the issue of copying the same code into both master and client code I will make a python package to hold the data.

Python packages are a way to collect common code together in easy to use modules. This allows multiple files to import the same code, reducing code duplication.

One of the major problems with copying code is that when improvements are made, some places that it is used can be missed. This means that a piece of code that was initially copied into a number of places may now differ. This shouldn’t be an issue if the improvement did not affect the overall functionality of the code. However if the change affected the behaviour you may now have multiple pieces of code doing slightly different things.

When programming you always try and avoid to have situations where this may occur and python packages provide a way to help resolve this problem.

As the amount of code we have increases I will refactor the code into different modules in the package. By keeping the code modular it will be easy to build much more complex node behaviour.

Creating a python package

To create our package we need to create a folder for it to live in. The package I will create will be called RpiCluster. Therefore I have created a folder in my project directory called RpiCluster.

To finish creating the package you need to add an __init__.pyfile to the folder. This file tells python that the folder is a python package. If this is not included python will not treat your directory as a package and will not allow importing files from it. This does not need to contain anything but we will use it at a later date to configure some details about the package.

The __init__.py file is safety feature so that folders which are named the same as python base modules do not hide these important modules. For example if your code has a stringfolder you do not want it overriding the python string module.

Now we have our package I can look at moving some of our code into it.

Moving our logging into the package

I am going to move the logging functionality into our newly created package. I am doing this because both the master and slave scripts will be logging their processes.

To start with I have copied all the logging code to a new file called MainLogger.py. One of the improvements I am going to make is to allow each script to customize the name of the file it logs to.

This has the improvement in that each of our scripts can configure a different logger name and location to store the file. For now, while we only have a single master script this doesn’t matter too much but it will be helpful in the future.

The first section of the logger script reproduced below was originally in the main script.

import logging
import time

# First lets set up the formatting of the logger
logFormatter = logging.Formatter("%(asctime)s [%(threadName)-12.12s] [%(levelname)-5.5s]  %(message)s")

# Get the base logger and set the default level
logger = logging.getLogger()
logger.setLevel(logging.DEBUG)

# Create a console handler to output the logging to the screen
consoleHandler = logging.StreamHandler()
consoleHandler.setFormatter(logFormatter)
consoleHandler.setLevel(logging.INFO)
logger.addHandler(consoleHandler)

Here again we set up the logger with our custom formatter and set up a console handler. I have initially removed the section that adds a file handler to log the files to.

Instead of setting up the file handler by default I have moved this into a method. This allows any script calling the method to set up its own file handler using the function below.

def add_file_logger(filename):
    # Create a handler to store the logs to a file
    fileHandler = logging.FileHandler(filename)
    fileHandler.setFormatter(logFormatter)
    logger.addHandler(fileHandler)

Here I allow the script to pass in their filename which will be used to create a file handler. Now I have the basics of my logging code set up in my new RpiCluster package.

Using our RpiCluster package in our main script

The only modifications made so far to the main script has been to remove all the logging setup. Now I can import the new package using the following import statement

from RpiCluster.MainLogger import add_file_logger, logger

The first part of the import statement tells python where the symbols we are going to import exist. Here we use the dot format to say that we are looking for symbols inside the RpiCluster package and inside the MainLogger.py file.

The import statement following that tells it to import our function to add the handler and the logger variable created. When the MainLogger file is loaded the code runs creating the logger object. Here we make that variable available in our main script.

Now I have imported the new package I can use the logger as before. The rest of the code in the script has not changed.

The full code is available on Github, any comments or questions can be raised there as issues or posted below.