{"id":881,"date":"2017-11-29T13:00:32","date_gmt":"2017-11-29T13:00:32","guid":{"rendered":"http:\/\/chewett.co.uk\/blog\/?p=881"},"modified":"2020-09-26T22:05:39","modified_gmt":"2020-09-26T21:05:39","slug":"raspberry-pi-cluster-node-02-packaging-common-functionality","status":"publish","type":"post","link":"https:\/\/chewett.co.uk\/blog\/881\/raspberry-pi-cluster-node-02-packaging-common-functionality\/","title":{"rendered":"Raspberry Pi Cluster Node \u2013 02 Packaging common functionality"},"content":{"rendered":"\n

This post builds on the first step to create a Raspberry Pi Cluster node<\/a> to package the common functionality that will be shared between master and clients.<\/p>\n\n\n\n\n\n\n\n

Packaging common functionality<\/h2>\n\n\n\n

With the Raspberry Pi Cluster project there will be a number of things all nodes will do. This can lead to a large amount of duplicated code. To resolve the issue of copying the same code into both master and client code I will make a python package to hold the data.<\/p>\n\n\n\n

Python packages are a way to collect common code together in easy to use modules. This allows multiple files to import the same code, reducing code duplication.<\/p>\n\n\n\n

One of the major problems with copying code is that when improvements are made, some places that it is used can be missed. This means that a piece of code that was initially copied into a number of places may now differ. This shouldn’t be an issue if the improvement did not affect the overall functionality of the code. However if the change affected the behaviour you may now have multiple pieces of code doing slightly different things.<\/p>\n\n\n\n

When programming you always try and avoid to have situations where this may occur and python packages provide a way to help resolve this problem.<\/p>\n\n\n\n

As the amount of code we have increases I will refactor the code into different modules in the package. By keeping the code modular it will be easy to build much more complex node behaviour.<\/p>\n\n\n\n

Creating a python package<\/h2>\n\n\n\n

To create our package we need to create a folder for it to live in. The package I will create will be called RpiCluster<\/code>. Therefore I have created a folder in my project directory called RpiCluster<\/code>.<\/p>\n\n\n\n

To finish creating the package you need to add an __init__.py<\/code>file to the folder. This file tells python that the folder is a python package. If this is not included python will not treat your directory as a package and will not allow importing files from it. This does not need to contain anything but we will use it at a later date to configure some details about the package.<\/p>\n\n\n\n

The __init__.py<\/code> file is safety feature so that folders which are named the same as python base modules do not hide these important modules. For example if your code has a string<\/code>folder you do not want it overriding the python string<\/code> module.<\/p>\n\n\n\n

Now we have our package I can look at moving some of our code into it.<\/p>\n\n\n\n

Moving our logging into the package<\/h2>\n\n\n\n

I am going to move the logging functionality into our newly created package. I am doing this because both the master and slave scripts will be logging their processes.<\/p>\n\n\n\n

To start with I have copied all the logging code to a new file called MainLogger.py<\/code>. One of the improvements I am going to make is to allow each script to customize the name of the file it logs to.<\/p>\n\n\n\n

This has the improvement in that each of our scripts can configure a different logger name and location to store the file. For now, while we only have a single master script this doesn’t matter too much but it will be helpful in the future.<\/p>\n\n\n\n

The first section of the logger script reproduced below was originally in the main script.<\/p>\n\n\n\n

import logging\nimport time\n\n# First lets set up the formatting of the logger\nlogFormatter = logging.Formatter(\"%(asctime)s [%(threadName)-12.12s] [%(levelname)-5.5s]  %(message)s\")\n\n# Get the base logger and set the default level\nlogger = logging.getLogger()\nlogger.setLevel(logging.DEBUG)\n\n# Create a console handler to output the logging to the screen\nconsoleHandler = logging.StreamHandler()\nconsoleHandler.setFormatter(logFormatter)\nconsoleHandler.setLevel(logging.INFO)\nlogger.addHandler(consoleHandler)<\/pre><\/pre>\n\n\n\n
Here again we set up the logger with our custom formatter and set up a console handler. I have initially removed the section that adds a file handler to log the files to.<\/p>\n\n\n\n
Instead of setting up the file handler by default I have moved this into a method. This allows any script calling the method to set up its own file handler using the function below.<\/p>\n\n\n\n
def add_file_logger(filename):\n    # Create a handler to store the logs to a file\n    fileHandler = logging.FileHandler(filename)\n    fileHandler.setFormatter(logFormatter)\n    logger.addHandler(fileHandler)<\/pre><\/pre>\n\n\n\n
Here I allow the script to pass in their filename which will be used to create a file handler. Now I have the basics of my logging code set up in my new RpiCluster<\/code> package.<\/p>\n\n\n\n
Using our RpiCluster package in our main script<\/h2>\n\n\n\n
The only modifications made so far to the main script has been to remove all the logging setup. Now I can import the new package using the following import statement<\/p>\n\n\n\n
from RpiCluster.MainLogger import add_file_logger, logger<\/pre><\/pre>\n\n\n\n
The first part of the import statement tells python where the symbols we are going to import exist. Here we use the dot format to say that we are looking for symbols inside the RpiCluster<\/code> package and inside the MainLogger.py<\/code> file.<\/p>\n\n\n\n
The import statement following that tells it to import our function to add the handler and the logger variable created. When the MainLogger file is loaded the code runs creating the logger object. Here we make that variable available in our main script.<\/p>\n\n\n\n
Now I have imported the new package I can use the logger as before. The rest of the code in the script has not changed.<\/p>\n\n\n\n
The full code is available on Github<\/a>, any comments or questions can be raised there as issues or posted below.<\/p>\n","protected":false},"excerpt":{"rendered":"
This post builds on the first step to create a Raspberry Pi Cluster node to package the common functionality that will be shared between master and clients.<\/p>\n","protected":false},"author":1,"featured_media":886,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_exactmetrics_skip_tracking":false,"_exactmetrics_sitenote_active":false,"_exactmetrics_sitenote_note":"","_exactmetrics_sitenote_category":0,"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[122],"tags":[39,184,196,37,101],"class_list":["post-881","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-raspberry-pi-cluster","tag-linux","tag-python","tag-python-packaging","tag-raspberry-pi","tag-raspberry-pi-cluster"],"wppr_data":{"cwp_meta_box_check":"No"},"jetpack_publicize_connections":[],"jetpack_featured_media_url":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2017\/11\/rpi_cluster_02_packaging.jpg?fit=800%2C800&ssl=1","jetpack_shortlink":"https:\/\/wp.me\/p2toWX-ed","jetpack_sharing_enabled":true,"jetpack-related-posts":[{"id":2680,"url":"https:\/\/chewett.co.uk\/blog\/2680\/raspberry-pi-cluster-node-16-python-3-codebase-refactor\/","url_meta":{"origin":881,"position":0},"title":"Raspberry Pi Cluster Node \u2013 16 Python 3 Codebase Refactor","author":"Chewett","date":"October 24, 2020","format":false,"excerpt":"This post builds on\u00a0my previous posts in the Raspberry Pi Cluster series\u00a0by improving the codebase for Python 3. Moving to Python 3 Python 2 was marked end of life on January 1st, 2020 and therefore applications should ideally be no longer using Python 2. There will still be a lot\u2026","rel":"","context":"In "Raspberry Pi Cluster"","block_context":{"text":"Raspberry Pi Cluster","link":"https:\/\/chewett.co.uk\/blog\/category\/raspberry-pi-cluster\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2020\/10\/raspi_cluster_16_python3refactor_posticon_OUTPUT.png?fit=1200%2C628&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2020\/10\/raspi_cluster_16_python3refactor_posticon_OUTPUT.png?fit=1200%2C628&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2020\/10\/raspi_cluster_16_python3refactor_posticon_OUTPUT.png?fit=1200%2C628&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2020\/10\/raspi_cluster_16_python3refactor_posticon_OUTPUT.png?fit=1200%2C628&ssl=1&resize=700%2C400 2x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2020\/10\/raspi_cluster_16_python3refactor_posticon_OUTPUT.png?fit=1200%2C628&ssl=1&resize=1050%2C600 3x"},"classes":[]},{"id":1054,"url":"https:\/\/chewett.co.uk\/blog\/1054\/raspberry-pi-tutorial-website-online-now\/","url_meta":{"origin":881,"position":1},"title":"Raspberry Pi Tutorial Website online now!","author":"Chewett","date":"March 10, 2018","format":false,"excerpt":"This post talks about the new page on my website with linking to all my Raspberry Pi Cluster tutorials. Raspberry Pi Cluster Tutorial Webpage Now I have a couple Raspberry Pi Cluster tutorials I decided to link to all of them on my website. This will form the basis of\u2026","rel":"","context":"In "Raspberry Pi Cluster"","block_context":{"text":"Raspberry Pi Cluster","link":"https:\/\/chewett.co.uk\/blog\/category\/raspberry-pi-cluster\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2018\/03\/rpi_tutorial_website_online.jpg?fit=800%2C800&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2018\/03\/rpi_tutorial_website_online.jpg?fit=800%2C800&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2018\/03\/rpi_tutorial_website_online.jpg?fit=800%2C800&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2018\/03\/rpi_tutorial_website_online.jpg?fit=800%2C800&ssl=1&resize=700%2C400 2x"},"classes":[]},{"id":2127,"url":"https:\/\/chewett.co.uk\/blog\/2127\/raspberry-pi-cluster-node-14-a-simple-webserver\/","url_meta":{"origin":881,"position":2},"title":"Raspberry Pi Cluster Node \u2013 14 A simple webserver","author":"Chewett","date":"April 17, 2019","format":false,"excerpt":"This tutorial focuses on creating a simple webserver that displays the status of the master using python Bottle. What will the webserver be used for? To interact with the cluster I am planning on making a small set of webpages. Initially these will just display information about the cluster but\u2026","rel":"","context":"In "Raspberry Pi Cluster"","block_context":{"text":"Raspberry Pi Cluster","link":"https:\/\/chewett.co.uk\/blog\/category\/raspberry-pi-cluster\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2019\/04\/rpi_cluster_14_a_simple_webserver.jpg?fit=800%2C800&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2019\/04\/rpi_cluster_14_a_simple_webserver.jpg?fit=800%2C800&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2019\/04\/rpi_cluster_14_a_simple_webserver.jpg?fit=800%2C800&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2019\/04\/rpi_cluster_14_a_simple_webserver.jpg?fit=800%2C800&ssl=1&resize=700%2C400 2x"},"classes":[]},{"id":1913,"url":"https:\/\/chewett.co.uk\/blog\/1913\/raspberry-pi-cluster-node-10-more-advanced-connection-handling\/","url_meta":{"origin":881,"position":3},"title":"Raspberry Pi Cluster Node \u2013 10 More Advanced Connection Handling","author":"Chewett","date":"January 16, 2019","format":false,"excerpt":"This post builds on\u00a0my previous posts in the Raspberry Pi Cluster series\u00a0by improving the connection code so it wont crash when the master or slave disconnects. Using Exceptions to handle socket issues The improvements to the cluster code are going to add an exception to the communication code. This will\u2026","rel":"","context":"In "Raspberry Pi Cluster"","block_context":{"text":"Raspberry Pi Cluster","link":"https:\/\/chewett.co.uk\/blog\/category\/raspberry-pi-cluster\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2019\/01\/rpi_cluster_10_advanced_connection_handling.jpg?fit=800%2C800&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2019\/01\/rpi_cluster_10_advanced_connection_handling.jpg?fit=800%2C800&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2019\/01\/rpi_cluster_10_advanced_connection_handling.jpg?fit=800%2C800&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2019\/01\/rpi_cluster_10_advanced_connection_handling.jpg?fit=800%2C800&ssl=1&resize=700%2C400 2x"},"classes":[]},{"id":741,"url":"https:\/\/chewett.co.uk\/blog\/741\/raspberry-pi-cluster-node-01-logging-liveness\/","url_meta":{"origin":881,"position":4},"title":"Raspberry Pi Cluster Node – 01 Logging Liveness","author":"Chewett","date":"November 1, 2017","format":false,"excerpt":"This post describes how to make a simple python script that logs the node is alive every 10 seconds. Why we are going to log each node is alive As discussed in the previous post on Distributed Computing on the Raspberry Pi Cluster there will be many slaves and a\u2026","rel":"","context":"In "Raspberry Pi Cluster"","block_context":{"text":"Raspberry Pi Cluster","link":"https:\/\/chewett.co.uk\/blog\/category\/raspberry-pi-cluster\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2017\/10\/rpi_cluster_01_logging_liveness.jpg?fit=800%2C800&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2017\/10\/rpi_cluster_01_logging_liveness.jpg?fit=800%2C800&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2017\/10\/rpi_cluster_01_logging_liveness.jpg?fit=800%2C800&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2017\/10\/rpi_cluster_01_logging_liveness.jpg?fit=800%2C800&ssl=1&resize=700%2C400 2x"},"classes":[]},{"id":1001,"url":"https:\/\/chewett.co.uk\/blog\/1001\/raspberry-pi-cluster-node-04-configuration-files-configparser\/","url_meta":{"origin":881,"position":5},"title":"Raspberry Pi Cluster Node \u2013 04 Configuration Files with ConfigParser","author":"Chewett","date":"February 10, 2018","format":false,"excerpt":"This post builds on the\u00a0third step to create a Raspberry Pi Cluster Node\u00a0to store our configuration settings in a config file. Here we move all the configuration settings in our script into a useful .cfg file using the python ConfigParser. Why use Configuration Files? When developing a system there will\u2026","rel":"","context":"In "Raspberry Pi Cluster"","block_context":{"text":"Raspberry Pi Cluster","link":"https:\/\/chewett.co.uk\/blog\/category\/raspberry-pi-cluster\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2018\/02\/rpi_cluster_04_config_parser.jpg?fit=800%2C800&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2018\/02\/rpi_cluster_04_config_parser.jpg?fit=800%2C800&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2018\/02\/rpi_cluster_04_config_parser.jpg?fit=800%2C800&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2018\/02\/rpi_cluster_04_config_parser.jpg?fit=800%2C800&ssl=1&resize=700%2C400 2x"},"classes":[]}],"_links":{"self":[{"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/posts\/881","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/comments?post=881"}],"version-history":[{"count":7,"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/posts\/881\/revisions"}],"predecessor-version":[{"id":2651,"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/posts\/881\/revisions\/2651"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/media\/886"}],"wp:attachment":[{"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/media?parent=881"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/categories?post=881"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/tags?post=881"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}