{"id":1839,"date":"2018-12-19T13:00:36","date_gmt":"2018-12-19T13:00:36","guid":{"rendered":"http:\/\/chewett.co.uk\/blog\/?p=1839"},"modified":"2020-09-26T22:11:25","modified_gmt":"2020-09-26T21:11:25","slug":"raspberry-pi-cluster-node-08-slave-helper-functions","status":"publish","type":"post","link":"https:\/\/chewett.co.uk\/blog\/1839\/raspberry-pi-cluster-node-08-slave-helper-functions\/","title":{"rendered":"Raspberry Pi Cluster Node – 08 Slave Helper Functions"},"content":{"rendered":"\n
\"\"<\/figure>\n\n\n\n

This post builds on my previous posts in the Raspberry Pi Cluster series<\/a> by adding a number of slave helper functions.  This update will begin the process of fully automating the slaves.<\/p>\n\n\n\n\n\n\n\n

Preparing the Slaves for Automation<\/h2>\n\n\n\n

Before the slaves are ready to be fully automated there are a number of commands I want it to be able to perform when needed. Some of these are obvious  such as being able to be rebooted and shut down.<\/p>\n\n\n\n

The others are a little more complex, such as being able to update both the code it is running and operating system.<\/p>\n\n\n\n

Slave Commands<\/h2>\n\n\n\n

Below I will describe some of the commands that I have written for the slave and how I am going to use them. All of these functions are going to be stored in the NodeTasks.py<\/code> file.<\/p>\n\n\n\n

Reboot<\/h3>\n\n\n
\ndef reboot():\n    subprocess.call(["sudo", "reboot"])\n<\/pre><\/div>\n\n\n
The reboot command is going to be used for whenever the node needs to be rebooted. Typically this will be called once it has been updated to fully ensure it is running the latest software.<\/p>\n\n\n\n
Once the node has been rebooted it is expected to come back online and talk to the master. This will be handled in a future post.<\/p>\n\n\n\n
Shutdown<\/h3>\n\n\n
\ndef shutdown():\n    subprocess.call(["sudo", "shutdown", "-h"])\n<\/pre><\/div>\n\n\n
The shutdown command is going to be used when I want to take the nodes offline for work. This will be rarely used but will be important for maintenance.<\/p>\n\n\n\n
Once called a node should fully shut down so that its power can be removed. The node will be manually booted once maintenance is finished. Again it is assumed that it will automatically connect to the mater once booted.<\/p>\n\n\n\n
Update Node<\/h3>\n\n\n
\ndef update_node():\n    node_update_log_1 = subprocess.check_output(["sudo", "apt-get", "update"])\n    node_update_log_2 = subprocess.check_output(["sudo", "apt-get", "upgrade"])\n    return [node_update_log_1, node_update_log_2]\n<\/pre><\/div>\n\n\n
For now it is assumed all nodes are running apt based package systems. This means that to update them we can run apt-get update<\/code>, followed by apt-get upgrade<\/code>. <\/p>\n\n\n\n
The output of these two commands are returned from this function. This was not done for the previous two functions because rebooting and shutting down will not return any data, as the node will have turned off.<\/p>\n\n\n\n
Get Node Time<\/h3>\n\n\n
\ndef get_node_time():\n    return time.time()\n<\/pre><\/div>\n\n\n
It is important to make sure all nodes have the correct time. This is to ensure that the cluster is all running the same time. When messages are encrypted the time will become more important. This reasoning for this will be discussed in future blogposts.<\/p>\n\n\n\n
Get Node Codebase Revision<\/h3>\n\n\n
\ndef get_node_codebase_revision():\n    #Note: This is not nesscarily the revision of the code currently running\n    current_rev = subprocess.check_output(["git", "rev-parse", "HEAD"])\n    return current_rev.rstrip("\\r\\n")\n<\/pre><\/div>\n\n\n
To start with it is assumed that the nodes will synchronise code using a shared git repository. This means its important to know what revision each node is running.<\/p>\n\n\n\n
This command will call git to determine the current revision of the code. This however will only determine the current revision in the directory which may be different from the revision that is running.<\/p>\n\n\n\n
This problem will be resolved and discussed in detail in a future blogpost.<\/p>\n\n\n\n
Get Node Codebase Revision Time<\/h3>\n\n\n
\ndef get_node_codebase_revision_time():\n    node_rev = get_node_codebase_revision()\n    rev_time = subprocess.check_output(["git", "show", "-s", "--format=%ct", node_rev])\n    return int(rev_time.rstrip("\\r\\n"))\n<\/pre><\/div>\n\n\n
Since git does not have sequential revision numbers like SVN or Perforce you cannot quickly know whether one revision is ahead of another. However by using git show<\/code> you can obtain the time of the commit.<\/p>\n\n\n\n
This will be used by the master to determine which node has the later codebase, and update other nodes not on this version. This uses the previously defined function get_node_codebase_revision<\/code> to get the current codebase revision.<\/p>\n\n\n\n
Update Node Codebase<\/h3>\n\n\n
\ndef update_node_codebase():\n    update_log = subprocess.check_output(["git", "pull"])\n    return update_log\n<\/pre><\/div>\n\n\n
The final command is to update the node codebase using git<\/code>.  As mentioned earlier it is assumed that all nodes are deployed from a git repository.<\/p>\n\n\n\n
By using the command git pull<\/code> the code deployed on the node will be updated from the master git repository. It is assumed here that the node will be configured so no password is needed to update the git repository. <\/p>\n\n\n\n
Therefore to update the nodes we should just be able to run git pull<\/code> and the node will successfully be updated. This will be used in conjunction with rebooting the node to update it to the latest version.<\/p>\n\n\n\n
Summary of newly added functions<\/h2>\n\n\n\n
Now we have a number of functions available to the Nodes to able to control them. At this stage they are still not ready to run master defined scripts but these will form the basis of the automation.<\/p>\n\n\n\n
The next tutorial will focus on converting the master script to accept any number of slave nodes.<\/p>\n\n\n\n
 The full code is\u00a0available on Github<\/a>, any comments or questions can be raised there as issues or posted below. <\/p>\n","protected":false},"excerpt":{"rendered":"
This post builds on my previous posts in the Raspberry Pi Cluster series by adding a number of slave helper functions.  This update will begin the process of fully automating the slaves.<\/p>\n","protected":false},"author":1,"featured_media":1843,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_exactmetrics_skip_tracking":false,"_exactmetrics_sitenote_active":false,"_exactmetrics_sitenote_note":"","_exactmetrics_sitenote_category":0,"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"Today I talk about the eight #RaspberryPi #Distributed #Computing tutorial, adding some basic slave commands","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[122],"tags":[102,184,185,37,101],"class_list":["post-1839","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-raspberry-pi-cluster","tag-distributed-computing","tag-python","tag-rasbian","tag-raspberry-pi","tag-raspberry-pi-cluster"],"wppr_data":{"cwp_meta_box_check":"No"},"jetpack_publicize_connections":[],"jetpack_featured_media_url":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2018\/12\/rpi_cluster_08_slave_functions.jpg?fit=800%2C800&ssl=1","jetpack_shortlink":"https:\/\/wp.me\/p2toWX-tF","jetpack_sharing_enabled":true,"jetpack-related-posts":[{"id":2090,"url":"https:\/\/chewett.co.uk\/blog\/2090\/raspberry-pi-cluster-node-13-abstracting-slave-code\/","url_meta":{"origin":1839,"position":0},"title":"Raspberry Pi Cluster Node \u2013 13 Abstracting Slave Code","author":"Chewett","date":"March 23, 2019","format":false,"excerpt":"This post builds on my previous posts in the Raspberry Pi Cluster series by abstracting the slave code so it is ready for more complex slaves. Why I am abstracting the Slave code As the system becomes more complex there will be a number of slaves performing different tasks. Currently\u2026","rel":"","context":"In "Raspberry Pi Cluster"","block_context":{"text":"Raspberry Pi Cluster","link":"https:\/\/chewett.co.uk\/blog\/category\/raspberry-pi-cluster\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2019\/03\/rpi_cluster_13_abstracting_slave_code.jpg?fit=800%2C800&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2019\/03\/rpi_cluster_13_abstracting_slave_code.jpg?fit=800%2C800&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2019\/03\/rpi_cluster_13_abstracting_slave_code.jpg?fit=800%2C800&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2019\/03\/rpi_cluster_13_abstracting_slave_code.jpg?fit=800%2C800&ssl=1&resize=700%2C400 2x"},"classes":[]},{"id":2179,"url":"https:\/\/chewett.co.uk\/blog\/2179\/raspberry-pi-cluster-node-15-a-more-complex-webserver\/","url_meta":{"origin":1839,"position":1},"title":"Raspberry Pi Cluster Node \u2013 15 A more complex webserver","author":"Chewett","date":"May 8, 2019","format":false,"excerpt":"This tutorial focuses on improving the webserver to display information about the slaves connected to the master using python Bottle. Refactoring the Master Script To start with the changes, I am going to focus on the master script. This is going to move the changes into a new class. class\u2026","rel":"","context":"In "Raspberry Pi Cluster"","block_context":{"text":"Raspberry Pi Cluster","link":"https:\/\/chewett.co.uk\/blog\/category\/raspberry-pi-cluster\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2019\/05\/rpi_cluster_15_more_complex_webserver.jpg?fit=800%2C800&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2019\/05\/rpi_cluster_15_more_complex_webserver.jpg?fit=800%2C800&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2019\/05\/rpi_cluster_15_more_complex_webserver.jpg?fit=800%2C800&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2019\/05\/rpi_cluster_15_more_complex_webserver.jpg?fit=800%2C800&ssl=1&resize=700%2C400 2x"},"classes":[]},{"id":658,"url":"https:\/\/chewett.co.uk\/blog\/658\/distributed-computing-raspberry-pi-cluster\/","url_meta":{"origin":1839,"position":2},"title":"Distributed computing on the Raspberry Pi Cluster","author":"Chewett","date":"September 23, 2017","format":false,"excerpt":"This post describes the planned setup for distributed computing on the Raspberry Pi Cluster. A basic distributed system A very basic distributed system will have one Raspberry Pi asking as a master machine. This will coordinate work among all of the slaves. Eventually, we will move away from having a\u2026","rel":"","context":"In "Raspberry Pi Cluster"","block_context":{"text":"Raspberry Pi Cluster","link":"https:\/\/chewett.co.uk\/blog\/category\/raspberry-pi-cluster\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2017\/09\/distributed_computing_design.jpg?fit=800%2C800&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2017\/09\/distributed_computing_design.jpg?fit=800%2C800&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2017\/09\/distributed_computing_design.jpg?fit=800%2C800&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2017\/09\/distributed_computing_design.jpg?fit=800%2C800&ssl=1&resize=700%2C400 2x"},"classes":[]},{"id":1872,"url":"https:\/\/chewett.co.uk\/blog\/1872\/raspberry-pi-cluster-node-09-multi-slave-master\/","url_meta":{"origin":1839,"position":3},"title":"Raspberry Pi Cluster Node \u2013 09 Multi Slave Master","author":"Chewett","date":"January 2, 2019","format":false,"excerpt":"This post builds on\u00a0my previous posts in the Raspberry Pi Cluster series\u00a0by changing the master so that it accepts multiple slaves connecting to it. Creating a thread to handle each client Typically to handle multiple operations occurring at once in a program, you will use additional threads or processes. These\u2026","rel":"","context":"In "Raspberry Pi Cluster"","block_context":{"text":"Raspberry Pi Cluster","link":"https:\/\/chewett.co.uk\/blog\/category\/raspberry-pi-cluster\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2018\/12\/rpi_cluster_09_multi_slave_master.jpg?fit=800%2C800&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2018\/12\/rpi_cluster_09_multi_slave_master.jpg?fit=800%2C800&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2018\/12\/rpi_cluster_09_multi_slave_master.jpg?fit=800%2C800&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2018\/12\/rpi_cluster_09_multi_slave_master.jpg?fit=800%2C800&ssl=1&resize=700%2C400 2x"},"classes":[]},{"id":741,"url":"https:\/\/chewett.co.uk\/blog\/741\/raspberry-pi-cluster-node-01-logging-liveness\/","url_meta":{"origin":1839,"position":4},"title":"Raspberry Pi Cluster Node – 01 Logging Liveness","author":"Chewett","date":"November 1, 2017","format":false,"excerpt":"This post describes how to make a simple python script that logs the node is alive every 10 seconds. Why we are going to log each node is alive As discussed in the previous post on Distributed Computing on the Raspberry Pi Cluster there will be many slaves and a\u2026","rel":"","context":"In "Raspberry Pi Cluster"","block_context":{"text":"Raspberry Pi Cluster","link":"https:\/\/chewett.co.uk\/blog\/category\/raspberry-pi-cluster\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2017\/10\/rpi_cluster_01_logging_liveness.jpg?fit=800%2C800&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2017\/10\/rpi_cluster_01_logging_liveness.jpg?fit=800%2C800&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2017\/10\/rpi_cluster_01_logging_liveness.jpg?fit=800%2C800&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2017\/10\/rpi_cluster_01_logging_liveness.jpg?fit=800%2C800&ssl=1&resize=700%2C400 2x"},"classes":[]},{"id":1001,"url":"https:\/\/chewett.co.uk\/blog\/1001\/raspberry-pi-cluster-node-04-configuration-files-configparser\/","url_meta":{"origin":1839,"position":5},"title":"Raspberry Pi Cluster Node \u2013 04 Configuration Files with ConfigParser","author":"Chewett","date":"February 10, 2018","format":false,"excerpt":"This post builds on the\u00a0third step to create a Raspberry Pi Cluster Node\u00a0to store our configuration settings in a config file. Here we move all the configuration settings in our script into a useful .cfg file using the python ConfigParser. Why use Configuration Files? When developing a system there will\u2026","rel":"","context":"In "Raspberry Pi Cluster"","block_context":{"text":"Raspberry Pi Cluster","link":"https:\/\/chewett.co.uk\/blog\/category\/raspberry-pi-cluster\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2018\/02\/rpi_cluster_04_config_parser.jpg?fit=800%2C800&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2018\/02\/rpi_cluster_04_config_parser.jpg?fit=800%2C800&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2018\/02\/rpi_cluster_04_config_parser.jpg?fit=800%2C800&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2018\/02\/rpi_cluster_04_config_parser.jpg?fit=800%2C800&ssl=1&resize=700%2C400 2x"},"classes":[]}],"_links":{"self":[{"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/posts\/1839","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/comments?post=1839"}],"version-history":[{"count":7,"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/posts\/1839\/revisions"}],"predecessor-version":[{"id":2657,"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/posts\/1839\/revisions\/2657"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/media\/1843"}],"wp:attachment":[{"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/media?parent=1839"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/categories?post=1839"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/tags?post=1839"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}