{"id":2090,"date":"2019-03-23T13:00:36","date_gmt":"2019-03-23T13:00:36","guid":{"rendered":"https:\/\/chewett.co.uk\/blog\/?p=2090"},"modified":"2020-09-26T22:29:36","modified_gmt":"2020-09-26T21:29:36","slug":"raspberry-pi-cluster-node-13-abstracting-slave-code","status":"publish","type":"post","link":"https:\/\/chewett.co.uk\/blog\/2090\/raspberry-pi-cluster-node-13-abstracting-slave-code\/","title":{"rendered":"Raspberry Pi Cluster Node \u2013 13 Abstracting Slave Code"},"content":{"rendered":"\n
\"\"<\/figure>\n\n\n\n

This post builds on my previous posts in the Raspberry Pi Cluster series<\/a> by abstracting the slave code so it is ready for more complex slaves. <\/p>\n\n\n\n\n\n\n\n

Why I am abstracting the Slave code<\/h2>\n\n\n\n

As the system becomes more complex there will be a number of slaves performing different tasks.<\/p>\n\n\n\n

Currently our basic slave reports a heartbeat to let the master know its still alive. When it joins the cluster it sends over data about the node. Once this has been sent it then receives cluster information about the master. Finally, the slave will begin repeatedly sending heartbeat data.<\/p>\n\n\n\n

For all slaves we want to ensure that they follow this standard pattern. All of the reconnection code, and sending of the node details needs to be uniform across all slaves.<\/p>\n\n\n\n

To do this I am going to abstract the code that performs this into a single class. The only difference between the slaves will be what it does once it has connected to the master.<\/p>\n\n\n\n

For the basic slave we will continue to send the heartbeat, but for the more complex slaves they will perform additional steps.<\/p>\n\n\n\n

In addition to this I am planning to run the connection code in a separate thread. This will mean that while each slave is communicating with the master it is able to perform other tasks.<\/p>\n\n\n\n

Abstracting the Slave code<\/h2>\n\n\n\n

Changing the slave code to use a thread will work similar to how we handle each slave connecting to the master. A class is created whose parent is
threading.Thread<\/code> and when used it is created and then started in a new thread.<\/p>\n\n\n\n

In this case we are going to call it the RpiBasicSlaveThread<\/code> class.<\/p>\n\n\n

\nclass RpiBasicSlaveThread(threading.Thread):\n\n    def __init__(self, master_ip, socket_port):\n        threading.Thread.__init__(self)\n        self.client_number = random.randint(1, 100000)\n        self.server_address = (master_ip, socket_port)\n        self.sock = None\n<\/pre><\/div>\n\n\n
In the init this takes the master ip, and the socket port which will be used to create the socket connection. In addition to this, a random client number is generated to use used as means of identification.<\/p>\n\n\n\n
As in previous code, the client number could conflict with another slave, so going forward this will be improved to ensure no client has the same ID.<\/p>\n\n\n\n
Here the run()<\/code> method is called once the thread has been started after calling start()<\/code>.<\/p>\n\n\n
\n    def run(self):\n        logger.info("Starting script...")\n\n        while True:\n            logger.info("Connecting to the master...")\n            connected = False\n            while connected is False:\n                try:\n                    self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)\n                    self.sock.connect(self.server_address)\n                    connected = True\n                except socket.error as e:\n                    logger.info("Failed to connect to master, waiting 60 seconds and trying again")\n                    time.sleep(60)\n\n            logger.info("Successfully connected to the master")\n\n            try:\n                logger.info("Sending an initial hello to master")\n                send_message(self.sock, create_payload(get_base_machine_info(), 'computer_details'))\n                send_message(self.sock, create_payload("computer_details", "info"))\n\n                message = get_message(self.sock)\n                logger.info("We have information about the master " + json.dumps(message['payload']))\n\n                while True:\n                    self.perform_action()\n\n            except DisconnectionException as e:\n                logger.info("Got disconnection exception with message: " + e.message)\n                logger.info("Slave will try and reconnect once master is back online")\n<\/pre><\/div>\n\n\n
This run method is primarily the contents of the original basic slave script. Here it will attempt to connect to the master until it suceeds. This also has disconnection code handling an automatic reconnect if the master is offline for a period of time.<\/p>\n\n\n\n
The one change that has been made here is to instead of perform the heartbeat code, it calls self.perform_action()<\/code>.<\/p>\n\n\n
\n    def perform_action(self):\n        logger.info("Now sending a keepalive to the master")\n        send_message(self.sock, create_payload("I am still alive, client: {num}".format(num=self.client_number)))\n        time.sleep(5)\n<\/pre><\/div>\n\n\n
Here this method performs the basic heartbeat we had in the basic slave script. However since this is a method it can easily be overwritten from a child class.<\/p>\n\n\n\n
Going forward new slaves will be a subclass of this class and override this perform_action()<\/code> method. This means that all the slaves will have the same connection code and the handling for disconnection.<\/p>\n\n\n\n
The benefit of this is that as we improve the connection handling code, all slaves will benefit from this code.<\/p>\n\n\n\n
Reviewing the basic_slave.py script<\/h2>\n\n\n\n
Now after the majority of this code has been abstracted the basic_slave script is now significantly shorter.<\/p>\n\n\n
\nconfig = ConfigParser.ConfigParser()\nconfig.read(os.path.join(os.path.dirname(os.path.realpath(__file__)), 'rpicluster.cfg'))\n\nsocket_port = config.getint("slave", "socket_port")\nmaster_ip = config.get("slave", "master_ip")\n\nadd_file_logger("slave.log")\nbasic_slave_thread = RpiBasicSlaveThread(master_ip, socket_port)\nbasic_slave_thread.start()\n<\/pre><\/div>\n\n\n
The main change now involves creating the RpiBasicSlaveThread<\/code> object, and then starting it running in a new thread with start()<\/code>. The rest of the logic that was previously here is now handled now lives in the newly created class.<\/p>\n\n\n\n
Summary of changes<\/h2>\n\n\n\n
Here we have moved the majority of the slave code into a new class. This has been refactored slightly so that we may start to implement new slaves.<\/p>\n\n\n\n
 The full code is\u00a0available on Github<\/a>, any comments or questions can be raised there as issues or posted below. <\/p>\n","protected":false},"excerpt":{"rendered":"
This post builds on my previous posts in the Raspberry Pi Cluster series by abstracting the slave code so it is ready for more complex slaves.<\/p>\n","protected":false},"author":1,"featured_media":2093,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_exactmetrics_skip_tracking":false,"_exactmetrics_sitenote_active":false,"_exactmetrics_sitenote_note":"","_exactmetrics_sitenote_category":0,"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"Today I talk about abstracting the slave code in my #RaspberryPi Cluster #DistributedComputing","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[122],"tags":[102,184,185,37],"class_list":["post-2090","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-raspberry-pi-cluster","tag-distributed-computing","tag-python","tag-rasbian","tag-raspberry-pi"],"wppr_data":{"cwp_meta_box_check":"No"},"jetpack_publicize_connections":[],"jetpack_featured_media_url":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2019\/03\/rpi_cluster_13_abstracting_slave_code.jpg?fit=800%2C800&ssl=1","jetpack_shortlink":"https:\/\/wp.me\/p2toWX-xI","jetpack_sharing_enabled":true,"jetpack-related-posts":[{"id":1913,"url":"https:\/\/chewett.co.uk\/blog\/1913\/raspberry-pi-cluster-node-10-more-advanced-connection-handling\/","url_meta":{"origin":2090,"position":0},"title":"Raspberry Pi Cluster Node \u2013 10 More Advanced Connection Handling","author":"Chewett","date":"January 16, 2019","format":false,"excerpt":"This post builds on\u00a0my previous posts in the Raspberry Pi Cluster series\u00a0by improving the connection code so it wont crash when the master or slave disconnects. Using Exceptions to handle socket issues The improvements to the cluster code are going to add an exception to the communication code. This will\u2026","rel":"","context":"In "Raspberry Pi Cluster"","block_context":{"text":"Raspberry Pi Cluster","link":"https:\/\/chewett.co.uk\/blog\/category\/raspberry-pi-cluster\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2019\/01\/rpi_cluster_10_advanced_connection_handling.jpg?fit=800%2C800&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2019\/01\/rpi_cluster_10_advanced_connection_handling.jpg?fit=800%2C800&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2019\/01\/rpi_cluster_10_advanced_connection_handling.jpg?fit=800%2C800&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2019\/01\/rpi_cluster_10_advanced_connection_handling.jpg?fit=800%2C800&ssl=1&resize=700%2C400 2x"},"classes":[]},{"id":1839,"url":"https:\/\/chewett.co.uk\/blog\/1839\/raspberry-pi-cluster-node-08-slave-helper-functions\/","url_meta":{"origin":2090,"position":1},"title":"Raspberry Pi Cluster Node – 08 Slave Helper Functions","author":"Chewett","date":"December 19, 2018","format":false,"excerpt":"This post builds on\u00a0my previous posts in the Raspberry Pi Cluster series\u00a0by adding a number of slave helper functions.\u00a0 This update will begin the process of fully automating the slaves. Preparing the Slaves for Automation Before the slaves are ready to be fully automated there are a number of commands\u2026","rel":"","context":"In "Raspberry Pi Cluster"","block_context":{"text":"Raspberry Pi Cluster","link":"https:\/\/chewett.co.uk\/blog\/category\/raspberry-pi-cluster\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2018\/12\/rpi_cluster_08_slave_functions.jpg?fit=800%2C800&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2018\/12\/rpi_cluster_08_slave_functions.jpg?fit=800%2C800&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2018\/12\/rpi_cluster_08_slave_functions.jpg?fit=800%2C800&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2018\/12\/rpi_cluster_08_slave_functions.jpg?fit=800%2C800&ssl=1&resize=700%2C400 2x"},"classes":[]},{"id":1872,"url":"https:\/\/chewett.co.uk\/blog\/1872\/raspberry-pi-cluster-node-09-multi-slave-master\/","url_meta":{"origin":2090,"position":2},"title":"Raspberry Pi Cluster Node \u2013 09 Multi Slave Master","author":"Chewett","date":"January 2, 2019","format":false,"excerpt":"This post builds on\u00a0my previous posts in the Raspberry Pi Cluster series\u00a0by changing the master so that it accepts multiple slaves connecting to it. Creating a thread to handle each client Typically to handle multiple operations occurring at once in a program, you will use additional threads or processes. These\u2026","rel":"","context":"In "Raspberry Pi Cluster"","block_context":{"text":"Raspberry Pi Cluster","link":"https:\/\/chewett.co.uk\/blog\/category\/raspberry-pi-cluster\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2018\/12\/rpi_cluster_09_multi_slave_master.jpg?fit=800%2C800&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2018\/12\/rpi_cluster_09_multi_slave_master.jpg?fit=800%2C800&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2018\/12\/rpi_cluster_09_multi_slave_master.jpg?fit=800%2C800&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2018\/12\/rpi_cluster_09_multi_slave_master.jpg?fit=800%2C800&ssl=1&resize=700%2C400 2x"},"classes":[]},{"id":1964,"url":"https:\/\/chewett.co.uk\/blog\/1964\/raspberry-pi-cluster-node-11-automatic-slave-reconnection\/","url_meta":{"origin":2090,"position":3},"title":"Raspberry Pi Cluster Node \u2013 11 Automatic Slave Reconnection","author":"Chewett","date":"January 30, 2019","format":false,"excerpt":"This post builds on\u00a0my previous posts in the Raspberry Pi Cluster series\u00a0by modifying the slave to automatically reconnect to the master when the connection is lost. Making the Slave automatically rejoin the master There are a lot of situations where the slave may fail to communicate with the master. These\u2026","rel":"","context":"In "Raspberry Pi Cluster"","block_context":{"text":"Raspberry Pi Cluster","link":"https:\/\/chewett.co.uk\/blog\/category\/raspberry-pi-cluster\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2019\/01\/rpi_cluster_11_automaic_slave_reconnection.jpg?fit=800%2C800&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2019\/01\/rpi_cluster_11_automaic_slave_reconnection.jpg?fit=800%2C800&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2019\/01\/rpi_cluster_11_automaic_slave_reconnection.jpg?fit=800%2C800&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2019\/01\/rpi_cluster_11_automaic_slave_reconnection.jpg?fit=800%2C800&ssl=1&resize=700%2C400 2x"},"classes":[]},{"id":2179,"url":"https:\/\/chewett.co.uk\/blog\/2179\/raspberry-pi-cluster-node-15-a-more-complex-webserver\/","url_meta":{"origin":2090,"position":4},"title":"Raspberry Pi Cluster Node \u2013 15 A more complex webserver","author":"Chewett","date":"May 8, 2019","format":false,"excerpt":"This tutorial focuses on improving the webserver to display information about the slaves connected to the master using python Bottle. Refactoring the Master Script To start with the changes, I am going to focus on the master script. This is going to move the changes into a new class. class\u2026","rel":"","context":"In "Raspberry Pi Cluster"","block_context":{"text":"Raspberry Pi Cluster","link":"https:\/\/chewett.co.uk\/blog\/category\/raspberry-pi-cluster\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2019\/05\/rpi_cluster_15_more_complex_webserver.jpg?fit=800%2C800&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2019\/05\/rpi_cluster_15_more_complex_webserver.jpg?fit=800%2C800&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2019\/05\/rpi_cluster_15_more_complex_webserver.jpg?fit=800%2C800&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2019\/05\/rpi_cluster_15_more_complex_webserver.jpg?fit=800%2C800&ssl=1&resize=700%2C400 2x"},"classes":[]},{"id":1781,"url":"https:\/\/chewett.co.uk\/blog\/1781\/raspberry-pi-cluster-node-07-sending-data-to-the-slave\/","url_meta":{"origin":2090,"position":5},"title":"Raspberry Pi Cluster Node \u2013 07 Sending data to the Slave","author":"Chewett","date":"December 5, 2018","format":false,"excerpt":"This post builds on\u00a0my previous posts in the Raspberry Pi Cluster series\u00a0by adding the ability to receive data from the master. In this\u00a0update, I will be adding a way\u00a0for the slave to request data and\u00a0have it returned by the master. Moving machine details into its own file The first thing\u2026","rel":"","context":"In "Raspberry Pi Cluster"","block_context":{"text":"Raspberry Pi Cluster","link":"https:\/\/chewett.co.uk\/blog\/category\/raspberry-pi-cluster\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2018\/12\/rpi_cluster_07_sending_to_slave.jpg?fit=800%2C800&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2018\/12\/rpi_cluster_07_sending_to_slave.jpg?fit=800%2C800&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2018\/12\/rpi_cluster_07_sending_to_slave.jpg?fit=800%2C800&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2018\/12\/rpi_cluster_07_sending_to_slave.jpg?fit=800%2C800&ssl=1&resize=700%2C400 2x"},"classes":[]}],"_links":{"self":[{"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/posts\/2090","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/comments?post=2090"}],"version-history":[{"count":5,"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/posts\/2090\/revisions"}],"predecessor-version":[{"id":2662,"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/posts\/2090\/revisions\/2662"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/media\/2093"}],"wp:attachment":[{"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/media?parent=2090"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/categories?post=2090"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/tags?post=2090"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}