{"id":1964,"date":"2019-01-30T13:00:59","date_gmt":"2019-01-30T13:00:59","guid":{"rendered":"http:\/\/chewett.co.uk\/blog\/?p=1964"},"modified":"2020-09-26T22:27:44","modified_gmt":"2020-09-26T21:27:44","slug":"raspberry-pi-cluster-node-11-automatic-slave-reconnection","status":"publish","type":"post","link":"https:\/\/chewett.co.uk\/blog\/1964\/raspberry-pi-cluster-node-11-automatic-slave-reconnection\/","title":{"rendered":"Raspberry Pi Cluster Node \u2013 11 Automatic Slave Reconnection"},"content":{"rendered":"\n
\"\"<\/figure>\n\n\n\n

This post builds on my previous posts in the Raspberry Pi Cluster series<\/a> by modifying the slave to automatically reconnect to the master when the connection is lost.<\/p>\n\n\n\n\n\n\n\n

Making the Slave automatically rejoin the master<\/h2>\n\n\n\n

There are a lot of situations where the slave may fail to communicate with the master.<\/p>\n\n\n\n

These range from hardware issues such as the slave temporarily losing networking capability to software issues in the master code requiring reboot.<\/p>\n\n\n\n

To make the cluster robust against many different types of failure it will need to resume operation whenever there is an issue.<\/p>\n\n\n\n

By improving the slave, allowing it to automatically reconnect to the master this handles many situations where the master fails. Once the master comes back online the slave will detect this and reconnect. Once reconnected, the slave will continue operations and the operation of the cluster should seamlessly resume.<\/p>\n\n\n\n

This will also mean that when the master needs to restart the slave will automatically reconnect once it is back online.<\/p>\n\n\n\n

For this tutorial we will ignore slave failures, which will be handled in a later tutorial. It is assumed that when the master goes down, it will automatically come back online at an unspecified period later on.<\/p>\n\n\n\n

Modifying the slave to reconnect<\/h2>\n\n\n\n

Currently we catch the disconnection exception thrown when the master disconnects. This then ends our script which stops the slave running. Today we are going to change it so that once it disconnects the slave attempts to reconnect. To do this we are going to encase this code in a While True loop.<\/p>\n\n\n

\nsock = None\nclient_number = random.randint(1, 100000)\nserver_address = (master_ip, socket_port)\n<\/pre><\/div>\n\n\n
The first piece of work is to refactor our socket code and to move the constant data outside of the new loop. Here we store our client number, a socket reference that we set to None to start with, and the server address we are going to use.<\/p>\n\n\n\n
Next we create the While loop that our slave code will be encased in<\/p>\n\n\n
\nwhile True:\n    logger.info("Connecting to the master...")\n    connected = False\n    while connected is False:\n        try:\n            sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)\n            sock.connect(server_address)\n            connected = True\n        except socket.error as e:\n            logger.info("Failed to connect to master, waiting 60 seconds and trying again")\n            time.sleep(60)\n\n    logger.info("Successfully connected to the master")\n<\/pre><\/div>\n\n\n
Here we create the socket and attempt to connect to the master. If this succeeds we set connected<\/code> to True<\/code> and begin the slave code previously written.<\/p>\n\n\n\n
If this fails however we catch the exception and log that it failed to connect. The slave will wait 60 seconds and then attempt to connect to the master again. This will continue until it has successfully connected to the master.<\/p>\n\n\n\n
Since this code is within a While True loop, the slave code will continue to run until the master disconnects. Once this happens the loop is reset and the slave attempts to connect again. This will again keep trying until it successfully connects.<\/p>\n\n\n\n
Summary of changes to the Rpi Cluster<\/h2>\n\n\n\n
Now our slave will automatically reconnect to the master no matter how many times it has been restarted. This moves the cluster one step forward towards a system which will automatically repair if there is an issue.<\/p>\n\n\n\n
In the next tutorial we will look at how we can make the cluster code run on boot using rc.local<\/code>.<\/p>\n\n\n\n
The full code is\u00a0available on Github<\/a>, any comments or questions can be raised there as issues or posted below.<\/p>\n","protected":false},"excerpt":{"rendered":"
This post builds on my previous posts in the Raspberry Pi Cluster series by modifying the slave to automatically reconnect to the master when the connection is lost.<\/p>\n","protected":false},"author":1,"featured_media":1967,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_exactmetrics_skip_tracking":false,"_exactmetrics_sitenote_active":false,"_exactmetrics_sitenote_note":"","_exactmetrics_sitenote_category":0,"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"Today I talk about the latest Post in the #RaspberryPi Cluster Series, Automatic Slave Reconnection #python #DistributedComputing","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[122],"tags":[102,184,185,37],"class_list":["post-1964","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-raspberry-pi-cluster","tag-distributed-computing","tag-python","tag-rasbian","tag-raspberry-pi"],"wppr_data":{"cwp_meta_box_check":"No"},"jetpack_publicize_connections":[],"jetpack_featured_media_url":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2019\/01\/rpi_cluster_11_automaic_slave_reconnection.jpg?fit=800%2C800&ssl=1","jetpack_shortlink":"https:\/\/wp.me\/p2toWX-vG","jetpack_sharing_enabled":true,"jetpack-related-posts":[{"id":1781,"url":"https:\/\/chewett.co.uk\/blog\/1781\/raspberry-pi-cluster-node-07-sending-data-to-the-slave\/","url_meta":{"origin":1964,"position":0},"title":"Raspberry Pi Cluster Node \u2013 07 Sending data to the Slave","author":"Chewett","date":"December 5, 2018","format":false,"excerpt":"This post builds on\u00a0my previous posts in the Raspberry Pi Cluster series\u00a0by adding the ability to receive data from the master. In this\u00a0update, I will be adding a way\u00a0for the slave to request data and\u00a0have it returned by the master. Moving machine details into its own file The first thing\u2026","rel":"","context":"In "Raspberry Pi Cluster"","block_context":{"text":"Raspberry Pi Cluster","link":"https:\/\/chewett.co.uk\/blog\/category\/raspberry-pi-cluster\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2018\/12\/rpi_cluster_07_sending_to_slave.jpg?fit=800%2C800&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2018\/12\/rpi_cluster_07_sending_to_slave.jpg?fit=800%2C800&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2018\/12\/rpi_cluster_07_sending_to_slave.jpg?fit=800%2C800&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2018\/12\/rpi_cluster_07_sending_to_slave.jpg?fit=800%2C800&ssl=1&resize=700%2C400 2x"},"classes":[]},{"id":1913,"url":"https:\/\/chewett.co.uk\/blog\/1913\/raspberry-pi-cluster-node-10-more-advanced-connection-handling\/","url_meta":{"origin":1964,"position":1},"title":"Raspberry Pi Cluster Node \u2013 10 More Advanced Connection Handling","author":"Chewett","date":"January 16, 2019","format":false,"excerpt":"This post builds on\u00a0my previous posts in the Raspberry Pi Cluster series\u00a0by improving the connection code so it wont crash when the master or slave disconnects. Using Exceptions to handle socket issues The improvements to the cluster code are going to add an exception to the communication code. This will\u2026","rel":"","context":"In "Raspberry Pi Cluster"","block_context":{"text":"Raspberry Pi Cluster","link":"https:\/\/chewett.co.uk\/blog\/category\/raspberry-pi-cluster\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2019\/01\/rpi_cluster_10_advanced_connection_handling.jpg?fit=800%2C800&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2019\/01\/rpi_cluster_10_advanced_connection_handling.jpg?fit=800%2C800&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2019\/01\/rpi_cluster_10_advanced_connection_handling.jpg?fit=800%2C800&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2019\/01\/rpi_cluster_10_advanced_connection_handling.jpg?fit=800%2C800&ssl=1&resize=700%2C400 2x"},"classes":[]},{"id":2127,"url":"https:\/\/chewett.co.uk\/blog\/2127\/raspberry-pi-cluster-node-14-a-simple-webserver\/","url_meta":{"origin":1964,"position":2},"title":"Raspberry Pi Cluster Node \u2013 14 A simple webserver","author":"Chewett","date":"April 17, 2019","format":false,"excerpt":"This tutorial focuses on creating a simple webserver that displays the status of the master using python Bottle. What will the webserver be used for? To interact with the cluster I am planning on making a small set of webpages. Initially these will just display information about the cluster but\u2026","rel":"","context":"In "Raspberry Pi Cluster"","block_context":{"text":"Raspberry Pi Cluster","link":"https:\/\/chewett.co.uk\/blog\/category\/raspberry-pi-cluster\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2019\/04\/rpi_cluster_14_a_simple_webserver.jpg?fit=800%2C800&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2019\/04\/rpi_cluster_14_a_simple_webserver.jpg?fit=800%2C800&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2019\/04\/rpi_cluster_14_a_simple_webserver.jpg?fit=800%2C800&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2019\/04\/rpi_cluster_14_a_simple_webserver.jpg?fit=800%2C800&ssl=1&resize=700%2C400 2x"},"classes":[]},{"id":2090,"url":"https:\/\/chewett.co.uk\/blog\/2090\/raspberry-pi-cluster-node-13-abstracting-slave-code\/","url_meta":{"origin":1964,"position":3},"title":"Raspberry Pi Cluster Node \u2013 13 Abstracting Slave Code","author":"Chewett","date":"March 23, 2019","format":false,"excerpt":"This post builds on my previous posts in the Raspberry Pi Cluster series by abstracting the slave code so it is ready for more complex slaves. Why I am abstracting the Slave code As the system becomes more complex there will be a number of slaves performing different tasks. Currently\u2026","rel":"","context":"In "Raspberry Pi Cluster"","block_context":{"text":"Raspberry Pi Cluster","link":"https:\/\/chewett.co.uk\/blog\/category\/raspberry-pi-cluster\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2019\/03\/rpi_cluster_13_abstracting_slave_code.jpg?fit=800%2C800&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2019\/03\/rpi_cluster_13_abstracting_slave_code.jpg?fit=800%2C800&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2019\/03\/rpi_cluster_13_abstracting_slave_code.jpg?fit=800%2C800&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2019\/03\/rpi_cluster_13_abstracting_slave_code.jpg?fit=800%2C800&ssl=1&resize=700%2C400 2x"},"classes":[]},{"id":1839,"url":"https:\/\/chewett.co.uk\/blog\/1839\/raspberry-pi-cluster-node-08-slave-helper-functions\/","url_meta":{"origin":1964,"position":4},"title":"Raspberry Pi Cluster Node – 08 Slave Helper Functions","author":"Chewett","date":"December 19, 2018","format":false,"excerpt":"This post builds on\u00a0my previous posts in the Raspberry Pi Cluster series\u00a0by adding a number of slave helper functions.\u00a0 This update will begin the process of fully automating the slaves. Preparing the Slaves for Automation Before the slaves are ready to be fully automated there are a number of commands\u2026","rel":"","context":"In "Raspberry Pi Cluster"","block_context":{"text":"Raspberry Pi Cluster","link":"https:\/\/chewett.co.uk\/blog\/category\/raspberry-pi-cluster\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2018\/12\/rpi_cluster_08_slave_functions.jpg?fit=800%2C800&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2018\/12\/rpi_cluster_08_slave_functions.jpg?fit=800%2C800&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2018\/12\/rpi_cluster_08_slave_functions.jpg?fit=800%2C800&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2018\/12\/rpi_cluster_08_slave_functions.jpg?fit=800%2C800&ssl=1&resize=700%2C400 2x"},"classes":[]},{"id":1872,"url":"https:\/\/chewett.co.uk\/blog\/1872\/raspberry-pi-cluster-node-09-multi-slave-master\/","url_meta":{"origin":1964,"position":5},"title":"Raspberry Pi Cluster Node \u2013 09 Multi Slave Master","author":"Chewett","date":"January 2, 2019","format":false,"excerpt":"This post builds on\u00a0my previous posts in the Raspberry Pi Cluster series\u00a0by changing the master so that it accepts multiple slaves connecting to it. Creating a thread to handle each client Typically to handle multiple operations occurring at once in a program, you will use additional threads or processes. These\u2026","rel":"","context":"In "Raspberry Pi Cluster"","block_context":{"text":"Raspberry Pi Cluster","link":"https:\/\/chewett.co.uk\/blog\/category\/raspberry-pi-cluster\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2018\/12\/rpi_cluster_09_multi_slave_master.jpg?fit=800%2C800&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2018\/12\/rpi_cluster_09_multi_slave_master.jpg?fit=800%2C800&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2018\/12\/rpi_cluster_09_multi_slave_master.jpg?fit=800%2C800&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2018\/12\/rpi_cluster_09_multi_slave_master.jpg?fit=800%2C800&ssl=1&resize=700%2C400 2x"},"classes":[]}],"_links":{"self":[{"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/posts\/1964","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/comments?post=1964"}],"version-history":[{"count":5,"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/posts\/1964\/revisions"}],"predecessor-version":[{"id":2660,"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/posts\/1964\/revisions\/2660"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/media\/1967"}],"wp:attachment":[{"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/media?parent=1964"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/categories?post=1964"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/tags?post=1964"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}