{"id":1872,"date":"2019-01-02T13:00:48","date_gmt":"2019-01-02T13:00:48","guid":{"rendered":"http:\/\/chewett.co.uk\/blog\/?p=1872"},"modified":"2020-09-26T22:11:58","modified_gmt":"2020-09-26T21:11:58","slug":"raspberry-pi-cluster-node-09-multi-slave-master","status":"publish","type":"post","link":"https:\/\/chewett.co.uk\/blog\/1872\/raspberry-pi-cluster-node-09-multi-slave-master\/","title":{"rendered":"Raspberry Pi Cluster Node \u2013 09 Multi Slave Master"},"content":{"rendered":"\n
\"\"<\/figure>\n\n\n\n

This post builds on my previous posts in the Raspberry Pi Cluster series<\/a> by changing the master so that it accepts multiple slaves connecting to it.<\/p>\n\n\n\n\n\n\n\n

Creating a thread to handle each client<\/h2>\n\n\n\n

Typically to handle multiple operations occurring at once in a program, you will use additional threads or processes. These run at the same time as your initial “main” thread and work very similarly to it.<\/p>\n\n\n\n

In this tutorial I am going to be changing my master to create one thread per client, and handle the client wholly in the thread.<\/p>\n\n\n\n

While using threading there is the possibility for bugs to appear due to the multi-threaded nature of code. This initial implementation will not use any complex features of threads, so we will largely ignore these problems.<\/p>\n\n\n\n

This post doesn’t go into the details of how the GIL (Global Interpreter Lock) can slow down multi-threaded programs in python. This is a much more advanced topic and will be addressed in the future. For more information you can read the Global Interpreter Lock<\/a> page on the python Wiki.<\/p>\n\n\n\n

Creating a thread handler in python<\/h2>\n\n\n\n

In python one of the easiest ways to run threads in python is to create a class that inherits from threading.Thread<\/code>.<\/p>\n\n\n\n

At a basic level, you can create a run<\/code> method inside your class. This method is called once the thread is created and has been requested to start running.<\/p>\n\n\n\n

To start the thread and perform your processing, you need to instantiate the class. Once created, calling the start<\/code> method will call your run<\/code> method in the newly created thread. At the point the program in your main thread will continue running at the same time as the child thread.<\/p>\n\n\n\n

For my system I will have the main thread handle connecting to the slaves. Once the connection is created it will be passed to another thread to process.<\/p>\n\n\n\n

Below is the example of my class to handle the threads.<\/p>\n\n\n

\nclass RpiClusterClient(threading.Thread):\n\n    def __init__(self, clientsocket, address):\n        threading.Thread.__init__(self)\n        self.clientsocket = clientsocket\n        self.address = address\n\n    def run(self):\n        message = True\n        while message:\n            message = get_message(self.clientsocket)\n            if message:\n                if message['type'] == 'message':\n                    logger.info("Received message: " + message['payload'])\n                elif message['type'] == 'computer_details':\n                    logger.info("Computer specifications: " + json.dumps(message['payload']))\n                elif message['type'] == 'info':\n                    logger.info("Slave wants to know my info about " + message['payload'])\n                    if message['payload'] == 'computer_details':\n                        self.clientsocket.send(create_payload(get_base_machine_info(), "master_info"))\n                    else:\n                        self.clientsocket.send(create_payload("unknown", "bad_message"))\n\n            else:\n                logger.info("Client disconnected at address {addr}".format(addr=self.address))\n\n<\/pre><\/div>\n\n\n
Here I create my class, RpiClusterClient<\/code>, which is inheriting from the threading.Thread<\/code> class so that I can run processing in this child thread.<\/p>\n\n\n\n
The __init__<\/code> method is used to store data about the slave, the client socket and the address. The client socket is used to communicate with the connected slave so must be passed to the new thread.<\/p>\n\n\n\n
The run method is moved from the original code created in the basic_master.py<\/code> file in previous tutorials. Since this was previously discussed I wont go over what this is doing.<\/p>\n\n\n\n
Improving the master to create threads<\/h2>\n\n\n\n
Now there is a class to handle the thread processing we can improve the master to use it.<\/p>\n\n\n\n
Instead of listening to a single connection, I have set it to listen to up to 10. For a larger system this number may need to be increased.<\/p>\n\n\n
\nsocket.listen(10) #listen to 10 connects\nwhile True:\n    (clientsocket, address) = socket.accept()\n    logger.info("Got client at {address}".format(address=address))\n\n    rpiClient = RpiClusterClient(clientsocket, address)\n    rpiClient.start()\n<\/pre><\/div>\n\n\n
Since I am connecting to more than one slave the socket.accept()<\/code> call is within a while True loop to run forever.<\/p>\n\n\n\n
Inside this loop we accept the connection, then create an instance of the RpiClusterClient<\/code>. Once created start()<\/code> is called on it to initialise the thread and begin running the run()<\/code> method.<\/p>\n\n\n\n
Once start is called the thread handles the connection to the slave. At this point the master does not talk to the slave in the main thread. All the communication is handled in the child threads.<\/p>\n\n\n\n
Summary of thread handling improvements<\/h2>\n\n\n\n
This tutorial focuses on improving the master so that it may talk to many different slaves. This now means the cluster is able to start distributing tasks to many nodes at once.<\/p>\n\n\n\n
Currently there is no error handling for nodes connecting and disconnecting. The next tutorial will focus on hardening the master and slaves connection\/disconnection code.<\/p>\n\n\n\n
The full code is\u00a0available on Github<\/a>, any comments or questions can be raised there as issues or posted below.<\/p>\n","protected":false},"excerpt":{"rendered":"
This post builds on my previous posts in the Raspberry Pi Cluster series by changing the master so that it accepts multiple slaves connecting to it.<\/p>\n","protected":false},"author":1,"featured_media":1877,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_exactmetrics_skip_tracking":false,"_exactmetrics_sitenote_active":false,"_exactmetrics_sitenote_note":"","_exactmetrics_sitenote_category":0,"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"Today I have written about improving the #RaspberryPi so that the master can connect to many slaves #DistributedComputing","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[122],"tags":[102,184,185,37],"class_list":["post-1872","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-raspberry-pi-cluster","tag-distributed-computing","tag-python","tag-rasbian","tag-raspberry-pi"],"wppr_data":{"cwp_meta_box_check":"No"},"jetpack_publicize_connections":[],"jetpack_featured_media_url":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2018\/12\/rpi_cluster_09_multi_slave_master.jpg?fit=800%2C800&ssl=1","jetpack_shortlink":"https:\/\/wp.me\/p2toWX-uc","jetpack_sharing_enabled":true,"jetpack-related-posts":[{"id":2090,"url":"https:\/\/chewett.co.uk\/blog\/2090\/raspberry-pi-cluster-node-13-abstracting-slave-code\/","url_meta":{"origin":1872,"position":0},"title":"Raspberry Pi Cluster Node \u2013 13 Abstracting Slave Code","author":"Chewett","date":"March 23, 2019","format":false,"excerpt":"This post builds on my previous posts in the Raspberry Pi Cluster series by abstracting the slave code so it is ready for more complex slaves. Why I am abstracting the Slave code As the system becomes more complex there will be a number of slaves performing different tasks. Currently\u2026","rel":"","context":"In "Raspberry Pi Cluster"","block_context":{"text":"Raspberry Pi Cluster","link":"https:\/\/chewett.co.uk\/blog\/category\/raspberry-pi-cluster\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2019\/03\/rpi_cluster_13_abstracting_slave_code.jpg?fit=800%2C800&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2019\/03\/rpi_cluster_13_abstracting_slave_code.jpg?fit=800%2C800&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2019\/03\/rpi_cluster_13_abstracting_slave_code.jpg?fit=800%2C800&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2019\/03\/rpi_cluster_13_abstracting_slave_code.jpg?fit=800%2C800&ssl=1&resize=700%2C400 2x"},"classes":[]},{"id":2127,"url":"https:\/\/chewett.co.uk\/blog\/2127\/raspberry-pi-cluster-node-14-a-simple-webserver\/","url_meta":{"origin":1872,"position":1},"title":"Raspberry Pi Cluster Node \u2013 14 A simple webserver","author":"Chewett","date":"April 17, 2019","format":false,"excerpt":"This tutorial focuses on creating a simple webserver that displays the status of the master using python Bottle. What will the webserver be used for? To interact with the cluster I am planning on making a small set of webpages. Initially these will just display information about the cluster but\u2026","rel":"","context":"In "Raspberry Pi Cluster"","block_context":{"text":"Raspberry Pi Cluster","link":"https:\/\/chewett.co.uk\/blog\/category\/raspberry-pi-cluster\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2019\/04\/rpi_cluster_14_a_simple_webserver.jpg?fit=800%2C800&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2019\/04\/rpi_cluster_14_a_simple_webserver.jpg?fit=800%2C800&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2019\/04\/rpi_cluster_14_a_simple_webserver.jpg?fit=800%2C800&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2019\/04\/rpi_cluster_14_a_simple_webserver.jpg?fit=800%2C800&ssl=1&resize=700%2C400 2x"},"classes":[]},{"id":1913,"url":"https:\/\/chewett.co.uk\/blog\/1913\/raspberry-pi-cluster-node-10-more-advanced-connection-handling\/","url_meta":{"origin":1872,"position":2},"title":"Raspberry Pi Cluster Node \u2013 10 More Advanced Connection Handling","author":"Chewett","date":"January 16, 2019","format":false,"excerpt":"This post builds on\u00a0my previous posts in the Raspberry Pi Cluster series\u00a0by improving the connection code so it wont crash when the master or slave disconnects. Using Exceptions to handle socket issues The improvements to the cluster code are going to add an exception to the communication code. This will\u2026","rel":"","context":"In "Raspberry Pi Cluster"","block_context":{"text":"Raspberry Pi Cluster","link":"https:\/\/chewett.co.uk\/blog\/category\/raspberry-pi-cluster\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2019\/01\/rpi_cluster_10_advanced_connection_handling.jpg?fit=800%2C800&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2019\/01\/rpi_cluster_10_advanced_connection_handling.jpg?fit=800%2C800&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2019\/01\/rpi_cluster_10_advanced_connection_handling.jpg?fit=800%2C800&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2019\/01\/rpi_cluster_10_advanced_connection_handling.jpg?fit=800%2C800&ssl=1&resize=700%2C400 2x"},"classes":[]},{"id":901,"url":"https:\/\/chewett.co.uk\/blog\/901\/raspberry-pi-cluster-node-03-basic-node-communication-two-nodes\/","url_meta":{"origin":1872,"position":3},"title":"Raspberry Pi Cluster Node \u2013 03 Basic node communication between two nodes","author":"Chewett","date":"December 9, 2017","format":false,"excerpt":"This post builds on the second step to create a Raspberry Pi Cluster Node to create a hello world application across two different scripts. Here we will create a socket connection between the two and use it to send a message to the master. Designing the client-master Raspberry Pi Nodes\u2026","rel":"","context":"In "Raspberry Pi Cluster"","block_context":{"text":"Raspberry Pi Cluster","link":"https:\/\/chewett.co.uk\/blog\/category\/raspberry-pi-cluster\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2017\/12\/rpi_cluster_03_basic_communication.jpg?fit=800%2C800&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2017\/12\/rpi_cluster_03_basic_communication.jpg?fit=800%2C800&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2017\/12\/rpi_cluster_03_basic_communication.jpg?fit=800%2C800&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2017\/12\/rpi_cluster_03_basic_communication.jpg?fit=800%2C800&ssl=1&resize=700%2C400 2x"},"classes":[]},{"id":232,"url":"https:\/\/chewett.co.uk\/blog\/232\/building-raspberry-pi-cluster\/","url_meta":{"origin":1872,"position":4},"title":"Building a Raspberry Pi cluster","author":"Chewett","date":"October 16, 2016","format":false,"excerpt":"Today's blog marks the official start of my Raspberry Pi cluster project. Here I will be documenting the process of assembling,\u00a0building, and running a small cluster.\u00a0This blog will include both code snippets, \u00a0hardware diagrams and the results of various projects using the cluster. Aims of the project The\u00a0Raspberry Pi cluster\u2026","rel":"","context":"In "Raspberry Pi Cluster"","block_context":{"text":"Raspberry Pi Cluster","link":"https:\/\/chewett.co.uk\/blog\/category\/raspberry-pi-cluster\/"},"img":{"alt_text":"dsc_0525","src":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2016\/10\/DSC_0525.jpg?resize=350%2C200","width":350,"height":200},"classes":[]},{"id":2179,"url":"https:\/\/chewett.co.uk\/blog\/2179\/raspberry-pi-cluster-node-15-a-more-complex-webserver\/","url_meta":{"origin":1872,"position":5},"title":"Raspberry Pi Cluster Node \u2013 15 A more complex webserver","author":"Chewett","date":"May 8, 2019","format":false,"excerpt":"This tutorial focuses on improving the webserver to display information about the slaves connected to the master using python Bottle. Refactoring the Master Script To start with the changes, I am going to focus on the master script. This is going to move the changes into a new class. class\u2026","rel":"","context":"In "Raspberry Pi Cluster"","block_context":{"text":"Raspberry Pi Cluster","link":"https:\/\/chewett.co.uk\/blog\/category\/raspberry-pi-cluster\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2019\/05\/rpi_cluster_15_more_complex_webserver.jpg?fit=800%2C800&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2019\/05\/rpi_cluster_15_more_complex_webserver.jpg?fit=800%2C800&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2019\/05\/rpi_cluster_15_more_complex_webserver.jpg?fit=800%2C800&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2019\/05\/rpi_cluster_15_more_complex_webserver.jpg?fit=800%2C800&ssl=1&resize=700%2C400 2x"},"classes":[]}],"_links":{"self":[{"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/posts\/1872","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/comments?post=1872"}],"version-history":[{"count":6,"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/posts\/1872\/revisions"}],"predecessor-version":[{"id":2658,"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/posts\/1872\/revisions\/2658"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/media\/1877"}],"wp:attachment":[{"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/media?parent=1872"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/categories?post=1872"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/tags?post=1872"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}