{"id":2744,"date":"2020-12-19T13:00:00","date_gmt":"2020-12-19T13:00:00","guid":{"rendered":"https:\/\/chewett.co.uk\/blog\/?p=2744"},"modified":"2020-12-17T23:50:03","modified_gmt":"2020-12-17T23:50:03","slug":"raspberry-pi-cluster-node-17-influxdb-machine-stats-monitoring","status":"publish","type":"post","link":"https:\/\/chewett.co.uk\/blog\/2744\/raspberry-pi-cluster-node-17-influxdb-machine-stats-monitoring\/","title":{"rendered":"Raspberry Pi Cluster Node \u2013 17 InfluxDB Machine Stats Monitoring"},"content":{"rendered":"\n
\"\"<\/figure>\n\n\n\n

This post builds on my previous posts in the Raspberry Pi Cluster series<\/a> by performing a small refactor and storing cluster vitals data to InfluxDB.<\/p>\n\n\n\n\n\n\n\n

Refactoring the codebase a little<\/h2>\n\n\n\n

To start with I will refactor the codebase a little so it is more split up. By introducing a few new package layers it should be easier to work with.<\/p>\n\n\n\n

First we are going to refactor the code a little to move all classes that deal with secondary nodes into their own package. In addition the primaries will also be moved into their own area. These creates two new package locations:<\/p>\n\n\n\n

RaspberryPiCluster\/PrimaryNodes\nRaspberryPiCluster\/SecondaryNodes<\/pre>\n\n\n\n
Once that has been done I am tweaking the naming of some functions and files. The MachineInfo file has been renamed to NodeVitals. This is because the file allows the “vital information” about a node to be collected.<\/p>\n\n\n\n
Inside this the get_base_machine_info<\/code> has been renamed to get_node_baseinfo<\/code>. This function is now designed to return “base information” about the node which wont change often. This would include information such as RAM, CPU, and SWAP size.<\/p>\n\n\n\n
The temporal data (data that changes based on time) has been removed from this function and added to a new function current_node_vitals<\/code>. This is designed to be called to obtain the “current” state of the node and any changing “vital information”.<\/p>\n\n\n\n
In addition to this, the method now returns an object VitalsPayload<\/code> which is designed to hold information about the current vitals. By abstracting this data into an object it can have helper methods to load and format the payload to send to the primary.<\/p>\n\n\n\n
Reporting current node vitals for secondary nodes<\/h2>\n\n\n\n
Now we have refactored the node vital methods it can be used to report data to the primary. This will be helpful as the primary will then be able to determine how loaded each node is. This could be important when deciding what work should be scheduled on what node.<\/p>\n\n\n\n
Once the secondary has connected to the master it currently enters a loop where it will call perform_action<\/code> repeatedly. In the new code below it will check if its vitals have been sent in the last 60 seconds. If it hasn’t then it will first report its vitals to the primary and then perform some action.<\/p>\n\n\n
\nvitals_last_sent = 0\nwhile True:\n\tif (vitals_last_sent + 60) < time.time():\n\t\tlogger.info("Sending vitals to primary node")\n\t\tcurrent_vitals = get_current_node_vitals()\n\t\tself.connection_handler.send_message(current_vitals.get_flat_payload(), "vitals")\n\t\tvitals_last_sent = time.time()\n\tself.perform_action()\n<\/pre><\/div>\n\n\n
When the primary begins to distribute work it will be able to use this information to determine what should run.<\/p>\n\n\n\n
At the moment if perform_action<\/code> takes a long time this will not be updated very frequently so I will look at making this a second thread in the future.<\/p>\n\n\n\n
Storing Vital Data in InfluxDb<\/h2>\n\n\n\n
With the above changes the secondary is now regularly sending its vital data to the primary. We are going to store this data using InfluxDB so we can see it over time.<\/p>\n\n\n\n
To send data to InfluxDB I am using the official python library influxdb<\/code>. This allows connecting to, reading, and writing data into an Influx Database. Currently as the primary is the only node writing to Influx I will only install the Influx library on that node.<\/p>\n\n\n\n
To handle the connection to InfluxDB I have created a class hold the information.<\/p>\n\n\n
\nimport datetime\nfrom influxdb import InfluxDBClient\n\n\nclass RpiInfluxClient:\n    """ Simple helper class which will hold the details of the RPI Influx Client and make it a little easier to use """\n\n    def __init__(self, influxdb_host, influxdb_port, influxdb_database_prefix):\n\n        self.influxdb_host = influxdb_host\n        self.influxdb_port = influxdb_port\n        self.influxdb_database_prefix = influxdb_database_prefix\n        self.node_vitals_database_name = self.influxdb_database_prefix + "node_vitals"\n        self.node_name = None\n        # We only connect once we are ready to connect\n        self.influx_client = None\n\n    def add_node_name(self, node_name):\n        self.node_name = node_name\n\n    def connect(self):\n\n        """ Connects to the influx db Database using their client and sets up the databases needed (if needed)"""\n        self.influx_client = InfluxDBClient(self.influxdb_host, self.influxdb_port)\n        self.influx_client.create_database(self.node_vitals_database_name)\n        self.influx_client.create_retention_policy("node_vitals_year_rp", '365d', 3, database=self.node_vitals_database_name, default=True)\n\n<\/pre><\/div>\n\n\n
This is given all the information needed to create a connection to talk to the Influx database. The connect<\/code> method is used to begin the connection and set up the data structures in Influx. This configures a simple retention policy for the data in addition to the database to store the data.<\/p>\n\n\n\n
There are also two methods which are used to write data into InfluxDB. The first is a private method to write a datapoint to InfluxDB. This will be used to keep a standard format when writing data to InfluxDB.<\/p>\n\n\n\n
The second is a function to take vitals and store it to InfluxDB using the first. This splits apart the Vitals datastructure and sends the data one by one.<\/p>\n\n\n
\ndef _write_datapoint(self, measurement, values):\n\tif self.node_name is None:\n\t\t# We shouldnt ever encounter this but if we do I want it to fail hard so we can debug (not a good idea for production though)\n\t\traise Exception("Cannot write node value without node name")\n\n\tpoints_to_write = [\n\t\t{\n\t\t\t"measurement": measurement,\n\t\t\t"tags": {\n\t\t\t\t"node": self.node_name\n\t\t\t},\n\t\t\t"time": datetime.datetime.now().strftime("%Y-%m-%dT%H:%M:%SZ"),\n\t\t\t"fields": values\n\t\t}\n\t]\n\n\tself.influx_client.write_points(points_to_write, database=self.node_vitals_database_name)\n\ndef log_vitals(self, vitals):\n\t# TODO: Write these in one write_points API call rather than lots of smaller ones\n\tself._write_datapoint("cpu", {\n\t\t"frequency": vitals.cpu_frequency,\n\t\t"percentage": vitals.cpu_percentage\n\t})\n\tself._write_datapoint("ram", {\n\t\t"free": vitals.ram_free\n\t})\n\tself._write_datapoint("swap", {"free": vitals.swap_free})\n<\/pre><\/div>\n\n\n
Since the Python InfluxDB client allows for writing multiple values at once a future improvement is to make a single call to write_points<\/code>. I have left a note here in the comments to do this.<\/p>\n\n\n\n
Over time as more data is added into the vitals object, this will log more data if available. However currently this is just logging the basic data in the object.<\/p>\n\n\n\n
After these changes the primary is aware of the current node vitals of all connected nodes, and is storing this for tracking purposes.<\/p>\n\n\n\n
Summary of adding node monitoring<\/h2>\n\n\n\n
After some refactoring the secondary nodes now all report their data to the primary. This is logged by the primary into an InfluxDB which can be viewed to monitor the status of the cluster.<\/p>\n\n\n\n
Going forward more information will be added to keep an eye on the nodes.<\/p>\n\n\n\n
The full code is available on Github<\/a>, any comments or questions can be raised there as issues or posted below.<\/p>\n","protected":false},"excerpt":{"rendered":"
This post builds on my previous posts in the Raspberry Pi Cluster series by performing a small refactor and storing cluster vitals data to InfluxDB.<\/p>\n","protected":false},"author":1,"featured_media":2753,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_exactmetrics_skip_tracking":false,"_exactmetrics_sitenote_active":false,"_exactmetrics_sitenote_note":"","_exactmetrics_sitenote_category":0,"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"Today I am talking about monitoring the #RaspberryPi cluster using #InfluxDB","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[122],"tags":[102,184,198,37,101,393],"class_list":["post-2744","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-raspberry-pi-cluster","tag-distributed-computing","tag-python","tag-python-3","tag-raspberry-pi","tag-raspberry-pi-cluster","tag-raspberry-pi-os"],"wppr_data":{"cwp_meta_box_check":"No"},"jetpack_publicize_connections":[],"jetpack_featured_media_url":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2020\/12\/raspi_cluster_17_influxdb_posticon_OUTPUT.png?fit=1200%2C628&ssl=1","jetpack_shortlink":"https:\/\/wp.me\/p2toWX-Ig","jetpack_sharing_enabled":true,"jetpack-related-posts":[{"id":2680,"url":"https:\/\/chewett.co.uk\/blog\/2680\/raspberry-pi-cluster-node-16-python-3-codebase-refactor\/","url_meta":{"origin":2744,"position":0},"title":"Raspberry Pi Cluster Node \u2013 16 Python 3 Codebase Refactor","author":"Chewett","date":"October 24, 2020","format":false,"excerpt":"This post builds on\u00a0my previous posts in the Raspberry Pi Cluster series\u00a0by improving the codebase for Python 3. Moving to Python 3 Python 2 was marked end of life on January 1st, 2020 and therefore applications should ideally be no longer using Python 2. There will still be a lot\u2026","rel":"","context":"In "Raspberry Pi Cluster"","block_context":{"text":"Raspberry Pi Cluster","link":"https:\/\/chewett.co.uk\/blog\/category\/raspberry-pi-cluster\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2020\/10\/raspi_cluster_16_python3refactor_posticon_OUTPUT.png?fit=1200%2C628&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2020\/10\/raspi_cluster_16_python3refactor_posticon_OUTPUT.png?fit=1200%2C628&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2020\/10\/raspi_cluster_16_python3refactor_posticon_OUTPUT.png?fit=1200%2C628&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2020\/10\/raspi_cluster_16_python3refactor_posticon_OUTPUT.png?fit=1200%2C628&ssl=1&resize=700%2C400 2x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2020\/10\/raspi_cluster_16_python3refactor_posticon_OUTPUT.png?fit=1200%2C628&ssl=1&resize=1050%2C600 3x"},"classes":[]},{"id":2739,"url":"https:\/\/chewett.co.uk\/blog\/2739\/installing-influxdb-on-a-raspberry-pi\/","url_meta":{"origin":2744,"position":1},"title":"Installing InfluxDB on a Raspberry Pi","author":"Chewett","date":"December 5, 2020","format":false,"excerpt":"Today I am writing a quick describing how you can install InfluxDB on a Raspberry Pi What is InfluxDB? InfluxDB is a relatively popular time series database that allows storing metrics and data. It is specifically designed to store and operate on data that is bound by a time component.\u2026","rel":"","context":"In "Raspberry Pi Cluster"","block_context":{"text":"Raspberry Pi Cluster","link":"https:\/\/chewett.co.uk\/blog\/category\/raspberry-pi-cluster\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2020\/11\/raspberrypi_influxdb_posticon_OUTPUT.png?fit=1200%2C628&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2020\/11\/raspberrypi_influxdb_posticon_OUTPUT.png?fit=1200%2C628&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2020\/11\/raspberrypi_influxdb_posticon_OUTPUT.png?fit=1200%2C628&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2020\/11\/raspberrypi_influxdb_posticon_OUTPUT.png?fit=1200%2C628&ssl=1&resize=700%2C400 2x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2020\/11\/raspberrypi_influxdb_posticon_OUTPUT.png?fit=1200%2C628&ssl=1&resize=1050%2C600 3x"},"classes":[]},{"id":2780,"url":"https:\/\/chewett.co.uk\/blog\/2780\/raspberry-pi-cluster-node-18-raspberry-pi-temperature-monitoring\/","url_meta":{"origin":2744,"position":2},"title":"Raspberry Pi Cluster Node \u2013 18 Raspberry Pi Temperature Monitoring","author":"Chewett","date":"February 20, 2021","format":false,"excerpt":"This post builds on\u00a0my previous posts in the Raspberry Pi Cluster series\u00a0by starting to log temperature with the RaspberryPiVcgencmd Python module. Installing RaspberryPiVcgencmd RaspberryPiVcgencmd is a small python module aimed to control vcgencmd and allow programmatic access to it. This can be installed with the following command. python3 -m pip\u2026","rel":"","context":"In "Raspberry Pi Cluster"","block_context":{"text":"Raspberry Pi Cluster","link":"https:\/\/chewett.co.uk\/blog\/category\/raspberry-pi-cluster\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2021\/02\/raspi_cluster_18_cputemperature_posticon_OUTPUT.png?fit=1200%2C628&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2021\/02\/raspi_cluster_18_cputemperature_posticon_OUTPUT.png?fit=1200%2C628&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2021\/02\/raspi_cluster_18_cputemperature_posticon_OUTPUT.png?fit=1200%2C628&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2021\/02\/raspi_cluster_18_cputemperature_posticon_OUTPUT.png?fit=1200%2C628&ssl=1&resize=700%2C400 2x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2021\/02\/raspi_cluster_18_cputemperature_posticon_OUTPUT.png?fit=1200%2C628&ssl=1&resize=1050%2C600 3x"},"classes":[]},{"id":2674,"url":"https:\/\/chewett.co.uk\/blog\/2674\/moving-the-raspberry-pi-cluster-to-a-raspberry-pi-4b-4gb\/","url_meta":{"origin":2744,"position":3},"title":"Moving the Raspberry Pi Cluster to a Raspberry Pi 4B 4GB","author":"Chewett","date":"October 10, 2020","format":false,"excerpt":"Today I am talking about moving the Raspberry Pi Cluster primary node to a Raspberry Pi 4B 4GB model. I also discuss the improvements that the Raspberry Pi 4B has made over previous generations. Why move to a Raspberry Pi 4B Generally running the cluster does not require a large\u2026","rel":"","context":"In "Raspberry Pi Cluster"","block_context":{"text":"Raspberry Pi Cluster","link":"https:\/\/chewett.co.uk\/blog\/category\/raspberry-pi-cluster\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2020\/10\/raspberrypi_4_posticon_OUTPUT.png?fit=1200%2C628&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2020\/10\/raspberrypi_4_posticon_OUTPUT.png?fit=1200%2C628&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2020\/10\/raspberrypi_4_posticon_OUTPUT.png?fit=1200%2C628&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2020\/10\/raspberrypi_4_posticon_OUTPUT.png?fit=1200%2C628&ssl=1&resize=700%2C400 2x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2020\/10\/raspberrypi_4_posticon_OUTPUT.png?fit=1200%2C628&ssl=1&resize=1050%2C600 3x"},"classes":[]},{"id":881,"url":"https:\/\/chewett.co.uk\/blog\/881\/raspberry-pi-cluster-node-02-packaging-common-functionality\/","url_meta":{"origin":2744,"position":4},"title":"Raspberry Pi Cluster Node \u2013 02 Packaging common functionality","author":"Chewett","date":"November 29, 2017","format":false,"excerpt":"This post builds on the first step to create a Raspberry Pi Cluster node to package the common functionality that will be shared between master and clients. Packaging common functionality With the Raspberry Pi Cluster project there will be a number of things all nodes will do. This can lead\u2026","rel":"","context":"In "Raspberry Pi Cluster"","block_context":{"text":"Raspberry Pi Cluster","link":"https:\/\/chewett.co.uk\/blog\/category\/raspberry-pi-cluster\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2017\/11\/rpi_cluster_02_packaging.jpg?fit=800%2C800&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2017\/11\/rpi_cluster_02_packaging.jpg?fit=800%2C800&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2017\/11\/rpi_cluster_02_packaging.jpg?fit=800%2C800&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2017\/11\/rpi_cluster_02_packaging.jpg?fit=800%2C800&ssl=1&resize=700%2C400 2x"},"classes":[]},{"id":1839,"url":"https:\/\/chewett.co.uk\/blog\/1839\/raspberry-pi-cluster-node-08-slave-helper-functions\/","url_meta":{"origin":2744,"position":5},"title":"Raspberry Pi Cluster Node – 08 Slave Helper Functions","author":"Chewett","date":"December 19, 2018","format":false,"excerpt":"This post builds on\u00a0my previous posts in the Raspberry Pi Cluster series\u00a0by adding a number of slave helper functions.\u00a0 This update will begin the process of fully automating the slaves. Preparing the Slaves for Automation Before the slaves are ready to be fully automated there are a number of commands\u2026","rel":"","context":"In "Raspberry Pi Cluster"","block_context":{"text":"Raspberry Pi Cluster","link":"https:\/\/chewett.co.uk\/blog\/category\/raspberry-pi-cluster\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2018\/12\/rpi_cluster_08_slave_functions.jpg?fit=800%2C800&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2018\/12\/rpi_cluster_08_slave_functions.jpg?fit=800%2C800&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2018\/12\/rpi_cluster_08_slave_functions.jpg?fit=800%2C800&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2018\/12\/rpi_cluster_08_slave_functions.jpg?fit=800%2C800&ssl=1&resize=700%2C400 2x"},"classes":[]}],"_links":{"self":[{"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/posts\/2744","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/comments?post=2744"}],"version-history":[{"count":9,"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/posts\/2744\/revisions"}],"predecessor-version":[{"id":2756,"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/posts\/2744\/revisions\/2756"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/media\/2753"}],"wp:attachment":[{"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/media?parent=2744"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/categories?post=2744"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/tags?post=2744"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}