The shutdown command is going to be used when I want to take the nodes offline for work. This will be rarely used but will be important for maintenance.
Once called a node should fully shut down so that its power can be removed. The node will be manually booted once maintenance is finished. Again it is assumed that it will automatically connect to the mater once booted.
For now it is assumed all nodes are running apt based package systems. This means that to update them we can run apt-get update, followed by apt-get upgrade.
The output of these two commands are returned from this function. This was not done for the previous two functions because rebooting and shutting down will not return any data, as the node will have turned off.
Get Node Time
It is important to make sure all nodes have the correct time. This is to ensure that the cluster is all running the same time. When messages are encrypted the time will become more important. This reasoning for this will be discussed in future blogposts.
Get Node Codebase Revision
#Note: This is not nesscarily the revision of the code currently running
current_rev = subprocess.check_output(["git", "rev-parse", "HEAD"])
To start with it is assumed that the nodes will synchronise code using a shared git repository. This means its important to know what revision each node is running.
This command will call git to determine the current revision of the code. This however will only determine the current revision in the directory which may be different from the revision that is running.
This problem will be resolved and discussed in detail in a future blogpost.
Since git does not have sequential revision numbers like SVN or Perforce you cannot quickly know whether one revision is ahead of another. However by using git show you can obtain the time of the commit.
This will be used by the master to determine which node has the later codebase, and update other nodes not on this version. This uses the previously defined function get_node_codebase_revision to get the current codebase revision.
The final command is to update the node codebase using git. As mentioned earlier it is assumed that all nodes are deployed from a git repository.
By using the command git pull the code deployed on the node will be updated from the master git repository. It is assumed here that the node will be configured so no password is needed to update the git repository.
Therefore to update the nodes we should just be able to run git pull and the node will successfully be updated. This will be used in conjunction with rebooting the node to update it to the latest version.
Summary of newly added functions
Now we have a number of functions available to the Nodes to able to control them. At this stage they are still not ready to run master defined scripts but these will form the basis of the automation.
The next tutorial will focus on converting the master script to accept any number of slave nodes.
The full code is available on Github, any comments or questions can be raised there as issues or posted below.