Updating running nodes is the process of provisioning a software im- age to a node without rebooting the node. A configuration setting called excludelistupdate (section 6.6.1), is used by the imageupdate command (section 6.6.2) when carrying out an update to a running node.
The converse of the imageupdate command is the grabimage com- mand, which grabs a node state and creates a software image from it. The grabimage command is discussed in section 9.5.2.
6.6.1 Updating Running Nodes: Configuration With
excludelistupdate
Changes made to the contents of the head node’s software image for nodes become part of the provisioning system according to its housekeep- ing system (section 6.2.4). The image is then installed from the provision- ing system onto a regular node when it (the regular node) reboots via a provisioning request (section 6.4.7).
However, updating a running node with the latest changes from the software image is also possible without rebooting it to re-install the im- age. Such an update can be requested usingcmsh or cmgui, and is queued and delegated to a provisioning node just like an ordinary provisioning request.
Like the provisioning requests done at the time of install it uses an ex- clude list, with the same structure and rsync patterns syntax to those detailed in section 6.4.7. This exclude list is however defined in the excludelistupdate property of the node’s category. To distinguish the intention behind the exclude lists, the administrator should note that it is theexcludelistupdate that is being discussed here, in contrast with the excludelistsyncinstall/excludelistfullinstall from section 6.4.7.
So, theexcludelistupdate property settings here concern an update to a running system, while the other two from section 6.4.7 are about an install during node start-up.
Similar to thesync case of section 6.4.7, which uses excludelistsync, theupdate case in this section uses excludelistupdate.
The running node update type of synchronization uses the excludelistupdate property to specify what files and directories to exclude from consideration when copying parts of the filesys- tem from a known good software image to the node. The excludelistupdateinstall property is in the form of a list of exclusions, or more accurately in the form of two sub-lists.
The contents of the sub-lists specify the parts of the filesystem that should be retained on the node duringupdate synchronization. The in- tention behind this is to have the node synchronize quickly, updating only the files from the image to the node that need updating due to the changes on the software image, and otherwise keeping files that are already on the node hard disk unchanged. The contents of the sub-lists are thus the files and directories that are expected to be present in a running node.
Anything already on the node that matches the content of these sub- lists is not overwritten by image content during anexcludelistupdate update. However, image content that is not on the node is copied over to the node only for items matching the first sub-list. The remaining files and directories on the node, that is, the ones that are not in the sub-lists, lose their original contents, and are copied over from the software image. A samplecmsh one-liner which opens up a text editor in a category to set the exclude list for updates for is:
The exclude list for updates can be edited incmgui as indicated in figure 6.22.
Figure 6.22: Setting up exclude lists withcmgui for node updates In addition to the paths excluded using theexcludelistupdate prop- erty, the provisioning system automatically adds any NFS, Lustre, FUSE, PanFS, FhGFS, GlusterFS, and GPFS imported file systems on the node. If this were not done, all data on these filesystems would be wiped since they are not part of the software image.
6.6.2 Updating Running Nodes
Updating Running Nodes: Withcmsh Using imageupdate
Using a defined excludelistupdate property (section 6.6.1), the imageupdate command of cmsh is used to start an update on a running node:
Example
[bright52->device]% imageupdate -n node001
Performing dry run (use synclog command to review result, then pass -w \ to perform real update)...
Tue Jan 11 12:13:33 2011 bright52: Provisioning started on node node001 [bright52->device]% imageupdate -n node001: image update in progress ... [bright52->device]%
Tue Jan 11 12:13:44 2011 bright52: Provisioning completed on node node0\ 01
By default the imageupdate command performs a dry run, which means no data on the node is actually written. Before passing the “-w” switch, it is recommended to analyze the rsync output using thesynclog command (section 6.4.7).
If the user is now satisfied with the changes that are to be made, the imageupdate command is invoked again with the “-w” switch to imple- ment them:
Example
[bright52->device]% imageupdate -n node001 -w Provisioning started on node node001
node001: image update in progress ...
Updating Running Nodes: Withcmgui Using The “Update node” Button Incmgui an image update can be carried out by selecting the specific node or specific category from the resource tree. Then, within the tasks tabbed pane that opens up, the “Update node” button is clicked (figure 6.23). This opens up a dialog which has a dry-run checkbox marked by default.
Figure 6.23: Updating A Running Node Withcmgui
The dry-run can be reviewed by clicking on the “Provisioning Log” button further down the same tabbed pane. The update can then be done again with the dry-run check mark off to actually implement the update. Updating Running Nodes: Considerations
Updating an image via cmsh or cmgui automatically updates the pro- visioners first via the updateprovisioners command (section 6.2.4) if the provisioners have not been updated in the last 5 minutes. So, if there has been an update within the last 5 minutes, then provisioners do not get an updated image when doing the updates. Running the updateprovisioners command just before running the imageupdate com- mand therefore usually makes sense.
Also, when updating services, the services on the nodes may not restart since theinit process may not notice the replacement.
For these reasons, especially for more extensive changes, it can be safer for the administrator to simply reboot the nodes instead of using imageupdate to provision the images to the nodes. A reboot ensures that a node will have the latest image and that services should start up as in- tended.
Updating Running Nodes: Pre- And Post-update Scripts
Two further scripts associated with the imageupdate command and its cmgui equivalent may run as part the execution of the update. These are located at/cm/images/default-image/cm/local/apps/cmd/scripts/ in the default software image:
• The imageupdate_initialize script runs before the software image starts updating. If the imageupdate_initialize script exits with non- zero, then the image does not update
• The imageupdate_finalize script runs after animageupdate command is run on that node, and right after the software image has updated. These differ from theinitialize (section 6.4.5) and finalize (sec- tion 6.4.11) scripts because they run on nodes that are fully up rather than
on nodes that are booting, so they are able to access a fully running sys- tem, and because they run on all nodes using that image rather than also being configurable for individual nodes within that image.