-
Notifications
You must be signed in to change notification settings - Fork 6
Configuration Manual XML
This page documents the deprecated XML format. New deployments should use the YAML format.
Etch is a tool for managing the configuration of Unix systems. It is primarily designed for the management of text configuration files, but is also capable of creating binary files, symbolic links and directories. For each file that etch is managing you can configure pre and post actions which can install related software packages, restart services or perform other actions.
Etch uses a client/server model. The server holds your full configuration repository and generates and sends clients their specific configuration. This allows you to have etch manage relatively private information like SSH keys, SSL keys, Kerberos keytabs, etc. Clients communicate with the server over HTTP/HTTPS. You'll generally want to use HTTPS in production to protect any privileged configuration you send to clients.
Etch allows decision making based on a set of attributes associated with the node it is running on. Many of these attributes (such as OS version and hostname) are determined automatically by etch using the Facter library. Etch provides the ability to assign systems to "groups". Groups can be used to describe services that a machine is expected to provide, or to group machines by organizational unit. Nodes can be assigned to groups via XML files, or etch can fetch them from an external source via a user-supplied script. This can be used to fetch groups from a central configuration database, like nVentory.
Etch is designed to automate the actions and processes that human administrators use to configure systems. As such etch does not try to hide the complexity or abstract any of the features and functions of the underlying operating system and applications that you are configuring. Tools which do so inherently limit you to the features and functionality that is exposed by their abstraction layer. Rather, etch allows you to automate and replicate a complex configuration repeatedly. If you are setting up 20 servers with a similar, complex configuration by hand you might just do the minimum necessary on each server to get the application working because that's all you have time to do. With etch you define that configuration and any necessary variations once and let etch replicate it for you, allowing you to spend the time to define the configuration more completely and correctly.
On a typical run etch is asked to go through the entire configuration repository and look for any files which need to be updated on the system. The way etch handles this is by going through the repository and generating each file in the repository in memory. The generated file is then compared to the file on the system. Only if they differ is the new file written out to disk, and any corresponding pre and post entries executed. Similar behavior is used for symlinks and directories. For example, with symlinks the destination of the link as configured in the repository is compared against the destination of the existing link (if any) and only if they differ is the link replaced.
Etch maintains a repository of the original system files, as well a log of all changes it has made to each file it is managing. If, when writing out a file, etch detects that it hasn't written that file out before it will save a copy of the existing original file and start a history log for the file. Future updates to the file are recorded in the history log.
Etch does per-file locking as it runs, so multiple copies of etch can run simultaneously with minimal interference.
--generate-all, or a filename
Etch requires that you give it either a specific file to generate or the --generate-all flag, which causes etch to scan through the entire configuration repository and generate all possible files.
--dry-run
Prints the contents of generated files to the screen instead of writing them out to disk.
--damp-run
Performs a dry run, but runs "setup" entries for files. Normally all setup/pre/post entries are ignored for a dry run. However, files with setup entries will generally fail to build if the setup entry hasn't been run. This option gives you a middle-ground between a dry run (which has no effect on the system, but will fail if setup entries need to run) and a normal run of etch.
--interactive
Causes etch to pause before making each change and prompt the user for confirmation. This option is very handy when running etch on a new machine or any other time when you are unsure of what etch will do and would like a chance to confirm its actions.
--full-file
Normally etch will print a diff to show what changes it will make to a file. This will cause etch to display the full new file contents instead.
--filename-only
Similar to the previous option, but in the opposite direction. Etch will only display the name of file to be changed.
--lock-force
Force the removal of any existing lockfiles. Normally only lockfile over 2 hours old are removed.
--debug
Prints lots of messages about what etch is doing.
--server
Point etch to an alternate server
--tag
Request a specific repository tag from the server
--test-base
Use an alternate local working directory (for use by test suite only)
The nodes.xml file must contain an entry for each node on which etch will run. This file is where you make group assignments.
Groups are used to describe services that a node should be providing.
Here's an example nodes.xml file that will be referred to throughout this manual:
<nodes>
<node name="server1.example.com">
<group>nfs_server</group>
<group>dhcp_server</group>
</node>
<node name="server2.example.com">
<group>mail_server</group>
</node>
</nodes>The nodegroups.xml file is an optional file where you can define node groups, the groups you refer to in nodes.xml, and create a hierarchy of node groups.
Here's an example nodegroups.xml file:
<nodegroups>
<nodegroup name="dns_server">
<child>dns_primary</child>
<child>dns_secondary</child>
</nodegroup>
<nodegroup name="http_server">
<child>intranet_server</child>
<child>yum_server</child>
</nodegroup>
</nodegroups>In this example, if a node were in the yum_server group in nodes.xml it would inherit the http_server group as well.
The etch source tree is arranged to mirror the directory structure of the client system. For example, the configuration related to /etc/resolv.conf can be found in the source/etc/resolv.conf/ directory. When called with the --generate-all flag etch does a scan through the source tree looking for directories containing config.xml files.
The directory path containing the config.xml file defines for etch the path of the file on which it is operating. If creating a file, etch will create any necessary directories on the system in order to hold the resulting file. For example, if etch is processing source/etc/foo/foo.conf/config.xml and decides that foo.conf should exist on this system then etch will create /etc and /etc/foo if necessary before writing out /etc/foo/foo.conf.
Each file that you expect etch to manage must be represented by a config.xml file in the source tree. These config.xml files define what action you want etch to take (file, symlink, directory, etc.), how to create that file, what permissions and ownership to assign to the file, any pre and post actions, etc.
The rest of this section on config.xml files will demonstrate various features using snippits of config.xml files. In order to give you some context of how those various snippits fit together to form a complete config.xml file here is a sample config.xml file which shows many of the available features. You can refer back to it while reading the rest of this section. (This example is heavily contrived in order to demonstrate certain features. A more realistic sendmail.cf example is available as part of our SampleConfigs repository.)
<config>
<revert/>
<depend>/etc/mail/aliases</depend>
<setup>
<exec operatingsystem="RedHat" group="mail_server">rpm --quiet -q m4 sendmail-cf || yum -y install m4 sendmail-cf</exec>
</setup>
<pre>
<exec operatingsystem="RedHat">rpm --quiet -q sendmail || yum -y install sendmail</exec>
</pre>
<file>
<owner>mailadmin</owner>
<group>mailadmins</group>
<perms>440</perms>
<warning_file>mywarning.txt</warning_file>
<source>
<script group="mail_server">sendmail.cf.script</script>
</source>
</file>
<post>
<exec operatingsystem="RedHat">chkconfig sendmail on</exec>
<exec operatingsystem="RedHat">service sendmail restart</exec>
<exec operatingsystem="Solaris">/etc/init.d/sendmail stop</exec>
<exec operatingsystem="Solaris">/etc/init.d/sendmail start</exec>
</post>
</config>The general structure of the config.xml files is defined by the config.dtd file. However, before etch validates the config.xml file against the DTD it performs a filtering process against any attributes in the XML file. Any XML elements with attributes which don't evaluate to true on the client are filtered out of the config.xml file.
Attributes come from the Facter library. On any system where you've installed facter you can run the facter executable to get a list of facts that the etch client will supply to the etch server and that you can reference in your configuration.
Best Practice Note: Controlling individual etch configurations by hostname should be avoided where possible. The preferred method is to assign a representative group to the node in nodes.xml and enable the appropriate configuration based on that group. This makes it easier to move those groups to another node, or to add additional nodes with the same groups to expand capacity. However, some configurations truly are tied to the hostname (SSL certificates and Kerberos keytabs, for example) and are appropriate uses of the hostname or fqdn attributes.
Attribute comparisons can be done in several forms. The most simple is straight string comparison.
<plain operatingsystem="Solaris">source_file</plain>You can perform numerical comparisons on numeric values, particularly operatingsystemrelease. In order to maintain compatibility with the XML standard the less-than symbol in < and <= comparisons must be encoded.
<plain operatingsystem="Solaris" operatingsystemrelease=">=5.8">source_file</plain>
<plain operatingsystem="Solaris" operatingsystemrelease="<=5.8">source_file</plain>You can use full Ruby regular expressions as well. (Which are basically the same as Perl regular expressions if you're more familiar with those.)
<plain operatingsystem="/RedHat|CentOS/">source_file</plain>Finally, you can negate any form of comparison by inserting an exclamation point as the first character after the opening quote.
<plain operatingsystem="!Solaris">source_file</plain>As you'll see later on, there are several places where etch can be configured to invoke a script in order to perform tasks which aren't easily expressed in XML. This includes complicated decision making, and runtime generation of configuration files. For reasons that will become clear in a second, those scripts must be written in ruby. This scripts are generally quite simple, so even if you don't know ruby you can pick up what you need from the examples here and sample configs. When generating files these scripts are not expected to write out the file themselves, they merely generate the file in memory and pass it back to etch.
The information available for attribute filtering is also made available in your ruby script for your convenience. The following variables will be defined in your script:
- @facts
- @groups
In addition, a few other useful variables are made available:
@file
The path to the file that is being generated.
@original_file
This variable will be described in the Files section.
@sourcebase
The path to the etch source repository, i.e. /etc/etchserver/trunk/source. This allows you to easily refer to other files in the etch repository without using relative paths.
@sitelibbase
Path to a directory in the repository that can be used for site-specific ruby libraries that can be referenced from multiple scripts.
@debug
Indicates the user ran the client in debug mode. You might use it in your script output additional information, which will pass through to the Rails production.log file.
The other interesting thing about etch scripts is that anything output to standard output or standard error is ignored. (Well, strictly speaking, it ends up in the Rails production.log file.) In order to communicate back to etch you must send any output to the @contents variable. This helps to ensure than accidental output doesn't end up in your generated file.
In order to make common tasks like outputting file snippits or whole files from your script easier, etch chdir's to the directory containing the current config.xml file, so you can refer to other files in that directory using relative pathnames.
@contents << "line of text\n"
@contents = IO.read('file.txt')For those interested in the details, these scripts are "executed" in the same ruby instance as etch using 'eval'. They are run in their own namespace so that only the variables we've mentioned are available to the script and to prevent the script from accidentally messing up the rest of etch. Any additional ruby modules you wish to use in your scripts could be installed via a "server_setup" entry, which will be described a bit later down.
Many of the settings allowed in config.xml files have defaults defined in a defaults.xml file. Examples include file permissions and ownership, and the definition of a default warning message.
The "depend" element allows you to specify one or more files on which the current file depends. Etch will generate those files before generating the current file. Etch tracks which files it has already generated in order to avoid unnecessarily generating the same file multiple times in a run. Etch also detects any circular dependencies and exits if it encounters such a situation. For example, if the config.xml for /etc/foo.conf contains the following elements then /etc/bar.conf and /etc/bar2.conf will be generated before generating foo.conf.
<config>
<depend>/etc/bar.conf</depend>
<depend>/etc/bar2.conf</depend>
</config>The "dependcommand" element allows you to specify one or more sets of ConfigurationCommands on which the current file depends. For example:
<config>
<dependcommand>linux_packages</dependcommand>
</config>The "setup", "pre", "test_before_post", "post", and "test" elements allow you to specify actions to take before or after writing out a configuration file. The only form of action currently supported is to execute commands, specified using "exec" elements.
Pre and post entries have the expected behavior of being executed just before or after the file/link/directory is written to disk. Pre entries are commonly used to install related software packages. Post entries are commonly used to restart associated services. Besides the standard "exec" element, in the post section you can also specify commands as "exec_once" or "exec_once_per_run". "exec_once" commands are only run the first time etch modifies a file. They will only be run that one time. "exec_once_per_run" commands are run once at the end of the etch run. Each unique command is only run once per etch run no matter how many times you specify it in different config.xml files. This is useful if you have multiple config files for a service but only want to restart that service once even if multiple related files are updated in one run. Note however that unlike other commands all "exec_once_per_run" commands are run as the last step in an etch run, rather than being run right after a file is updated. So any tests for a file should not depend on any "exec_once_per_run" commands having been run, as the tests will still be run right after the file is updated.
Because pre and post entries are only executed when etch knows it will be writing out a file, you can avoid unnecessary clutter in the config.xml file by not filtering the pre and post entries on the same attributes you used to decide if the file was relevant to the system. Consider the following example. The group attribute on the post entry is unnecessary because the file will only be created on systems with that group, and etch will never run the post section otherwise.
<config>
<file>
<source>
<plain group="dns_server">named.conf</plain>
</source>
</file>
<post>
<exec group="dns_server">/etc/init.d/named restart</exec>
</post>
</config>However, many pre and post actions are specific to particular operating systems, and you should filter appropriately for those cases. The examples below demonstrate that.
Setup entries are executed every time etch runs, before etch generates the configuration file. This feature exists to allow you to install software or otherwise take actions that are necessary to generate the configuration file. A common example is using a "setup" section to install m4 in order to generate sendmail.cf files. Also, if you will be using a script to modify the original version of a file you will need to install the package that contains that file via setup rather than pre.
Note that at the point at which the "setup" entry is executed etch does not yet know if the configuration file applies to this host. If, for example, you know that the configuration file will only be used on hosts with a particular group then you may wish to filter the "setup" entries against that group, otherwise they will be run everywhere.
Test entries are used to test that the changes made haven't broken anything. You can run tests before the post section with "test_before_post", these are good for testing the syntax of the generated file. For example, 'apachectl -t' will test the syntax of your Apache config files. Normal test commands are executed after any post commands in case the post commands are necessary to activate the file. If you specify test commands then etch will make a backup of the file before changing it, and roll back the file if the test fails. Here's an example of how to use a test command when generating a sendmail.cf.
<config>
<setup>
<exec operatingsystem="RedHat" group="mail_server">rpm --quiet -q m4 sendmail-cf || yum -y install m4 sendmail-cf</exec>
</setup>
<pre>
<exec operatingsystem="RedHat">rpm --quiet -q sendmail || up2date -i sendmail</exec>
</pre>
<post>
<exec operatingsystem="RedHat">service sendmail restart</exec>
<exec operatingsystem="Solaris">/etc/init.d/sendmail stop</exec>
<exec operatingsystem="Solaris">/etc/init.d/sendmail start</exec>
</post>
<test>
<exec>echo "3,0 root" | /usr/sbin/sendmail -bt</exec>
</test>
</config>A note on syntax: The config.xml files must be valid XML. Most notably that means any "&" or "<" characters must be escaped. So if you want to do something like
<exec>mkdir /foo && chown foo:bar /foo</exec>you have to put
<exec>mkdir /foo && chown foo:bar /foo</exec>Sorry, we know that's ugly but thankfully it isn't too common.
Etch is most commonly used to generate files of one form or another. These are usually text configuration files, but etch is capable of generating any form of file, including binary files.
Metadata
Etch allows you to specify the permissions and ownership to assign to files that etch creates. Note that default values are defined in the defaults.xml file mentioned above, so you only need to specify these for a file if you need something other than the default values.
Normally etch will only compare the metadata specified in config.xml with the file's settings if the contents of the file are also being managed by etch. However, in some cases you might want etch to manage the metadata settings even if you don't want to manage the file contents. For those cases there is the "always_manage_metadata" option.
<config>
<file>
<owner>nobody</owner>
<group>nogroup</group>
<perms>644</perms>
<always_manage_metadata/>
</file>
</config>Sources
Files can be generated one of three ways: from a plain file which is simply copied into place, from a ruby script which generates the file at runtime, or from an ERB (embedded ruby) template. To group the file source definitions they must be enclosed in a "source" element.
The plain file feature is quite simple. The specified file is used as is, with the possible exception of inserting the warning message. (Warning messages are described in the next section.) The keyword plain does not imply text, plain files can be binary.
<config>
<file>
<source>
<plain group="dns_primary">named.conf.primary</plain>
<plain group="dns_secondary">named.conf.secondary</plain>
</source>
</file>
</config>The "script" feature allows you to generate files dynamically, or to perform more complicated decision making than is capable of being expressed in XML. The basic features of etch-run scripts are described in the Scripts section above.
<config>
<file>
<source>
<script>resolv.conf.script</script>
</source>
</file>
</config>Best Practice Note: The standard practice is to name scripts after the file they will be generating, with the .script extension added to the end. As shown in the example, the script to generate /etc/resolv.conf would be named resolv.conf.script.
One of the variables mentioned in the above Scripts section is @original_file. Etch is designed to allow a system administrator to configure etch to perform the same actions that a system admin would do when configuring a system by hand. Some configuration files are very long. Apache's httpd.conf is one example. In most cases an SA would not create an httpd.conf from scratch, but would instead edit the default file provided with Apache and make only the necessary changes. In order to make it easy for you to edit files in an idempotent fashion etch provides the @original_file variable, which points to a copy of the file as it first appeared before etch touched it. Before etch modifies a file for the first time it makes a backup copy in the /var/etch/orig directory on the client and transmits the file contents to the server where the contents are stored in a file in /etc/etchserver/orig/. In your script @original_file will point to the server copy in /etc/etchserver/orig/. If the file did not originally exist then @original_file will point to an empty (zero-length) file.
Here's an example of how to append to the original file:
# Insert the original file
@contents << IO.read(@original_file)
# Append a line of text
@contents << "# Local additions:\n"
# And then append the contents of extrachunk
@contents << IO.read('extrachunk')And an example of editing the original file:
IO.foreach(@original_file) do |line|
# Uncomment the disable line
if line =~ /^#disable/
line.gsub(/^#/, '')
end
@contents << line << "\n"
endThe template feature works as a middle ground between plain files and scripts, and is particularly appropriate in cases where most of the file contents are static but some customization is needed. The ERB templating system is fairly simple. Anything enclosed in <% %> markers is evaluated as Ruby code, any output via puts, write, etc. is included in the result. If you add an equal sign after the opening percent sign like <%= %> then the result of evaluating the enclosed code is inserted into the result. The templates are evaluated in the same context as scripts, so the variables described above for scripts (@facts, @groups, etc.) are also available in your template.
<config>
<file>
<source>
<template>resolv.conf.template</template>
</source>
</file>
</config>Best Practice Note: The standard practice is to name templates after the file they will be generating, with the .template extension added to the end. As shown in the example, the template to generate /etc/resolv.conf would be named resolv.conf.template.
Here's an example of how a template might be used to generate resolv.conf:
search <%= @facts['domain'] %>
nameserver 192.168.1.1
nameserver 192.168.1.2
You can specify a source multiple times in a config.xml. This is necessary if multiple groups require the same file, for example. However, after attribute filtering your instructions to etch about how to generate the file must be consistent. So this is valid on a machine that belongs to both groups:
<plain group="foo_server">source_file</plain>
<plain group="bar_server">source_file</plain>But this isn't valid on the same machine. Which of the two files should etch use?
<plain group="foo_server">source_file</plain>
<plain group="bar_server">different_source_file</plain>Warning Message
Etch has a variety of options for inserting a warning message into generated files. Warning messages commonly serve to warn users that they are viewing a generated file and shouldn't edit it by hand. The warning message should be disabled when generating a binary file.
By default etch inserts the warning.txt file. The warning is inserted at the top of the generated file using # as the comment character. A blank line is added below the warning to improve readability. The "warning_file" element can be used to specify a different warning file, or to disable the warning message if specified empty. The "warning_on_second_line" inserts the warning message starting on the second line of the generated file, rather than the first line as is default. This is useful when generating scripts or other files where the first line of the file is meaningful. The "no_space_around_warning" element tells etch to not insert the blank line below the warning as is default. Some file formats, such as Solaris crontabs, don't allow blank lines. The "comment_open" and "comment_close" elements allow you to specify a comment system such as is used in the C programming language, where comments begin with /* and end with */. The "comment_line" element allows you to specify a comment string to be inserted at the start of each line of the warning. Examples include the very common # character, or the semicolon used in resolv.conf and DNS zone files.
<config>
<file>
<warning_file>mywarn.txt</warning_file>
<warning_on_second_line/>
<no_space_around_warning/>
<comment_open>/*</comment_open>
<comment_close>*/</comment_close>
<comment_line># </comment_line>
</file>
</config>Other Special Options
Normally etch will not write out a zero-length file. This is intended to prevent problems with things like miswritten scripts. However, you may occasionally want to create an empty file. The "allow_empty" element instructs etch to do so.
Similarly, etch will not normally overwrite a directory with a file or symbolic link. If you wish it to do so the "overwrite_directory" allows you to do so. The original directory and contents are packaged up with tar and stored in the /var/etch/orig directory.
<config>
<file>
<allow_empty/>
<overwrite_directory/>
</file>
</config>Etch is capable of making symbolic links. Within a "link" element there are two ways to provide etch with the destination to link to. The "dest" element can be used where attribute filtering is sufficient to make the decision. For more complicated situations you can use a "script" element. The script is run the same way as a script used as a file source. Whatever is fed to @contents is used as the link destination.
<config>
<link>
<dest operatingsystem="Solaris">/other/place</dest>
</link>
</config>
<config>
<link>
<script>dest.script</script>
</link>
</config>An example of a script that could be used with the link feature:
if @facts['operatingsystem'] == 'CentOS'
@contents << '/other/place'
endBy default etch will refuse to create a symlink to a nonexistent destination. If you need to create a symlink which you know will lead to a nonexistent location, you can pass in "allow_nonexistent_dest" flag within the "link" element.
Etch can be configured to create directories. As mentioned in the intro for the source tree, etch will automatically create subdirectories as necessary for files it is writing out. So you only need to specifically configure etch to create a directory if you need an empty directory created, or if you want to set special permissions or ownership on a directory. Within a "directory" element there are two ways to trigger etch to go ahead and create the directory. The "create" element can be used where attribute filtering is sufficient to enable or disable the creation. For more complicated situations you can use a "script" element. The script is run the same way as a script used as a file source. If the script feeds what ruby considers to be a true value to the @contents variable then etch will proceed with the creation.
<config>
<directory>
<owner>named</owner>
<group>named</group>
<perms>700</perms>
<create kernel="Linux" group="dns_server"/>
</directory>
</config>
<config>
<directory>
<script>decide.script</script>
</directory>
</config>An example of a script that could be used with the directory feature:
if @facts['kernel'] == 'Linux' && @groups.include?('dns_server')
@contents << 1
endEtch can be configured via the "delete" element to delete a file. This is commonly used to remove init script entries in /etc/rc2.d/ or /etc/rc3.d/ on Solaris in order to disable services. Within a "delete" element there are two ways to trigger etch to go ahead with deleting the file. The "proceed" element can be used where attribute filtering is sufficient to enable or disable the deletion. For more complicated situations you can use a "script" element. The script is run the same way as a script used as a file source. If the script feeds what ruby considers to be a true value to the @contents variable then etch will proceed with the deletion.
<config>
<delete>
<proceed operatingsystem="Solaris"/>
</delete>
</config>
<config>
<delete>
<script>decide.script</script>
</delete>
</config>An example of a script that could be used with the delete feature:
if @facts['operatingsystem'] == 'Solaris'
@contents << 1
endIf you decide you no longer want etch to maintain a file you can simply remove the configuration from your repository. But if you want etch to undo its work and put the original file back you can use the revert feature. If the first element in your config.xml file is the "revert" element etch will restore the original state of the file.
<config>
<revert/>
</config>