Rollout Manual

I can't stand it when people release open source software, expect people to use it, and don't provide any sort of documentation. I'm not the sort of person who enjoys writing doco, but I do think it's required.

So here, in it's entirety: the Rollout Manual

Rollout

Introduction

Rollout is a system developed to manage UNIX servers. It is primarily focused towards Linux, but could be adapted to Solaris, HP-UX, AIX, etc.

It is written purely in Perl, and the configuration is also a Perl source file. Some Perl knowledge is required to edit the configuration, but copy-and-paste will probably suffice.

Overview

System Administration can generally be reduced to a set of steps, which must be completed in order. Some example steps include:

• Copy files, create symlinks


• Edit configuration files


• Start / stop services


• Add / modify users & groups


• Install applications

The idea behind Rollout is to automate all these steps in a configurable way. Rollout is by design indempotent - it can be run many times on a single server and produce the same results.

Running Rollout

Rollout is self-installing on servers. The following command line will install
rollout for the first time:

URL=http://rollout.localdomain; wget -O- $URL/rollout | perl - -s -u $URL

After it's installed, it can be run just by calling rollout. You must however give a comment on the rollout command line. This should be a short reason for running Rollout.

The first run uses the -s flag, which runs rollout in safe mode. In this mode, all the steps are run but no changes are actually made. This is used to check that nothing bad will happen when it's run for real.

After the safe mode run has been checked, it can be run again without the -s flag, and all steps will be applied.

NOTE: The first run requires the Net::HTTP perl module to already be installed. This is part of the libwww-perl package in Debian based distributions.

Command-line Arguments

The following command line arguments are available:

Full	Short	Description
verbose	v	Increase verbosity
quiet	q	Don't print anything except fatal errors
safe_mode	s	Show what will be changed, but don't actually do anything
url	u	Set the base Rollout HTTP url. This defaults to http://rollout.domain. This option is saved in the rollout script, so only needs to be specified once.
skip_step	k	Specify a step to be skipped, may be specified multiple times. e.g. -k network -k sysctl
hostname	h	Rollout configuration for a different host. If -f network is provided, the hostname will be changed to match the host supplied
force	f	Force a dangerous step to be run. e.g. -f network

Logging

Every Rollout run that is run without the -s flag will be logged by the 99-complete step. The default is to make a HTTP request to the rollout server and pass it the log. The sample complete.php script will receive the log and do nothing with it.

Configuration

The entire configuration for Rollout is stored in rollout.cfg. This file is a Perl source file, so it can be checked for correctness by running perl -c rollout.cfg.

A major design concept in the rollout configuration is inheritance. Since many devices will have similar configurations, they should inherit as much as possible using the ISA configuration item. Each class in the ISA section is also specified in the same config, and they can inherit from other classes. It's even possible to have a device directly inherit another device's configuration, but this would include everything, even network interfaces.

The only thing that needs to be changed in the file is the %m hash, which contains configuration for everything. A sample device configuration looks like this:

kickstart => {
  ISA => { RHEL4_i386 => 1, Web_Server => 1 }, # Inherit the RHEL4_i386 and Web_Server classes
  interfaces => {
    eth0 => {
      dhcp => 'kickstart',                     # eth0 gets its IP by DHCP
      mac => '00:50:8B:E8:A0:6D',
      primary => 1,
    },
    eth1 => {
      ip => '10.0.0.1',                        # eth1 is statically defined on its own network
      netmask => '255.255.255.0',
      mac => '00:D0:B7:E3:5A:F4',
    },
  },
  nfs_share => {
    "/kickstart" => {                          # Share the /kickstart directory by NFS
      "10.0.0.0/24" => "ro,no_root_squash",    # Allow the private LAN to access the nfs share
    },
  },
  firewall_allow => [;
    '# Rollout HTTP', # This comment is inserted directly into the rules list
    any => '8140',    # Allow any host to connect to tcp port 8140
    admin => '80',    # Allow the admin network to connect to tcp port 80
  ],
  packages => [;
    'dhcp',           # Make sure the dhcp and rpm-devel packages are installed
    'rpm-devel',
  ],
},

Configuration Items

The following items are available for use in rollout.cfg:

Item	Type	Description
apt_base	string	Specifies a URL that will be added to the APT sources.list for local packages
crontab	hash	A hash of cron.d files that will be created, with the entries for each file
cvs	cvs	Specifies which modules will be checked out of CVS, and where
deb_options	list	Allows preconfiguring deb packages before they are installed. Expert Option
dir_check	list	Creates directories and forces ownership & permissions
file_append	list	Appends lines to text files, replacing any lines that match. Can also run a command if the file is changed
file_attr	hash	Modifies owner, group and permissions on files
file_extract	hash	Downloads a tarball and extracts it to a directory
file_install	hash	Downloads a file and places it somewhere. Can also run a command if the file is changed
file_modify	list	Modifies existing files by applying regular expressions. Can also run a command if the file is changed
firewall_allow	list	Allows incoming ports in the firewall
firewall_append	list	Appends verbatim lines to the iptables configuration file
firewall_deny	list	Denies incoming ports in the firewall
group	hash	Ensures a system group exists
hosts_append	list	Appends a given line to the /etc/hosts file
immutable_file	list	Marks a certain file as immutable. This file will never be touched by any other steps
interfaces	hash	Defines configuration for the network interfaces
ISA	hash	Specifies a list of classes that this device or class inherits configuration from
nameservers	list	Specifies the default nameservers in /etc/resolv.conf
network	string	Marks the device or class as part of a network, for the purposes of firewalls and host files
nfs_mount	hash	Mounts a NFS share locally
nfs_share	hash	Shares a local directory by NFS
packages	list	Provides a list of packages to be installed. These can be debs or rpms
port_check	list	Adds elements to /etc/services
service	hash	Ensures that system services are running/stopped and should be started/stopped on boot
skip_steps	list	Mark a list of steps that will never be executed for this device
sudo	hash	Allow users to run commands with sudo
ssh_keys_add	list	Adds the list of keys to root's authorized_keys file
symlink_check	hash	Ensures a symlink exists and its destination is correct
sysctl	hash	Adds the items to /etc/sysctl.conf and applies them. These are for kernel configuration
user	string	Ensures each user exists, has the correct password and is in the correct groups

Sample Run

The following is pasted directly from the terminal after running Rollout on a server:

01-setup
02-network
  Installing /etc/network/interfaces from text
  Modified /etc/resolv.conf by appending options timeout:5 attempts:2
  Network config is considered dangerous, changes won't be applied unless you specify the "-f network" argument.
03-hosts
  Modified /etc/hosts by appending 172.16.40.54 mulberry
  Modified /etc/hosts by appending 172.16.40.184 kickstart
05-users
06-ssh_keys
  Installing /root/.ssh/authorized_keys from text
  Changing mode of /root/.ssh/authorized_keys to 644
15-packages
  Installing packages lshw
16-gpg
20-sysctl
  Modified /etc/sysctl.conf by appending net.ipv4.conf.all.log_martians = 1
  Modified /etc/sysctl.conf by appending net.ipv4.conf.all.accept_redirects = 0
  Modified /etc/sysctl.conf by appending net.ipv4.conf.all.send_redirects = 0
  Modified /etc/sysctl.conf by appending net.ipv4.conf.all.rp_filter = 1
  Modified /etc/sysctl.conf by appending net.ipv4.icmp_echo_ignore_broadcasts = 0
  Modified /etc/sysctl.conf by appending net.ipv4.tcp_syncookies = 1
  Modified /etc/sysctl.conf by appending net.ipv4.conf.all.accept_source_route = 0
21-cvs
24-dir_check
24-file_extract
25-file_install
  Installing /etc/pam_ldap.conf from http://rollout.domain/conf/pam_ldap.conf
  Installing /etc/libnss-ldap.conf from http://rollout.domain/conf/libnss-ldap.conf
  Changing mode of /etc/libnss-ldap.conf to 600
  Installing /etc/ntp.conf from http://rollout.domain/conf/ntp.conf
  Installing /etc/resolv.conf from http://rollout.domain/conf/resolv.conf
  Changing target of symlink /usr/local/bin/vis to /usr/local/bin/viw
27-port_check
70-nfs
80-file_append
80-file_modify
  Modified /etc/nsswitch.conf with s/^passwd:.*/passwd:  compat ldap/, s/^group:.*/group:  compat ldap/
80-file_uncomment
85-cron
90-services
99-complete

Steps

Step	Description
01-setup	Installs rollout locally Downloads and parses the rollout.cfg configuration file
02-network	Checks the interfaces configuration item This is a dangerous step and will always be run in safe mode unless -f network is specified On RedHat: Creates a file called /etc/sysconfig/network-interfaces/ifcfg-* for each interface. On Debian / Ubuntu: Creates the single file /etc/network/interfaces containing configuration for every interface. Sets the device's hostname if -h _hostname_ is specified Applies the skip_steps configuration item
03-hosts	Adds addresses to /etc/hosts for every device in this device's network. Uses the hosts_append and network configuration items.
05-users	Creates and modifies users and groups. Uses the user and group configuration items. This will set a user's password on creation, but will not reset it if changed locally.
06-ssh_keys	Creates and updates ~/.ssh/authorized_keys for every user, including root. Adds ssh keys for each user based on the user => { ssh_keys => [] } configuration item. Adds ssh keys for root based on the ssh_keys_add configuration item. The ssh keys are taken from the rollout:conf/authorized_keys file based on comment.
15-packages	Installs and removes packages as specified in the packages config item. Applies deb configuration items as specified in the deb_options config item. Adds apt_base items to the apt sources.list
16-gpg	Adds required ssh public keys to root's public keyring
20-sysctl	Adds all items in sysctl the config item to /etc/sysctl.conf and applies them.
21-cvs	Checks out any cvs modules as specified in the cvs config item. If the directory already exists, it is checked for CVS files. If they don't exist already, it is a fatal error. If the directory is an existing checkout, the directory will be updated by cvs.
24-dir_check	Creates any directories as specified in the dir_check config item. Each directory is checked for owner and permissions.
24-file_extract	Extracts any tarballs given in the file_extract config item.
25-file_install	Installs individual files specified in the file_install config item. Optionally runs a command if the file has changed. Creates symlinks specified in the symlink_check config item and ensures the destination is correct.
27-port_check	Adds items in the port_check configuration item to /etc/services
60-sudo	Creates /etc/sudoers allowing users to run commands as root, with the configuration in the sudo item.
70-nfs	Configures both nfs client and server, as specified in the nfs_mount and nfs_share config items. Forces the required packages to be installed to support a nfs client and/or server. Creates directories for any nfs client mountpoints. Configures nfs, portmap, sunrpc, etc to run on predefined ports.
80-file_append	Appends lines to existing configuration files as specified in the file_append config item. Optionally runs a command if the file has been changed.
80-file_modify	Applies regular expression changes to files given in the file_modify config item. Optionally runs a command if the file has been changed.
80-file_uncomment	Uncomments out lines in a configuration files matching regular expressions in the file_uncomment config item. Lines are uncommented by removing a leading #
81-file_attr	Change the owner, group and permissions of files as controlled by the file_attr config item.
85-cron	Creates a cron.d file for each item specified in the crontab config item. The created file will contain only the items specified in the config.
90-services	Starts and stops services specified in the services config item.
95-iptables	Creates the iptables configuration file. The location of this file varies from system to system. The format of the file is what is required by iptables-restore. The file is made up of default rules, firewall_deny rules, firewall_append lines and firewall_allow rules, in that order. If the config file has changed, runs iptables-restore on it to load the new rules.
99-complete	Log the run in /var/log/rollout.log Send a copy of the log to the rollout web server.