Hardening OpenBSD Internet Servers
Immutable Files, Securelevels and Mount Options

BSD systems include some options such as immutable files and securelevels that require root to have local access to make certain types of changes. Mount options, including read only filesystems may hinder intruders. These are discussed here.

• Read Only Filesystems
  
• Noexec, Nosuid and Nodev Mount Options
  
• Costs of Non Standard Mount Options
  
• Chflags, Immutable Files and Security Levels
  

Read Only Filesystems

Another obstacle to place in the way of intruders and unauthorized local users is to mount some filesystems as read only. Prime candidates for this would be / and /usr. Once a system configuration is stable, there should be very few if any files that need to change frequently in these locations. These could be mounted read only late in the boot process or even periodically via cron. Having these read only might save a few seconds rebooting after an unexpected shutdown such as a power failure.

The real reason for doing this is that it's going to break rootkits and other tools crackers use for covering their tracks after root access is gained. It won't stop any knowledgeable intruder but it may slow them down. A mounted filesystem can be made read only by "mount -ur /filesystem_mount_point" and returned to writable status just as easily with "mount -uw /filesystem_mount_point".

To cover their tracks after gaining root access, intruders have to be able replace executables with Trojans and modify system configuration files. A read only filesystem will stop this until it is made writable. Much cracking is done with the aid of scripts. There is a big difference between running a script and having root status with tracks covered in seconds and running it and seeing error messages. At that point, the intruder has to determine the cause of the errors, know or figure out the syntax of the command to make filesystems writable and manually execute these commands or add them to the scripts. It could be the difference between being caught and getting away with an intrusion. Setting an alarm if a read only filesystem became writable could even be made part of an intrusion detection system.

Anyone capable of writing cracking tools will have no difficulty dealing with read only filesystems. Because they are very much the exception though, the author may not think to include the necessary commands until a read only filesystem is encountered. Someone using tools created by others, which are probably the large majority of would be intruders, may well not know how to deal with the problem.

Noexec, Nosuid and Nodev Mount Options

Closely related to read only filesystems are the noexec, nosuid and nodev mount options. Noexec prevents executable files from being executed. This includes scripts as well as binary files but won't prevent a script from being interpreted by the current shell with the ".  " prefix. Normally, you'd not expect to find legitimate executables in the /tmp and /var filesystems but intruders might try to hide executables there. It's common to find executables in /home but should be very rare to find SUID programs in /home. Thus it makes sense for /home, /tmp and /var to be mounted nosuid. Nodev prevents a device file from being acknowledged. The only normal place for device files is /dev so it makes sense and does not appear to have adverse consequences to mount all filesystems but / as nodev.

The following commands at the end of /etc/rc.local would set all the mount options discussed above:

mount -ur /
mount -ur -o nodev /usr
mount -uw -o nodev,nosuid /home
mount -uw -o nodev,nosuid,noexec /var
mount -uw -o nodev,nosuid,noexec /tmp

Costs of Non Standard Mount Options

As with all security related measures, you need to ask whether any hoped for advantage is worth the cost. There is no question there is a cost; since I've been setting / and /usr to read only on some of my systems, I've lost track of how many times a read only error has stopped me from saving changes. It's usually only a few seconds to deal with this. Either the file system is made writable and minor changes redone or the file is saved to a writable filesystem and subsequently moved into place after making the original target writable. The read only filesystems have created a nuisance factor with no clear security benefit.

I recently installed the Analog, web statistics package on my OpenBSD web server. The read only /usr filesystem prevented analog from being able to use its DNS lock and cache files which it wants to locate in the analog directory which is a subdirectory of /usr/local. A quick search of the Analog documentation shows no obvious configuration option to relocate the DNS files to another location such as where the web logs or analog output is placed. While I may keep read only / and /usr filesystems on a firewall, a read only /usr on a web server with Analog simply isn't worth the effort. I have little doubt that each of the other suggested mount settings will break some other software.

Trying settings such as these on a test machine that doesn't really do anything may not turn up any but the most obvious problems such as the inability to access a device. Determining even this much would require attempting to access the device while the settings were in effect. I decided to try these mount settings on my live OpenBSD web server to see what if any problems would show up.

I immediately tried accessing the web site. Web pages were being served normally so the primary function of the machine did not appear to be affected and I left the settings in place. I don't remember if I thought to try CGI scripts at that time. It took a day to find a problem. It turns out the noexec setting on /var breaks the CGI version of Analog. I didn't learn this until I tried to get some immediate web statistics that the CGI version of Analog provides.

My production web site runs out of /home subdirectories but the administrative site uses the default install location for Apache on OpenBSD. These place the cgi-bin directory inside of /var/www and thus no CGI scripts in the default location will run with /var mounted noexec. The Analog CGI interface runs from this directory.

This shows the largest problem with using unusual mount settings such as those suggested above for security purposes. It's not that they break some essential, regularly used software which should be quickly obvious and easy to fix. It's that some infrequently used software may be affected. By the time the problem shows itself, the changes that caused it may be a dim memory or the person encountering the problem may not even know of the configuration changes that are responsible.

While working on the hardening process for OpenBSD 2.9, I became aware of another problem with read only / (root) filesystems. I set the read only property early while configuring my first 2.9 server and had plenty of opportunity to observe a series of error messages caused by mounting / read only. The messages were as follows:

date time hostname login: /etc/fbtab: chmod (/dev/console) Read only file system
date time hostname login: /etc/fbtab: chmod (/dev/wskbd0) Read only file system
date time hostname login: /etc/fbtab: chmod (/dev/wsmouse0) Read only file system

Each message was repeated twice and their length caused them to wrap. It was not easy to ignore these messages which filled much of the screen following some logins. They did not appear if / was returned to it's normal writable status. I do not know what practical consequences the failure of chmod had. It was not responsible for the keyboard lockups described in Custom Kernel: Large Unwanted Change in 2.9 as these lockups definitely occured when no file system was set to read only.

A similar /dev/console, but not /dev/wskbd0 or /dev/wsmouse0, error message appeared in OpenBSD 2.8. Because I did not set the read only option until very late in the 2.8 configuration process, rarely logged in at the console and the message was much less prominent, it either did not register or I never found the time to look into it.

There were three choices once I was aware of the messages in 2.9. I could ignore them as there didn't appear to be obvious consequences. I'm not comfortable deliberately ignoring system error messages that I am aware of. I could attempt to understand what the system was trying to do and accomplish that by some other means or try to surpress the error message if I decided they were not indicative of an important condition. Either way meant more time and more system customizations.

It's clear that if all the customizations discussed throughout the entire Hardening OpenBSD section are implemented, that I've past the point of diminishing returns. With the relatively small amount of well backed up, nearly static data on my OpenBSD systems, I can build a new OpenBSD system and restore my specific functions in much less time than it takes to develop an updated hardening process for each new OpenBSD release. Other systems with more valuable and more changing data that might be harder to rebuild might be worth more up-front security effort. In my case, I need to draw the line and chose the third option: eliminate the error messages by eliminating the cause, the non standard read only / file system.

Chflags, Immutable Files and Security Levels

A couple readers brought chflags, immutable files and security levels to my attention but I was initially skeptical of the potential. I'd heard of immutable or unchangeable files but also that once made immutable, a file could only be changed by booting into single user mode. One of UNIX's strengths is the ability to make system changes without rebooting and thus disrupting the services it provides. Losing this flexibility didn't seem worth the limited security enhancements, especially after I'd already found ways to control access to many of the tools necessary to make the changes in the first place.

It turns out, that at least for a firewall, you can make key system files immutable and or use securelevel 2 to prevent any changes to the firewall and NAT rules. You can do this and from the local console only, make subsequent changes to the firewall and NAT rules or unset the immutable flag, without interrupting the firewall function. You can do this because firewall and NAT functions are built into the kernel and on BSD systems you can actually switch back and forth between single user and multi user modes without rebooting the computer or interrupting the firewall and NAT functions in the kernel. I was skeptical until I actually saw it work multiple times.

Chflags is the command that makes a file immutable or unchangeable. It has several options including system and user append only flags but the one of particular interest is the schg or system immutable flag available only to the super user. Once this flag is set on a file, no one, including the super user, can make changes to the file until the schg flag is removed. If this flag could be turned off as easily as it is set, then it would be no more interesting than read only/writable filesystems, that can be changed at will by root.

The system immutable flag, however, works in conjunction with the security level so that it is much more difficult to turn the flag off than on. OpenBSD has four security levels; see the securelevel (7) man page for details. The lowest is -1 or "permanently insecure"; this allows a multi user mode as insecure as the standard single user mode and so is of no interest to those concerned with tightening security. 0 or insecure is the security level during booting and in single user mode. It allows the single user (root or the super user) to do some things that cannot be done in 1 or secure mode, the standard multi user mode. The most interesting, for our purposes, is to turn off the system immutable flag. In the normal multi user, secure mode, the system immutable and append only flags can be set on files but not unset. Security level 2 or highly secure mode, prevents some low level disk access and debugging related actions, prevents the system clock from being set back and prevents firewall and NAT rules from being changed.

Once the security level is at 1 or 2 it can only be lowered via the init program. Root can manually raise the security level from 1 to 2 via the sysctl program or /etc/rc.securelevel can be edited so that the security level will be 2 at the completion of the boot process.

Init can be used to change to from security level 1 or 2 by returning the system to level 0 and single user mode. The init(8) man page gives the command as "kill -s TERM 1". This never worked for me but "kill -15 1" worked quite reliably and "kill -TERM 1" works as well. If this is executed remotely, the network connection is broken as soon as enter is pressed.

If this is done from the local console or you're watching the local console when this is done remotely, you see messages from exiting daemons. The messages should vary depending on what has been running. They end with the date and time and "init: kernel security level changed from 1 to 0". This is followed by the "Enter pathname of shell or RETURN for sh:" prompt that should be familiar from the install process or floppy boot sequence. If you press enter, you are prompted for "Terminal type?" until you give a valid type. I used vt100, vt220 and pcvt25. If you accepted sh as your shell, you are left at the "#" single user prompt.

If you doubt that you're in single user mode, "ps ax" shows only three processes: /sbin/init, -sh (sh) and ps -ax. What's most interesting about this is if the computer that is switching between multi user and single user modes is a firewall running IP Filter or now Packet Filter, you can maintain network connections through it with no apparent effect, i.e. the kernel never reloads or stops performing the firewall and NAT processes that are part of it. This works on both bridging firewalls and routing firewalls with NAT.

I was able to run both web and telnet sessions through an OpenBSD firewall that underwent repeated changes back and forth between multi user and single user mode. I never saw any problem with either the web or telnet session. I know the traffic was going through the firewall because the only network connection with the web/telnet server was a single crossover cable connected to the "outer" NIC of a test firewall. I wouldn't be surprised, if the firewall were passing meaningful volumes of traffic, that at some point during the state changes, some packets might encounter some problems but my observations suggest that it's unlikely to be of more than a momentary nature.

I've had my outer bridging firewall set up this way some months and several times have switched to single user mode so I could make minor adjustments without ever interrupting traffic. Recently I finally setup an inner routing firewall with NAT and put it in single user mode while actively browsing the web on a NATed workstation connected to the Internet through the firewall. The mode change was not apparent to the workstation which could continue browsing. I used multiple Google searches and retrieved a variety of sites that I'd never connected to before. I was able to continue browsing in all phases of the switch to and from single user mode.

I let the firewall sit at each prompt that appears when entering single user mode and continued browsing without interruption. I did eventually determine that with OpenBSD 2.9 there is about a 5 second delay just after exiting single user mode as the firewall/NAT machine executes the system initialization scripts. I likely would not have found this except the first time I tried, I lost all network connectivity between the test workstation and the NIC on the firewall to which it was connected. Nothing would get this back. I rebooted, powered down, and manually reconfigured everything. Finally, suspecting a hardware failure, I disconnected everything and tested each part separately. When everything tested OK, I put everything back as it had been and the firewall/NAT was fine.

Following this I went through about 15 cycles of switching between single user and normal multi user mode. Twice in about 6 times I noticed some hesitation when retrieving new pages from the web. Not knowing if this was related to slow sites or web pages or the change back to normal multi user mode, I switched to using my own web servers which are lightly loaded and serve generally small text pages at LAN speeds. The browser cache was empty and the web servers were on the other side of the firewall/NAT machine. I did find that if I could switch back to the browsing PC immediately after typing 'exit' or pressing [Ctrl+D] while in single user mode, and start retrieving pages right away, the first page took 2 to 5 seconds after which page retrieval was essentially instantaneous. Subsequently I replaced the 2.9 firewall with OpneBSD 3.0 in a basic minimal configuration. I set up NAT but no firewall rules. In about six attempts I never could recreate the delay. As near as I could tell the changes on the NAT machine were simply not visible to the workstation going through it.

When you press "Ctrl+d" or "exit" from the single user shell, you see the end-of-boot messages that follow the various hardware related messages of a reboot. A normal "login:" prompt reappears on the console.

Prior to performing these experiments, I doubted the usefulness of immutable files and or securelevel 2. As your outer line of defense, nothing is likely to be more exposed than your firewall, at least if it's a standard routing rather than bridging, firewall. Generally a firewall is a good place to take every security measure that's practical. On the other hand, if your firewall is properly set up, NOTHING is going in or out of your network when the firewall is down.

Security is sometimes defined as having three characteristics: 1) confidentiality or assuring only authorized persons are allowed access to resources, 2) integrity or assuring than only authorized agents can make changes to resources and 3) availability or assuring that resources are available when needed. Thus any measure that forces systems to be not available when needed could be defined as an anti-security measure. Some systems have periodically scheduled downtime so a few minutes of downtime for a configuration change may be of no consequence but as requirements approach full 7 by 24 availability, any forced downtime for configuration changes becomes unacceptable.

I think my experiments have shown that at least for use as firewalls, OpenBSD provides some significant facilities for locking down key system configuration components, while still allowing for occasional system changes without necessitating visible downtime.

None of the capabilities that are generally referred to as services (daemons) are available in single user mode. Thus the use of immutable files and or security level 2 will, on occasion, necessitate some downtime for the services that are protected. Whether or not such downtime is acceptable is a matter of balancing system stability against any up-time requirements. These techniques are not suitable for a system requiring frequent configuration changes with stringent up-time requirements but may be appropriate on a stable system where limited downtime is acceptable as the types of changes under consideration can generally be scheduled.

If immutable files or security level 2 are to be used, I'd want to use them in a manner that minimized the amount of time in single user mode. For a firewall, preventing unauthorized changes to the filtering and network address translation rules, should be more important than any other configuration changes. Running a firewall at security level 2, once these rule sets are finalized, should be sufficient to assure this. Edit /etc/rc.securelevel to set securelevel=2, make /etc/rc.securelevel immutable ("# chflags schg /etc/rc.securelevel") and then manually set the security level to 2 ("# sysctl -w kern.securlevel=2") to do this. The reverse operation cannot be performed, i.e., sysctl cannot be used to change security level from 2 to 1.

When firewall rules need to be changed, put the system in single user mode ("# kill -15 1"). From the local console, make /etc/rc.securelevel changeable ("# chflags noschg /etc/rc.securelevel") and then edit it to set securelevel=1. Exit single user mode. Init executes the rc scripts and the system will be at security level 1 when it's returned to multi user mode. The firewall rules can be changed and tested in the normal manner.

As a simple mistake in a firewall rule could stop all network traffic or at least critical traffic, I would not attempt to change the firewall rules from single user mode while allowing the system to reboot to security level 2. Since security level 2 prevents firewall and NAT rule changes, a rule mistake could not simply be reversed (by switching rule sets). Thus, figuring out the mistake, getting back to single user mode and fixing it could easily take several minutes or longer. Instead, I'd use single user mode to change /etc/rc.securelevel and return the system to security level 1. I'd then change the firewall rules in the normal manner. Only after the new rules were tested, would I re-edit rc.securelevel, make it immutable and return the system to security level 2.

For other services or more extensive protection on a firewall, I would identify all the resources that were to be protected by the immutable flag. Then I'd write a pair of scripts, one to set all the immutable flags and the other to unset them all. When a satisfactory system configuration was achieved, the first script would be used to lock down all the protected resources. When subsequently, a configuration change needed to be made, the second script would be run from single user mode to unlock all the protected resources. The system would be rebooted to securelevel 1 and the configuration changes would be made and tested in the normal manner, whatever that might be, as if the immutable feature were not used. When the changes were satisfactorily tested, the protected resources would again be locked with the first script.

If security level 2 is to be used in conjunction with immutable files, the security level changes should also be scripted. The command "# sysctl -w kern.securelevel=2" in the first script will set the security level to 2. UNIX's here file capability can be used to automate the editing of /etc/rc.securelevel in both scripts, so the correct level would be set following any reboot.

Two sample scripts, syslock and sysunlock, that respectively lock down and unlock system initialization and security audit files are included. When run, syslock should display "kern.securelevel: 1 -> 2" as output. There should be no output from sysunlock which must be run in single user mode. The scripts include the automated edit of rc.securelevel.

The immutable flag can be used to lock down executable files rather than just system configuration files. If you do not consider the presence of any executable to represent a security hazard and do not wish to engage in the process of file removal, making all files in /bin, /sbin, /usr/bin, /usr/libexec, and /usr/sbin immutable would make these files less changeable than the file removal discussed elsewhere. If /usr/sbin/adduser is removed from the system there is nothing to stop a new one from being added unless it was in a directory that was made immutable. If /usr/sbin/adduser is immutable, it cannot be changed or replaced until the immutable flag is removed in single user mode. The files in the directories listed should never change except during a system install or upgrade or when a patch is applied. Adding these directories to the syslock and unlock scripts would be very simple and make it effectively impossible to install Trojan programs on a system set up this way.

If you normally run a system at securelevel 2 and also use NTP you should add the "-x" option to the command line that starts ntpd. Securelevel 2 will not allow the system time to be set back. As NTP's sole function is to keep a system time correct by checking with outside servers, it's normal operation will occasionally try to set the clock back by some fraction of a second. Securelevel 2 prevents this. The "-x" option prevents ntpd from stepping the clock but instead it "slews" the clock. In other words, instead of changing the time directly, ntpd either speeds up or slows down the clock until it comes into alignment with the correct time. As long as the clock is within several seconds of the correct time, this will have no negative effect on ntpd's operation. If the clock is significantly off, then slewing will take a long time to reach the correct time.

If immutable files will be used, two lines in OpenBSD 2.9 and 3.0 (and four lines in previous releases), in /etc/security should be changed from "cp -p $file $CUR" to "cp $file $CUR" so the immutable flag doesn't get copied to /var/backups. The sysunlock script clears these but there will be an error message in the daily output each time a file with the immutable flag is copied to /var/backups and /etc/security tries to execute chown on the copied file. A proliferation of immutable files, particularly to locations where the files are intended to rotate automatically, is unnecessarily messy.

Top of Page - Site Map

Copyright © 2000 - 2014 by George Shaffer. This material may be distributed only subject to the terms and conditions set forth in http://GeodSoft.com/terms.htm (or http://GeodSoft.com/cgi-bin/terms.pl). These terms are subject to change. Distribution is subject to the current terms, or at the choice of the distributor, those in an earlier, digitally signed electronic copy of http://GeodSoft.com/terms.htm (or cgi-bin/terms.pl) from the time of the distribution. Distribution of substantively modified versions of GeodSoft content is prohibited without the explicit written permission of George Shaffer. Distribution of the work or derivatives of the work, in whole or in part, for commercial purposes is prohibited unless prior written permission is obtained from George Shaffer. Distribution in accordance with these terms, for unrestricted and uncompensated public access, non profit, or internal company use is allowed.