Hardening OpenBSD Internet Servers

Priorities, Costs and Benefits

I first wrote the pages on Hardening OpenBSD Internet Servers after I'd applied these techniques to my OpenBSD 2.7 firewall and web server. In response to reader suggestions and other things I learned, I added new techniques and with each OpenBSD upgrade, I've updated and expanded the pages. Today the section is perhaps three times the original size. Each upgrade has been more difficult than the last because of the growing number things to do and check. Below I will rank the relative values of the techniques covered in Hardening OpenBSD as well as some related techniques, and discuss factors that make each specific technique more or less applicable.

From the beginning, I've maintained that security is about tradeoffs and matching the techniques to the specific site. I openly admit that some of the techniques described are extreme and that many system administrators will regard the resulting systems as unusable. I've always been security conscious and there's been a pride that the techniques I've used are significant, relative to the low value of the systems protected.

With each upgrade, I've had doubts about the value of what I was doing. The upgrade from 2.9 to 3.0 was both difficult and emotionally trying; see the overview of the "OpenBSD 2.9 to 3.0 Changes" to see why. With this upgrade, I've reached the conclusion that in most environments, several of, the techniques discussed in these pages are counter productive. It's not that they don't improve security but rather they draw limited resources from other tasks that may have a better security related return.

On Internet connected systems today, the threats increase and change increasingly rapidly. OpenBSD responds to real security threats more rapidly and effectively than any other alternative available. No other general purpose operating system is likely to ever come close, and so far, no specialized security oriented system comes close to matching OpenBSD's established track record. The surest way to create secure Internet connected computers is to combine sound administrative practices with the most recent release of OpenBSD. Security techniques that make it harder to upgrade OpenBSD may result in systems that are less secure than a more generic system on a newer release of OpenBSD. Many of the techniques discussed in these pages make upgrades more difficult because they need to be reapplied or merged back into the system with each upgrade.

There are still tradeoffs to be considered. To be useful, a computer will almost certainly contain components that differ from a default install. For example a web server will have at least web content and customized web configuration files that are not part of any default install. These also make upgrades more difficult though not the extent that extensive changes to the system initialization files do. An ultra hardened, truly single function server, such as a public web server that exposes only ports 80 and 443 to public access, and is behind a strong firewall with custom rules, may go safely for two or three years without an operating system upgrade, provided that the web server software is kept up to date with appropriate security fixes. If more ports are exposed such as SSH, then the risks increase and the need for a general OS upgrade is likely to be higher, sooner. The risks are of course greater if SSH is available outside the LAN than if its use is restricted by the firewall to local use only.

Generally, the more limited the purposes of a specific computer, the more likely that security related patches and upgrades may be applied to essential functions of the computer only and that general operating system upgrades may be delayed. The more diverse the functions performed by the computer, the more important it is that it be easily upgraded and the less likely some of the techniques covered in these pages will be applicable.

The Ten Practical Security Steps section of this web site identifies the steps that I generally consider to provide the most cost effective security regardless of the operating system. Much of what appears in Hardening OpenBSD does not appear in Ten Practical Security steps but some does. The Recovery CD discussed as part of Hardening OpenBSD is just one approach to system backups. As discussed here, it closely relates to the file removal process as it allows files to be removed that otherwise could not be. Whether or not you choose to remove files, developing and testing a comprehensive backup system is the first essential step towards security. Without good backups, regardless or what may cause a system failure, you are unlikely to recover a failed or compromised system. Once the procedures are developed and tested, most of the work of backups can be automated but there will always be media management issues. As essential as they are, even backups have some risk, as more than one administrator has restored when they intended to backup. Proper media rotation and off-site storage will minimize the risks of catastrophic losses.

If you read my 10 Security Steps and the Password Management page in the Passwords section, you will know that without unusual management support and training opportunities, that I believe IT lacks the resources and tools to impose good passwords on users. Systems should be set up so ordinary users cannot do appreciable damage with groups and directory permissions. What IT can do is insure its own staff or more specifically root accounts and anyone authorized to use root accounts are trained in the creation of strong passwords and required to use such. Only such users should be allowed to su to root. OpenBSD takes care of this automatically. As long as one or more users is in the wheel group, only these users can su. Remote root logins should always be disabled; this requires a one line change in the default /etc/sshd_config. This configuration forces all remote users using SSH to know two strong passwords, or have the right keys and know the root password to access the root account. The number of users on hardened systems should be severely restricted. Good passwords are a high return low risk technique but require modest ongoing efforts. Generally /etc/passwd, /etc/master.passwd and related files can be copied from one version to the next. It may be necessary to run pwd_mkdb to build the actual password database used by the system.

A firewall with a strong custom rule set is essential. Here I mean a dedicated firewall with two or more network interfaces that protects a LAN with multiple machines. In some special circumstances, as discussed in Setting Up a Single Host Firewall a host may act as it's own firewall, but generally I consider it a waste of time to setup full firewalls on hosts already protected by a properly configured exterior firewall. If the machine in question is not protected by a dedicated firewall, this step should be regarded as absolutely mandatory and even more important than strong administrator passwords. Setting up a good firewall is somewhat labor intensive but without one it's only a matter of time before Internet connected computers are compromised.

How to properly use UNIX groups and directory file permissions to protect files and directories, while allowing shared update access to those specifically authorized such access, is discussed in the Users and Security Groups page of this section which might more accurately be called "UNIX Groups and Directory Permissions". There is little if any risk in setting up good directory security but highly granular security which grants access to those who need it and denies to those who don't is labor intensive in both design and maintenance. The alternative is high risk.

Disabling unneeded services as described on the Removing Unnecessary Services page in this section is just short of essential. OpenBSD starts with relatively few services enabled. If you pay attention to the advice provided here, there is little chance of disabling a needed service and it takes only a small effort to disable unneeded services. The changes in /etc/rc.conf and /etc/inetd.conf files need to be repeated with each upgrade or merged into the new configuration files.

Discussed on this same page is the use of TCP Wrappers for protecting older TCP services. If you must use telnet or FTP, then use these with TCP Wrappers. The firewall page explains why I prefer TCP Wrappers in this special situation to a host based firewall, in addition to a dedicated outer firewall. TCP Wrappers requires only a small effort to setup but the changes in /etc/inetd.conf need to be preserved with each upgrade. /etc/hosts.allow and /etc/hosts.deny can be carried forward from one version to the next without change. The only risk with using TCP Wrappers is that it also controls sshd and sendmail so that if it is used, the hosts.allow file must be adjusted to permit sshd and sendmail to work as intended.

I did not treat host based intrusion detection as part of Hardening OpenBSD but rather on my Homegrown Intrusion Detection page. Some such techniques belong about here in the order of importance and cost benefit payoffs. OpenBSD's built in security auditing is a first step in this direction but is fully contained on a single host. A knowledgeable intruder who has gained root access is likely to be able to defeat this system. Free versions of Tripwire suffer from the same limitation or require awkward manual steps to remove the checksum database from the host.

I believe that some form of automated tracking of executable and system configuration files where the checksum results are automatically transferred to a remote system for analysis and to unchangeable (CD-R) storage media for archival storage is a must. It's also desirable to frequently monitor users and processes with the results also transferred to a remote system. Intrusion detection is somewhat labor intensive in both initial implementation and ongoing maintenance. This is very important but is an area some may regard as optional. Intrusion detection is generally risk free as it is normally a monitoring only process. Intrusion detection systems, whether host or network based, that incorporate automated responses to detected actions, may become part of denial of service attacks, unless used with great care.

The OpenBSD Installation page is here primarily as an aid for those new to OpenBSD. There are few security issues discussed on this page but the importance of not installing unneeded software is mentioned. It's completely pointless to even consider the labor intensive file removal discussed below, if unneeded junk is placed on the system during the install.

At this point, nearly all the techniques discussed in the 10 Security Steps that are also discussed in the Hardening OpenBSD section have been covered. Also, not coincidentally, all the techniques that are widely regarded as standard security techniques which are covered in the Hardening OpenBSD section have been covered. The remaining techniques covered in Hardening OpenBSD generally are applicable to special circumstances, are resource intensive relative to the gain, have associated risks, or a combination of more than one of these attributes.

Using immutable files and security level 2 has the potential to lock down key portions of a system with a very small effort, but to be an appropriate technique, it must be practical to administer any system using these techniques locally. Once set, there is no way to undo these settings from a remote location. On a system that is used exclusively as a firewall, there is no practical downside, provided the files made immutable can be changed locally when necessary. On a system that provides normal services (daemons), the up-time requirements of the system must be balanced against the stability of the system configuration, as any change to files that have been made immutable will require about one or two minutes of downtime to reset files, so that they may be updated. Because the use of these techniques can be implemented in local scripts not affected by upgrade processes, there is little impact on system upgrades. There are no risks except those related to the added difficulty of changing files once they have been made immutable, unless a file that must change regularly is made immutable, and such a condition is likely to be fixed quickly.

Modifying login banners requires a small effort with at best a tiny practical security gain. There is essentially no risk. The primary importance is legal. Depending on the type of organization that you are and where you are located, some such changes may be necessary to obtain full legal protection for your systems. The changes to implement this will need to be repeated or merged into each upgrade.

Using the built in system auditing process provided with OpenBSD has no risks as it is turned on following any install and is simply a monitoring process. Tightening the security on the protected files will take a moderate amount of time and provide a limited gain in security with very little risk. It's a step that's likely to need repeating or adjusting each time a system is upgraded.

Use of the nodev and nosuid mount options represent low effort and low risk techniques that will yield very limited security improvements. Primarily the nosuid option will prevent SUID and GUID files hidden in out of the way places though it's easy enough to find such files with cron scripts. Unless you monitor the nosuid mount settings via cron, you should be monitoring all file systems for SUID and GUID files anyhow. Keeping these across upgrades will require adding the change, most likely to /etc/rc.local, each upgrade.

The noexec mount option represents a much higher risk as the only file system it can be safely used on is the /tmp system, assuming this is actually a separate file system. /var may be suggested but various applications actually place executables in /var or expect them to be placed there, including OpenBSD's standard web install. Noexec will break these applications and provides no protection against shell scripts run in the current shell with dot prefix as in "$. ./runme". Keeping these accross upgrades will require a minor change each upgrade.

Removing files is a labor intensive, high risk process. The provided scripts greatly reduce the work but they must be carefully reviewed. Sticking to the most conservative settings suggested by the scripts probably will reduce the chances of making mistakes. Adding additional files to the scripts will increase the likelihood of making a mistake and causing problems.

Two adverse situations may result. One is removing files that are needed by some processes that you intend to run. In the worst case, the processes may be rarely used and the resulting errors not clear, making it difficult to identify what needs to be put back. If you experiment with extensive file removal, at some point you will mangle permission settings on directories or files. Either or both of two things may occur. The root or other key directories may lose their world readable attributes. If the / or /home directories are not world readable, non root users cannot log in. The SUID or GUID bit may be lost on programs that require this for proper operation. Working as root you are likely to miss both situations. It won't be until ordinary users, use the system that problems appear.

Finally removing files is a process that needs to be repeated with each upgrade. Assuming it's scripted, the work is greatly reduced but each new release adds and removes files. The former will at worst add files that you might not want on your system. The latter will cause errors in the remove script, which are almost impossible to detect until the script is run. Correcting these errors has a significant possibility of mangling file or directory permissions. You may also lose files by failing to faithfully restore after each remove attempt, necessitating a partial upgrade or restore from install media to restore missing files, which may undo other changes that have been made.

The drawbacks to file removal are offset by the fact that it provides the most granular control over locking down what functions can be performed on the system and denying the use of certain functions remotely but not locally if the CD-R, system lock, approach is used.

Building custom kernels has the highest cost to benefit ratio of all the techniques covered. While it's normally very simple to compile a generic kernel and not much harder to change a few options, systematically disabling every unneeded option is extremely labor intensive because it so hard to be sure what is and is not really needed. If you go slow and cautiously you'll go through a number of build cycles. If you get very aggressive, make all your changes at one time, and then have a problem, it could be very hard to find. Every new release introduces new changes and requires systematic comparison of old and new kernel configuration files to identify new functions on which decisions need to be made.

It's almost impossible to fully test custom kernels; about all you can do is ensure that major system functions have not been disabled. Determining if problems that appear in the future relate to the custom kernel can be difficult or impossible. If the problem is highly repeatable and occurs with the custom but not the GENERIC kernel, then you know the custom kernel is the likely cause. If the problem is highly repeatable and occurs with both the custom and GENERIC kernels, the custom kernel is not likely to be the cause. If a problem is infrequent, it will be very hard to determine if the custom kernel is the cause.

For all the work of building a custom kernel, there are normally no unequivocal advantages. Of course if you need IPX or Appletalk or something else not included in the GENERIC kernel, you may have no choice, but that is a totally different issue than the removal of unneeded options discussed in these pages. The custom kernel may be much smaller than the GENERIC and with the right benchmarks it should be measurably faster but none that I've seen, and some have been less than one third the size of the original, have shown any noticeable performance differences.

I know of no option that you can remove in creating a custom kernel that will prevent an intrusion the way turning off an unneeded network service may. At best some custom kernels may deny some preferred tools to a skilled intruder after they have gained access. Custom kernels are not recommended unless you have taken all other appropriate security measures and know just what you are doing and why. If you want to build a custom kernel just to see what it's about and demystify the process, that is a worthwhile learning experience and on a test machine there is nothing to lose if the process doesn't go well. In a work environment, absent a specific reason for creating a custom kernel, I think it will be hard to justify based on a cost benefit analysis.

The use of read only file systems is not recommended. Though this is a low effort technique to implement, it will almost surely break both system and application processes. Further it will act as a constant irritant to administrative staff who must turn it off every time they need to make a change or as is more likely, force minor changes to be repeated when they cannot be saved because the / file system was set to read only. The possibility that an intruder's root kit may be broken is not worth the costs. Both immutable files and file removal can break the same root kits and do so in a manner the intruder cannot undo. Read only filesystems may be made writable by root with a trivial command. Immutable files prevent changes to the system with less effort and with lower risks, because they can be applied on a file by file or directory basis, and removable files provide a very high degree of control over what functions may or may not be performed on the system.

A Check List page with a printer friendly version, reduces the techniques described here, to short action items with the value of the technique reduced to a 0 (not recommended) to 5 (essential) based on criteria from this page. Items are listed in suggested order of completion.

Top of Page - Site Map

Copyright © 2000 - 2014 by George Shaffer. This material may be distributed only subject to the terms and conditions set forth in http://GeodSoft.com/terms.htm (or http://GeodSoft.com/cgi-bin/terms.pl). These terms are subject to change. Distribution is subject to the current terms, or at the choice of the distributor, those in an earlier, digitally signed electronic copy of http://GeodSoft.com/terms.htm (or cgi-bin/terms.pl) from the time of the distribution. Distribution of substantively modified versions of GeodSoft content is prohibited without the explicit written permission of George Shaffer. Distribution of the work or derivatives of the work, in whole or in part, for commercial purposes is prohibited unless prior written permission is obtained from George Shaffer. Distribution in accordance with these terms, for unrestricted and uncompensated public access, non profit, or internal company use is allowed.