Hardening OpenBSD Internet Servers
Immutable Files, Securelevels and Mount Options
BSD systems include some options such as immutable files and
securelevels that require root to have local access to make
certain types of changes. Mount options, including read only
filesystems may hinder intruders. These are discussed here.
Read Only Filesystems
Another obstacle to place in the way of intruders and
unauthorized local users is to mount some filesystems as
read only. Prime candidates for this would be / and /usr.
Once a system configuration is stable, there should be very few
if any files that need to change frequently in these locations.
These could be mounted read only late in the boot process or even
periodically via cron. Having these read only might save a few
seconds rebooting after an unexpected shutdown such as a power
failure.
The real reason for doing this is that it's going to break
rootkits and other tools crackers use for covering their tracks
after root access is gained. It won't stop any knowledgeable
intruder but it may slow them down. A mounted filesystem can be
made read only by "mount -ur /filesystem_mount_point" and
returned to writable status just as easily with "mount -uw
/filesystem_mount_point".
To cover their tracks after gaining root access, intruders have
to be able replace executables with Trojans and modify system
configuration files. A read only filesystem will stop this until
it is made writable. Much cracking is done with the aid of
scripts. There is a big difference between running a script and
having root status with tracks covered in seconds and running it
and seeing error messages. At that point, the intruder has to
determine the cause of the errors, know or figure out the syntax
of the command to make filesystems writable and manually execute
these commands or add them to the scripts. It could be the
difference between being caught and getting away with an
intrusion. Setting an alarm if a read only filesystem became
writable could even be made part of an intrusion detection
system.
Anyone capable of writing cracking tools will have no difficulty
dealing with read only filesystems. Because they are very much
the exception though, the author may not think to include the
necessary commands until a read only filesystem is encountered.
Someone using tools created by others, which are probably the
large majority of would be intruders, may well not know how to
deal with the problem.
Noexec, Nosuid and Nodev Mount Options
Closely related to read only filesystems are the noexec, nosuid
and nodev mount options. Noexec prevents executable files from
being executed. This includes scripts as well as binary files
but won't prevent a script from being interpreted by the current
shell with the ". " prefix. Normally, you'd not expect
to find legitimate executables in the /tmp and /var filesystems
but intruders might try to hide executables there. It's common
to find executables in /home but should be very rare to find SUID
programs in /home. Thus it makes sense for /home, /tmp and /var
to be mounted nosuid. Nodev prevents a device file from being
acknowledged. The only normal place for device files is /dev
so it makes sense and does not appear to have adverse consequences
to mount all filesystems but / as nodev.
The following commands at the end of /etc/rc.local would set all
the mount options discussed above:
mount -ur /
mount -ur -o nodev /usr
mount -uw -o nodev,nosuid /home
mount -uw -o nodev,nosuid,noexec /var
mount -uw -o nodev,nosuid,noexec /tmp
Costs of Non Standard Mount Options
As with all security related measures, you need to ask whether
any hoped for advantage is worth the cost. There is no question
there is a cost; since I've been setting / and /usr to read only
on some of my systems, I've lost track of how many times a read
only error has stopped me from saving changes. It's usually only
a few seconds to deal with this. Either the file system is made
writable and minor changes redone or the file is saved to a
writable filesystem and subsequently moved into place after
making the original target writable. The read only filesystems
have created a nuisance factor with no clear security benefit.
I recently installed the Analog, web statistics package on my
OpenBSD web server. The read only /usr filesystem prevented analog
from being able to use its DNS lock and cache files which it wants
to locate in the analog directory which is a subdirectory of
/usr/local. A quick search of the Analog documentation shows no
obvious configuration option to relocate the DNS files to another
location such as where the web logs or analog output is placed. While
I may keep read only / and /usr filesystems on a firewall, a
read only /usr on a web server with Analog simply isn't worth the
effort. I have little doubt that each of the other suggested
mount settings will break some other software.
Trying settings such as these on a test machine that doesn't really
do anything may not turn up any but the most obvious problems such
as the inability to access a device. Determining even this much
would require attempting to access the device while the settings
were in effect. I decided to try these mount settings on my live
OpenBSD web server to see what if any problems would show up.
I immediately tried accessing the web site. Web pages were being
served normally so the primary function of the machine did not
appear to be affected and I left the settings in place. I don't
remember if I thought to try CGI scripts at that time. It took a
day to find a problem. It turns out the noexec setting on /var
breaks the CGI version of Analog. I didn't learn this until I
tried to get some immediate web statistics that the CGI version
of Analog provides.
My production web site runs out of /home subdirectories but the
administrative site uses the default install location for Apache
on OpenBSD. These place the cgi-bin directory inside of /var/www
and thus no CGI scripts in the default location will run with /var
mounted noexec. The Analog CGI interface runs from this directory.
This shows the largest problem with using unusual mount settings
such as those suggested above for security purposes. It's not
that they break some essential, regularly used software which should
be quickly obvious and easy to fix. It's that some infrequently
used software may be affected. By the time the problem shows
itself, the changes that caused it may be a dim memory or the
person encountering the problem may not even know of the configuration
changes that are responsible.
While working on the hardening process for OpenBSD 2.9, I became
aware of another problem with read only / (root) filesystems.
I set the read only property early while configuring my first 2.9
server and had plenty of opportunity to observe a series of
error messages caused by mounting / read only. The messages were
as follows:
date time hostname login: /etc/fbtab: chmod (/dev/console) Read only file system
date time hostname login: /etc/fbtab: chmod (/dev/wskbd0) Read only file system
date time hostname login: /etc/fbtab: chmod (/dev/wsmouse0) Read only file system
Each message was repeated twice and their length caused them to wrap.
It was not easy to ignore these messages which filled much of the
screen following some logins. They did not appear if / was returned to
it's normal writable status. I do not know what practical consequences
the failure of chmod had. It was not responsible for the keyboard lockups
described in
Custom Kernel: Large Unwanted Change in 2.9
as these lockups definitely occured when no file system was set to
read only.
A similar /dev/console, but not /dev/wskbd0 or /dev/wsmouse0,
error message appeared in OpenBSD 2.8. Because I did not set the
read only option until very late in the 2.8 configuration
process, rarely logged in at the console and the message was much
less prominent, it either did not register or I never found the
time to look into it.
There were three choices once I was aware of the messages in 2.9. I
could ignore them as there didn't appear to be obvious consequences.
I'm not comfortable deliberately ignoring system error messages that
I am aware of. I could attempt to understand what the system was
trying to do and accomplish that by some other means or try to surpress the
error message if I decided they were not indicative of an important
condition. Either way meant more time and more system customizations.
It's clear that if all the customizations discussed throughout
the entire Hardening OpenBSD section are implemented, that I've
past the point of diminishing returns. With the relatively small
amount of well backed up, nearly static data on my OpenBSD
systems, I can build a new OpenBSD system and restore my specific
functions in much less time than it takes to develop an updated
hardening process for each new OpenBSD release. Other systems
with more valuable and more changing data that might be harder to
rebuild might be worth more up-front security effort. In my
case, I need to draw the line and chose the third option:
eliminate the error messages by eliminating the cause, the non
standard read only / file system.
Chflags, Immutable Files and Security Levels
A couple readers brought chflags, immutable files and security
levels to my attention but I was initially skeptical of the
potential. I'd heard of immutable or unchangeable files but also
that once made immutable, a file could only be changed by booting
into single user mode. One of UNIX's strengths is the ability to
make system changes without rebooting and thus disrupting the
services it provides. Losing this flexibility didn't seem worth
the limited security enhancements, especially after I'd already
found ways to control access to many of the tools necessary to
make the changes in the first place.
It turns out, that at least for a firewall, you can make key system
files immutable and or use securelevel 2 to prevent any changes to
the firewall and NAT rules. You can do this and from the local
console only, make subsequent changes to the firewall and NAT rules
or unset the immutable flag, without interrupting the firewall
function. You can do this because firewall and NAT functions are
built into the kernel and on BSD systems you can actually switch
back and forth between single user and multi user modes without
rebooting the computer or interrupting the firewall and NAT
functions in the kernel. I was skeptical until I actually saw it
work multiple times.
Chflags is the command that makes a file immutable or
unchangeable. It has several options including system and user
append only flags but the one of particular interest is the schg
or system immutable flag available only to the super user. Once
this flag is set on a file, no one, including the super user, can
make changes to the file until the schg flag is removed. If this
flag could be turned off as easily as it is set, then it would be
no more interesting than read only/writable filesystems, that
can be changed at will by root.
The system immutable flag, however, works in conjunction with the
security level so that it is much more difficult to turn the flag
off than on. OpenBSD has four security levels; see the
securelevel
(7) man page for details. The lowest is -1 or
"permanently insecure"; this allows a multi user mode as insecure
as the standard single user mode and so is of no interest to
those concerned with tightening security. 0 or insecure is the
security level during booting and in single user mode. It allows
the single user (root or the super user) to do some things that
cannot be done in 1 or secure mode, the standard multi user mode.
The most interesting, for our purposes, is to turn off the system
immutable flag. In the normal multi user, secure mode, the system
immutable and append only flags can be set on files but not
unset. Security level 2 or highly secure mode, prevents some low
level disk access and debugging related actions, prevents the
system clock from being set back and prevents firewall and NAT rules
from being changed.
Once the security level is at 1 or 2 it can only be lowered via the
init program. Root can manually raise the security level from 1 to
2 via the sysctl program or /etc/rc.securelevel can be edited so that
the security level will be 2 at the completion of the boot process.
Init can be used to change to from security level 1 or 2 by returning
the system to level 0 and single user mode. The
init(8)
man page gives the command as "kill -s TERM 1". This never
worked for me but "kill -15 1" worked quite reliably and
"kill -TERM 1" works as well. If this is executed remotely, the
network connection is broken as soon as enter is pressed.
If this is done from the local console or you're watching the
local console when this is done remotely, you see messages from
exiting daemons. The messages should vary depending on what has
been running. They end with the date and time and "init: kernel
security level changed from 1 to 0". This is followed by the
"Enter pathname of shell or RETURN for sh:" prompt that should be
familiar from the install process or floppy boot sequence. If
you press enter, you are prompted for "Terminal type?" until you
give a valid type. I used vt100, vt220 and pcvt25. If you
accepted sh as your shell, you are left at the "#" single user
prompt.
If you doubt that you're in single user mode, "ps ax" shows only
three processes: /sbin/init, -sh (sh) and ps -ax. What's most
interesting about this is if the computer that is switching between
multi user and single user modes is a firewall running IP Filter or
now Packet Filter, you can maintain network connections through it
with no apparent effect, i.e. the kernel never reloads or stops
performing the firewall and NAT processes that are part of it. This
works on both bridging firewalls and routing firewalls with NAT.
I was able to run both web and telnet sessions through an
OpenBSD firewall that underwent repeated changes back and forth
between multi user and single user mode. I never saw any
problem with either the web or telnet session. I know the
traffic was going through the firewall because the only
network connection with the web/telnet server was a single
crossover cable connected to the "outer" NIC of a test firewall.
I wouldn't be surprised, if the firewall were passing
meaningful volumes of traffic, that at some point during the
state changes, some packets might encounter some problems but
my observations suggest that it's unlikely to be of more than
a momentary nature.
I've had my outer bridging firewall set up this way some months and
several times have switched to single user mode so I could make
minor adjustments without ever interrupting traffic. Recently I
finally setup an inner routing firewall with NAT and put it in
single user mode while actively browsing the web on a NATed
workstation connected to the Internet through the firewall. The
mode change was not apparent to the workstation which could continue
browsing. I used multiple Google searches and retrieved a variety of
sites that I'd never connected to before. I was able to continue
browsing in all phases of the switch to and from single user mode.
I let the firewall sit at each prompt that appears when entering
single user mode and continued browsing without interruption. I did
eventually determine that with OpenBSD 2.9 there is about a 5 second
delay just after exiting single user mode as the firewall/NAT
machine executes the system initialization scripts. I likely would
not have found this except the first time I tried, I lost all
network connectivity between the test workstation and the NIC on the
firewall to which it was connected. Nothing would get this
back. I rebooted, powered down, and manually reconfigured
everything. Finally, suspecting a hardware failure, I
disconnected everything and tested each part separately. When
everything tested OK, I put everything back as it had been and the
firewall/NAT was fine.
Following this I went through about 15 cycles of switching between
single user and normal multi user mode. Twice in about 6 times I
noticed some hesitation when retrieving new pages from the web. Not
knowing if this was related to slow sites or web pages or the change
back to normal multi user mode, I switched to using my own web
servers which are lightly loaded and serve generally small text
pages at LAN speeds. The browser cache was empty and the web
servers were on the other side of the firewall/NAT machine. I did
find that if I could switch back to the browsing PC immediately
after typing 'exit' or pressing [Ctrl+D] while in single user mode,
and start retrieving pages right away, the first page took 2 to 5
seconds after which page retrieval was essentially instantaneous.
Subsequently I replaced the 2.9 firewall with OpneBSD 3.0 in a basic
minimal configuration. I set up NAT but no firewall rules. In about
six attempts I never could recreate the delay. As near as I could
tell the changes on the NAT machine were simply not visible to the
workstation going through it.
When you press "Ctrl+d" or "exit" from the single user shell,
you see the end-of-boot messages that follow the various hardware
related messages of a reboot. A normal "login:" prompt reappears
on the console.
Prior to performing these experiments, I doubted the usefulness
of immutable files and or securelevel 2. As your outer line of
defense, nothing is likely to be more exposed than your firewall,
at least if it's a standard routing rather than bridging, firewall.
Generally a firewall is a good place to take every security measure
that's practical. On the other hand, if your firewall is properly
set up, NOTHING is going in or out of your network when the
firewall is down.
Security is sometimes defined as having three characteristics:
1) confidentiality or assuring only authorized persons are allowed
access to resources, 2) integrity or assuring than only authorized
agents can make changes to resources and 3) availability or
assuring that resources are available when needed. Thus any
measure that forces systems to be not available when needed could
be defined as an anti-security measure. Some systems have
periodically scheduled downtime so a few minutes of downtime for
a configuration change may be of no consequence but as requirements
approach full 7 by 24 availability, any forced downtime for
configuration changes becomes unacceptable.
I think my experiments have shown that at least for use as
firewalls, OpenBSD provides some significant facilities for locking
down key system configuration components, while still allowing for
occasional system changes without necessitating visible downtime.
None of the capabilities that are generally referred to as
services (daemons) are available in single user mode. Thus the use of
immutable files and or security level 2 will, on occasion,
necessitate some downtime for the services that are protected.
Whether or not such downtime is acceptable is a matter of
balancing system stability against any up-time requirements. These
techniques are not suitable for a system requiring frequent
configuration changes with stringent up-time requirements but may
be appropriate on a stable system where limited downtime is
acceptable as the types of changes under consideration can
generally be scheduled.
If immutable files or security level 2 are to be used, I'd want to
use them in a manner that minimized the amount of time in single
user mode. For a firewall, preventing unauthorized changes to
the filtering and network address translation rules, should be
more important than any other configuration changes. Running a
firewall at security level 2, once these rule sets are finalized,
should be sufficient to assure this. Edit /etc/rc.securelevel to
set securelevel=2, make /etc/rc.securelevel immutable ("# chflags
schg /etc/rc.securelevel") and then manually set the security
level to 2 ("# sysctl -w kern.securlevel=2") to do this.
The reverse operation cannot be performed, i.e., sysctl cannot
be used to change security level from 2 to 1.
When firewall rules need to be changed, put the system in single
user mode ("# kill -15 1"). From the local console, make
/etc/rc.securelevel changeable ("# chflags noschg
/etc/rc.securelevel") and then edit it to set securelevel=1. Exit
single user mode. Init executes the rc scripts and the system
will be at security level 1 when it's returned to multi user
mode. The firewall rules can be changed and tested in the normal
manner.
As a simple mistake in a firewall rule could stop all network
traffic or at least critical traffic, I would not attempt to change
the firewall rules from single user mode while allowing the system
to reboot to security level 2. Since security level 2 prevents
firewall and NAT rule changes, a rule mistake could not simply be
reversed (by switching rule sets). Thus, figuring out the mistake,
getting back to single user mode and fixing it could easily take
several minutes or longer. Instead, I'd use single user mode to
change /etc/rc.securelevel and return the system to security level
1. I'd then change the firewall rules in the normal manner. Only
after the new rules were tested, would I re-edit rc.securelevel,
make it immutable and return the system to security level 2.
For other services or more extensive protection on a firewall, I
would identify all the resources that were to be protected by the
immutable flag. Then I'd write a pair of scripts, one to set all
the immutable flags and the other to unset them all. When a
satisfactory system configuration was achieved, the first script
would be used to lock down all the protected resources. When
subsequently, a configuration change needed to be made, the second
script would be run from single user mode to unlock all the
protected resources. The system would be rebooted to securelevel 1
and the configuration changes would be made and tested in the normal
manner, whatever that might be, as if the immutable feature were not
used. When the changes were satisfactorily tested, the protected
resources would again be locked with the first script.
If security level 2 is to be used in conjunction with immutable
files, the security level changes should also be scripted. The
command "# sysctl -w kern.securelevel=2" in the first script will
set the security level to 2. UNIX's here file capability can be
used to automate the editing of /etc/rc.securelevel in both
scripts, so the correct level would be set following any reboot.
Two sample scripts,
syslock and
sysunlock,
that respectively lock down and unlock system initialization and
security audit files are included. When run, syslock should
display "kern.securelevel: 1 -> 2" as output. There should be no
output from sysunlock which must be run in single user mode. The
scripts include the automated edit of rc.securelevel.
The immutable flag can be used to lock down executable files rather
than just system configuration files. If you do not consider the
presence of any executable to represent a security hazard and do not
wish to engage in the process of file removal, making all files in
/bin, /sbin, /usr/bin, /usr/libexec, and /usr/sbin immutable would
make these files less changeable than the file removal discussed
elsewhere. If /usr/sbin/adduser is removed from the system there is
nothing to stop a new one from being added unless it was in a
directory that was made immutable. If /usr/sbin/adduser is immutable,
it cannot be changed or replaced until the immutable flag is removed
in single user mode. The files in the directories listed should never
change except during a system install or upgrade or when a patch is
applied. Adding these directories to the syslock and unlock scripts
would be very simple and make it effectively impossible to install
Trojan programs on a system set up this way.
If you normally run a system at securelevel 2 and also use NTP you
should add the "-x" option to the command line that starts ntpd.
Securelevel 2 will not allow the system time to be set back. As
NTP's sole function is to keep a system time correct by checking
with outside servers, it's normal operation will occasionally try
to set the clock back by some fraction of a second. Securelevel 2
prevents this. The "-x" option prevents ntpd from stepping the
clock but instead it "slews" the clock. In other words, instead of
changing the time directly, ntpd either speeds up or slows down the
clock until it comes into alignment with the correct time. As long
as the clock is within several seconds of the correct time, this
will have no negative effect on ntpd's operation. If the clock is
significantly off, then slewing will take a long time to reach the
correct time.
If immutable files will be used, two lines in OpenBSD 2.9 and 3.0
(and four lines in previous releases), in /etc/security should be
changed from "cp -p $file $CUR" to "cp $file $CUR" so the immutable
flag doesn't get copied to /var/backups. The sysunlock script clears
these but there will be an error message in the daily output each
time a file with the immutable flag is copied to /var/backups and
/etc/security tries to execute chown on the copied file. A
proliferation of immutable files, particularly to locations where
the files are intended to rotate automatically, is unnecessarily
messy.
Top of Page -
Site Map
Copyright © 2000 - 2014 by George Shaffer. This material may be
distributed only subject to the terms and conditions set forth in
http://GeodSoft.com/terms.htm
(or http://GeodSoft.com/cgi-bin/terms.pl).
These terms are subject to change. Distribution is subject to
the current terms, or at the choice of the distributor, those
in an earlier, digitally signed electronic copy of
http://GeodSoft.com/terms.htm (or cgi-bin/terms.pl) from the
time of the distribution. Distribution of substantively modified
versions of GeodSoft content is prohibited without the explicit written
permission of George Shaffer. Distribution of the work or derivatives
of the work, in whole or in part, for commercial purposes is prohibited
unless prior written permission is obtained from George Shaffer.
Distribution in accordance with these terms, for unrestricted and
uncompensated public access, non profit, or internal company use is
allowed.
|