by Objective
The objective of this exercise is to enable remote systems (clients) to be able to write to the System Log on a central Log Server (server), without losing the ability to write to their own local system log. This is achieved by creating a reverse SSH tunnel from the server to each client (on a special port) such that if the client configures their system log to use that port, the log entry will be sent across the tunnel to the log server.
Configuring SSH Access between Client and Server
The server will need to create an SSH connection to each client. The SSH tunnel will be initiated by the root user at the server but SSH connections by root have been disabled on all of my clients (for obvious reasons) so we will need to use a non-root user to create the SSH tunnel (e.g. someuser). The steps below were used to set up SSH access between server and client:
- Create a public key on the server as the user who will initiate the SSH tunnel to the clients (root)
server# ssh-keygen –t rsa - Copy the resultant public key file to the SSH directory for the someuser at the client
server# scp /.ssh/id_rsa.pub someuser@remotehost: - At the client, append the public key file just copied to the ~someuser/.ssh/uthorized_keys file
The server should now be able to crate an unchallenged SSH connection to the client (as user someuser)
Configuring SSH Tunnelling
Syslog normally uses port 514 and it follows that this is the port that would need to be tunnelled from the clients to the server in order to enable remote logging. However, the non-root user at the client (someuser) will not be allowed to open port 514 since it is below the range of ports it has access to (only root can access ports below 1024 on Unix). Therefore, what we require is a tunnelled connection from a port above 1024, say 1514. If you are using a firewall or IP Packet Filtering software, you will also need to configure this to allow TCP traffic on port 514 at the server.
Use the following command to test the reverse SSH tunnel between the server and client:
server# /usr/bin/ssh -nNTx -R 1514:127.0.0.1:514 someuser@remotehost.domain.com
If this works, then proceed to the next step. If it does not, try adding –vv to the ssh command to see additional information about why. I had a problem here and it turned out that port forwarding was disabled at my client. To enable port forwarding, I modified the SSH configuration after which I restarted SSH at the client:
client# vi /etc/ssh/sshd_config
Change the value of AllowTcpForwarding to yes
client# svcadm refresh ssh
Automating SSH Tunnelling
To enable the server to automatically create a reverse tunnel to a given client, the following entry should be added to the /etc/inittab file at the server:
log1:3:respawn:/usr/bin/ssh –nNTx
-R 1514:127.0.0.1:514
someuser@remotehost.domain.com > /dev/null 2>&1
This should occupy a single line only (but has been split over several lines here to increase readability). This someuser and remotehost.domain.com should be replaced with a valid user and client hostname for your system.
Configuring the Log Server
- Download the syslog-ng package from campin.net and copy it to the system that will become the loghost (do not install it yet). You could obtain a copy of syslog-ng from sunfreeware.com but the one from campin.net installs as a proper SMF service on Solaris and is a cleaner package to work with.
- Now remove the existing system-log service from the server. This is required so that syslog-ng can become the primary system logging service on this system.
# svcadm disable system-log
# svccfg delete system-log - Install the syslog-ng package downloaded above:
# pkgadd –d NCsysng-1.6.7-1.pkg - Ensure that it is listed as a valid service
# svcs –a | grep system-log-ng - Edit the configuration file
/usr/local/etc/syslog-ng/syslog-ng.conf(you might like to take a copy of the existing file first) and edit as appropriate:
options {
check_hostname(yes);
keep_hostname(yes);
chain_hostnames(no);
};
source inputs {
internal();
sun-streams("/dev/log");
udp(ip("127.0.0.1"));
tcp(ip("127.0.0.1") max_connections(100) keep-alive(yes));
};
destination logfile {
file("/var/adm/syslog-ng/$HOST/$YEAR/$MONTH/$FACILITY.$YEAR$MONTH$DAY"
owner(root) group(root) perm(0600)
create_dirs(yes) dir_perm(0700));
};
log {
source(inputs);
destination(logfile);
};
- Start the new syslog-ng service
# svcadm restart system-log-ng - Verify that the service is operating correctly (should not be listed in output from command below)
# svcs -xv system-log-ng
Configuring the Log Client
Repeat steps 1-7 above but in Step 5, add the following 2 additional settings to the configuration file:
destination remote {
tcp("127.0.0.1" port(1514));
};
log {
source(inputs);
destination(remote);
};
Note the inclusion of 127.0.0.1 and port(1514) here. This tells the syslog service to write to port 1514 on the local system. This port represents one end of the SSH tunnel and writing to it will result is the log entry being sent to port 514 on the log server, which will result in a new entry on that log server.
Notes
- Some sites recommend using the
keep-alive(yes)setting with the tcp() function above on the server and the client (to avoid SSH hang-ups). However, I found that this is not supported at the client and it caused my system-log-ng service to enter maintenance mode at the client. The reason given bysvcs –xvwas “restarting too quickly” which was very vague. I ended up searching through the manifest files for the system-log-ng service to see what command-line it was actually executing (/usr/local/sbin/syslog-ng) and then running that by hand. It was only then that I saw an error indicating an invalid parameter in my configuration file. - The use of
ip("127.0.0.1")intcp()andudp()in the server configuration ensures that the log server will only listen for local traffic on port 514 on that system. This is more secure.
Useful Links
Best of luck!