Harden the security on an Ubuntu 12.04 LTS server by installing and configuring the following:

1. Install and configure Firewall - ufw
2. Secure shared memory - fstab
3. SSH - Disable root login and change port
4. Protect su by limiting access only to admin group
5. Harden network with sysctl settings
6. Disable Open DNS Recursion – Bind9 DNS
7. Prevent IP Spoofing
8. Harden PHP for security
9. Install and configure Apache application firewall - ModSecurity
10. Protect from DDOS (Denial of Service) attacks with ModEvasive
11. Scan logs and ban suspicious hosts - DenyHosts and Fail2Ban
12. Intrusion Detection - PSAD
13. Check for RootKits - RKHunter and CHKRootKit
14. Scan open Ports - Nmap
15. Analyse system LOG files - LogWatch
16. SELinux - Apparmor
17. Audit your system security - Tiger

If you are looking for a GUI script to install and configure all the steps explained here automatically,
visit How to secure an Ubuntu 12.04 LTS server – Part 2 The GUI Installer script

Requirements:

• Ubuntu 12.04 LTS server with a standard LAMP stack installed.

1. Firewall - UFW

• A good place to start is to install a Firewall.
• UFW – Uncomplicated Firewall is a basic firewall that works very well and easy to configure with its Firewall configuration tool - gufw, or use  Shorewall, fwbuilder, or Firestarter.
• Use Firestarter GUI to configure your firewall or refer to the Ubuntu Server Guide,  UFW manual pages or the Ubuntu UFW community documentation.
• Install UFW and enable, open a terminal window and enter :

sudo apt-get install ufw
sudo ufw enable

• Check the status of the firewall.

sudo ufw ufw status verbose

• Allow SSH and Http services.

sudo ufw allow ssh
sudo ufw allow http

2. Secure shared memory.

• /dev/shm can be used in an attack against a running service, such as httpd. Modify /etc/fstab to make it more secure.
• Open a Terminal Window and enter the following :

sudo vi /etc/fstab

• Add the following line and save. You will need to reboot for this setting to take effect :

tmpfs     /dev/shm     tmpfs     defaults,noexec,nosuid     0     0

3. SSH Hardening – disable root login and change port.

• The easiest way to secure SSH is to disable root login and change the SSH port to something different than the standard port 22.
• Before disabling the root login create a new SSH user and make sure the user belongs to the admin group (see step 4. below regarding the admin group).
• If you change the SSH port also open the new port you have chosen on the firewall and close port 22.
• Open a Terminal Window and enter :

sudo vi /etc/ssh/sshd_config

• Change the following and save.

Port <ENTER YOUR PORT>
Protocol 2
PermitRootLogin no

• Restart SSH server, open a Terminal Window and enter :

sudo /etc/init.d/ssh restart

4. Protect su by limiting access only to admin group.

• To limit the use of su by admin users only we need to create an admin group, then add users and limit the use of su to the admin group.
• Add a admin group to the system and add your own admin username to the group by replacing <YOUR ADMIN USERNAME> below with your admin username.
• Open a terminal window and enter:

sudo groupadd admin
sudo usermod -a -G admin <YOUR ADMIN USERNAME>
sudo dpkg-statoverride --update --add root admin 4750 /bin/su

5. Harden network with sysctl settings.

• The /etc/sysctl.conf file contain all the sysctl settings.
• Prevent source routing of incoming packets and log malformed IP‘s enter the following in a terminal window:

sudo vi /etc/sysctl.conf

• Edit the /etc/sysctl.conf file and un-comment or add the following lines :

# IP Spoofing protection
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1

# Ignore ICMP broadcast requests
net.ipv4.icmp_echo_ignore_broadcasts = 1

# Disable source packet routing
net.ipv4.conf.all.accept_source_route = 0
net.ipv6.conf.all.accept_source_route = 0 
net.ipv4.conf.default.accept_source_route = 0
net.ipv6.conf.default.accept_source_route = 0

# Ignore send redirects
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0

# Block SYN attacks
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 2048
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_syn_retries = 5

# Log Martians
net.ipv4.conf.all.log_martians = 1
net.ipv4.icmp_ignore_bogus_error_responses = 1

# Ignore ICMP redirects
net.ipv4.conf.all.accept_redirects = 0
net.ipv6.conf.all.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0 
net.ipv6.conf.default.accept_redirects = 0

# Ignore Directed pings
net.ipv4.icmp_echo_ignore_all = 1

• To reload sysctl with the latest changes, enter:

sudo sysctl -p

6. Disable Open DNS Recursion - BIND DNS Server.

• Open a Terminal and enter the following :

sudo vi /etc/bind/named.conf.options

• Add the following to the Options section :

recursion no;

• Restart BIND DNS server. Open a Terminal and enter the following :

sudo /etc/init.d/bind9 restart

7. Prevent IP Spoofing.

• Open a Terminal and enter the following :

sudo vi /etc/host.conf

• Add or edit the following lines :

order bind,hosts
nospoof on

8. Harden PHP for security.

• Edit the php.ini file :

sudo vi /etc/php5/apache2/php.ini

• Add or edit the following lines :

disable_functions = exec,system,shell_exec,passthru
register_globals = Off
expose_php = Off
magic_quotes_gpc = On

9. Web Application Firewall - ModSecurity.

• See : How to install apache2 mod_security and mod_evasive on Ubuntu 12.04 LTS server

10. Protect from DDOS (Denial of Service) attacks - ModEvasive

• See : How to install apache2 mod_security and mod_evasive on Ubuntu 12.04 LTS server

11. Scan logs and ban suspicious hosts - DenyHosts and Fail2Ban.

• DenyHosts is a python program that automatically blocks SSH attacks by adding entries to /etc/hosts.deny. DenyHosts will also inform Linux administrators about offending hosts, attacked users and suspicious logins.
• Open a Terminal and enter the following :

sudo apt-get install denyhosts

• After installation edit the configuration file /etc/denyhosts.conf  and change the email, and other settings as required.
• To edit the admin email settings open a terminal window and enter:

sudo vi /etc/denyhosts.conf

• Change the following values as required on your server :

ADMIN_EMAIL = root@localhost
SMTP_HOST = localhost
SMTP_PORT = 25
#SMTP_USERNAME=foo
#SMTP_PASSWORD=bar
SMTP_FROM = DenyHosts nobody@localhost
#SYSLOG_REPORT=YES

• Fail2ban is more advanced than DenyHosts as it extends the log monitoring to other services including SSH, Apache, Courier, FTP, and more.
• Fail2ban scans log files and bans IPs that show the malicious signs – too many password failures, seeking for exploits, etc.
• Generally Fail2Ban then used to update firewall rules to reject the IP addresses for a specified amount of time, although any arbitrary other action could also be configured.
• Out of the box Fail2Ban comes with filters for various services (apache, courier, ftp, ssh, etc).
• Open a Terminal and enter the following :

sudo apt-get install fail2ban

• After installation edit the configuration file /etc/fail2ban/jail.local  and create the filter rules as required.
• To edit the settings open a terminal window and enter:

sudo vi /etc/fail2ban/jail.conf

• Activate all the services you would like fail2ban to monitor by changing enabled = false to enabled = true
• For example if you would like to enable the SSH monitoring and banning jail, find the line below and change enabled from false to true. Thats it.

[ssh]

enabled  = true
port     = ssh
filter   = sshd
logpath  = /var/log/auth.log
maxretry = 3

• If you have selected a non-standard SSH port in step 3 then you need to change the port setting in fail2ban from ssh which by default is port 22, to your new port number, for example if you have chosen 1234 then port = 1234

[ssh]

enabled  = true
port     = <ENTER YOUR SSH PORT NUMBER HERE>
filter   = sshd
logpath  = /var/log/auth.log
maxretry = 3

• If you would like to receive emails from Fail2Ban if hosts are banned change the following line to your email address.

destemail = root@localhost

• and change the following line from :

action = %(action_)s

• to:

action = %(action_mwl)s

• You can also create rule filters for the various services that you would like fail2ban to monitor that is not supplied by default.

sudo vi /etc/fail2ban/jail.local

• Good instructions on how to configure fail2ban and create the various filters can be found on HowtoForge - click here for an example
• When done with the configuration of Fail2Ban restart the service with :

sudo /etc/init.d/fail2ban restart

• You can also check the status with.

sudo fail2ban-client status

12. Intrusion Detection - PSAD.

• Cipherdyne PSAD is a collection of three lightweight system daemons that run on Linux machines and analyze iptables log messages to detect port scans and other suspicious traffic.
• Currently version 2.1 causes errors during install on Ubuntu 12.04, but apparently does work. Version 2.2 resolves these issues but is not yet available on the Ubuntu software repositories. It is recommended to manually compile and install version 2.2 from the source files available on the Ciperdyne website.
• To install the latest version from the source files follow these instruction : How to install PSAD Intrusion Detection on Ubuntu 12.04 LTS server
• OR install the older version from the Ubuntu software repositories, open a Terminal and enter the following :

sudo apt-get install psad

• Then for basic configuration see How to install PSAD Intrusion Detection on Ubuntu 12.04 LTS server and follow from step 2:

13. Check for rootkits - RKHunter and CHKRootKit.

• Both RKHunter and CHKRootkit basically do the same thing – check your system for rootkits. No harm in using both.
• Open a Terminal and enter the following :

sudo apt-get install rkhunter chkrootkit

• To run chkrootkit open a terminal window and enter :

sudo chkrootkit

• To update and run RKHunter. Open a Terminal and enter the following :

sudo rkhunter --update
sudo rkhunter --propupd
sudo rkhunter --check

14. Scan open ports - Nmap.

• Nmap (“Network Mapper”) is a free and open source utility for network discovery and security auditing.
• Open a Terminal and enter the following :

sudo apt-get install nmap

• Scan your system for open ports with :

nmap -v -sT localhost

• SYN scanning with the following :

sudo nmap -v -sS localhost

15. Analyse system LOG files - LogWatch.

• Logwatch is a customizable log analysis system. Logwatch parses through your system’s logs and creates a report analyzing areas that you specify. Logwatch is easy to use and will work right out of the package on most systems.
• Open a Terminal and enter the following :

sudo apt-get install logwatch libdate-manip-perl

• To view logwatch output use less :

sudo logwatch | less

• To email a logwatch report for the past 7 days to an email address, enter the following and replace mail@domain.com with the required email. :

sudo logwatch --mailto mail@domain.com --output mail --format html --range 'between -7 days and today'

16. SELinux - Apparmor.

• National Security Agency (NSA) has taken Linux to the next level with the introduction of Security-Enhanced Linux (SELinux). SELinux takes the existing GNU/Linux operating system and extends it with kernel and user-space modifications to make it bullet-proof.
• More information can be found here. Ubuntu Server Guide – Apparmor
• It is installed by default since Ubuntu 7.04.
• Open a Terminal and enter the following :

sudo apt-get install apparmor apparmor-profiles

• Check to see if things are running :

sudo apparmor_status

17. Audit your system security - Tiger.

• Tiger is a security tool that can be use both as a security audit and intrusion detection system.
• Open a Terminal and enter the following :

sudo apt-get install tiger

• To run tiger enter :

sudo tiger

• All Tiger output can be found in the /var/log/tiger
• To view the tiger security reports, open a Terminal and enter the following :

sudo less /var/log/tiger/security.report.*