This blog post is the second installment of a series I want to use to cover lessons learned and interesting observations from my honeypots. These honeypots are geographically dispersed and have been running for a few years. Hopefully, this and future posts will add some value for someone.
Observations
While looking over some of my honeypot logs, I observed the download of an open-source port scanner named masscan. In the documentation for this tool, the author claims it can scan the entire internet in less than five minutes. I can see the appeal for attackers, and naturally, the logs show evidence of the attacker attempting to use masscan to find other targets of opportunity.
Software Installation
The attacker first tries to install masscan using apt (Advanced Package Tool), a package manager for Debian-based Linux distributions like Ubuntu. This particular honeypot is a RedHat-based distribution, so when they switched to using yum, it appears to work, but is entirely simulated by the honeypot.
root@redacted:~# apt install masscan -bash: apt: command not found root@redacted:~# yum install masscan Loaded plugins: changelog, kernel-module, ovl, priorities, tsflags, versionlock 286 packages excluded due to repository priority protections Resolving Dependencies --> Running transaction check ---> Package masscan.0.29-5 x86_64.7.el7 will be installed --> Finished Dependency Resolution Beginning Kernel Module Plugin Finished Kernel Module Plugin <abbreviated> Installed: masscan.x86_64 2:0.29-5-7.el7 Complete!
Below is another instance where the attacker attempts to install additional software.
root@redacted:~# yum install git gcc make libpcap-devel
Loaded plugins: changelog, kernel-module, ovl, priorities, tsflags, versionlock
212 packages excluded due to repository priority protections
Resolving Dependencies
--> Running transaction check
---> Package libpcapdevel.0.4-8 x86_64.13.el7 will be installed
---> Package git.1.6-4 x86_64.13.el7 will be installed
---> Package gcc.1.23-7 x86_64.14.el7 will be installed
---> Package make.1.19-10 x86_64.9.el7 will be installed
This was the last observed attempt to install software using the native package manager and software library.
root@redacted:~# sudo yum groupinstall -y "Development Tools" Loaded plugins: changelog, kernel-module, ovl, priorities, tsflags, versionlock ovl: Error while doing RPMdb copy-up: [Errno 13] Permission denied: '/var/lib/rpm/.dbenv.lock' You need to be root to perform this command. root@redacted:~# yum groupinstall -y "Development Tools" Loaded plugins: changelog, kernel-module, ovl, priorities, tsflags, versionlock ovl: Error while doing RPMdb copy-up: [Errno 13] Permission denied: '/var/lib/rpm/.dbenv.lock' You need to be root to perform this command.
Software Download
I want the ability to study any tradecraft or tools an attacker leverages once they gain unauthorized access to a system. Therefore, the honeypots I deployed all allow attackers to download tools. The research value, or if you prefer, threat intelligence, has been more valuable than not. Below, the attacker is trying to clone a git repository of masscan. We can presume this is because the previous installation attempts did not work. Naturally, since the git installation failed, the attacker cannot clone the repository.
root@redacted:~# git clone https://github.com/robertdavidgraham/masscan
-bash: git: command not found
The last observed download attempt uses wget to retrieve a compressed archive of masscan. Nothing was logged after extraction of the archive.
root@redacted:~# wget https://github.com/robertdavidgraham/masscan/archive/1.0.3.tar.gz
--2020-11-26 08:06:03-- https://github.com/robertdavidgraham/masscan/archive/1.0.3.tar.gz
Connecting to github.com:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/x-gzip]
Saving to: `/root/1.0.3.tar.gz'
root@redacted:~# tar xvf 1.0.3.tar.gz
masscan-1.0.3
Actions on Objective
Since the honeypot simulated the attackers software installation attempts, when they try to use masscan it does not work. However, we gain some threat intelligence in the form of the command line execution of masscan. The attacker appears to want to use this particular system to scan the internet in an attempt to find systems with port 3389 open. The port is associated with the Microsoft Remote Desktop Protocol (RDP).
root@redacted:~# masscan --rate=900000 0.0.0.0/0 --exclude 255.255.255.255 -p3389 -oL RESULT_IP.txt -bash: masscan: command not found
Interestingly, right before the attacker disconnected, they tried using zmap, another fast internet scanner, to scan the internet for servers that have port 22 open. In the zmap command line, the output from the scan result is saved to a file called vnc.txt, which is likely a misdirection attempt because port 22 is generally associated with the Secure Shell (SSH) server.
root@redacted:~# zmap -p 22 -o vnc.txt -bash: zmap: command not found
Mitigations
So what are the takeaways from the observations above? Well, as it relates to a publicly facing secure shell (SSH) Server, there are several. Here is a TLDR list:
- Use Strong Passwords
- Enable Public Key Authentication
- Enable the AllowUsers directive
- Disallow Remote root Login
- Use File Integrity Monitoring (FIM)
Use Strong Passwords
The honeypots I have deployed are all designed using weak SSH password authentication. I want to see what the attacker does after they get in, so having a military-grade password with two-factor authentication (2FA) will limit my research opportunities. If, for some reason, you have to enable password authentication on your production SSH servers, then at least use a strong password that is not a dictionary word, not less than eight characters, and not included in a password list. I recommend you read NIST Special Publication 800-63B, Section 5.1.1.2, Memorized Secret Verifiers. When I query the passwords used in my honeypot data, it is easy to see that the attackers are using passwords that come from simple dictionary words, password leaks, and reports on the most commonly used passwords. I have included a screenshot below of the top 10 passwords I have seen used against my honeypots. Hopefully, you are not using any of them anywhere.
Enable Public Key Authentication
Public Key Authentication with SSH is probably one of the best ways to mitigate brute-force password attacks. I have seen threat actors gain access to the honeypots and try to enable Public Key authentication using their own key pair. I should mention that you also need to disable password authentication. The following configuration is what is recommended for your sshd_config file. There are some follow on steps like generating key pairs.
# /etc/ssh/sshd_config # See the sshd_config(5) manpage for details PubkeyAuthentication yes PasswordAuthentication no
Enable the AllowUsers directive
The AllowUsers directive is another good mitigation option if you have good accountability for authorized users and their source connections. For example, you would use the AllowUsers directive to restrict the users that can remotely connect using SSH. Keep in mind that this setting will not prevent unauthorized access if a user has a weak password. However, if you include the remote host, it limits the exposure. If you have many users, you can use the AllowGroups directive; unfortunately, you can not scope by the remote host. I have included a sample configuration below.
# /etc/ssh/sshd_config # See the sshd_config(5) manpage for details AllowUsers veronica@remote.fakelabs.org AllowGroups ssh_users remote_users
Disallow Remote root Login
Please disable remote root logins. All administrators should have an unprivileged account used for remote SSH logins, and then if they need privileged access, they should use sudo. I would recommend you also look at this NIST publication titled Security of Interactive and Automated Access Management Using Secure Shell (SSH). The configuration to disable root login is below.
# /etc/ssh/sshd_config # See the sshd_config(5) manpage for details PermitRootLogin no
There are two images below. The first depicts the aggregated top ten usernames used by attackers when attempting to log into my honeypots. There are over six million login events and about 70 percent of them were attempts using root as the username.
This second image is a pie chart for another perspective on the same data. So if you have an SSH server on the internet and you allow remote root login using password authentication, I strongly urge you to consider adopting some of the recommendations in the NIST document or your server could end up as a launch point for masscan or a number of other adversarial objectives.
Use File Integrity Monitoring (FIM)
In the observations above you can see that the attacker tries to install software using different techniques, which creates all kinds of noise for file integrity monitoring (FIM) software. This did not strike me as good offensive tradecraft, but the attacker probably assumes most admins are not using FIM or even paying attention to what software is installed on their systems. There are a variety of vendors and open-source solutions that can provide file integrity monitoring. I would choose one that fits your use case, tune it for your operational environment and make sure the alerts it generates are actionable. Remember, OS updates will create a lot of noise so have a process in place to handle those situations.
Configure Egress Filtering
You can use the host-based firewall to block outbound connections not strictly necessary, which can help reduce the risk that an attacker uses your system as a platform for launching an internet-wide scanning campaign. I recommend a default deny outbound configuration and then allowing outbound connections to remote hosts that are necessary. Most stateful firewalls have a related connection feature to simplify some of the concerns with servers that have to respond to any number of unique remote hosts. Egress filtering may not be feasible for every organization or individual, but worth considering, especially if your server has a limited scope of outbound connections. If an attacker has privileged access to a system, then a solution could be to do the filtering at a core or edge routing device.
Hopefully, you got some value from this article. Stay tuned for more in this series. If you have any ideas or suggestions feel free to reach out to me.
Thanks for reading.