The so-called server intrusion or hacking in this guide means that an unauthorized person or program logging into the server for its own purposes and using its computing resources usually has a bad influence.

Disclaimer: If your server is invaded by a state agency like NSA or a criminal group, then you won't notice any problems, and these technologies can't detect their existence.

However, most of the compromised servers are invaded by programs like auto-attack programs or cheap attackers like "script kiddies" and idiot criminals.

Such an attacker will abuse server resources while accessing the server and will not take steps to hide what they are doing.

Symptoms of the compromised server

When servers are invaded by inexperienced attackers or automated attackers, they tend to consume 100% of their resources. They may consume CPU resources to mine digital currencies or send spam, and may also consume bandwidth to launch DoS attacks.

So the first manifestation of the problem is that the server "slows down." This may indicate that the page on the website is open very slowly, or that the email takes a long time to send.

So what should you look at?

Check 1 – Who is currently logged in?

You first need to see who is currently logged in on the server. It is not complicated to find that an attacker logs in to the server.

Its corresponding command is w. Running w will output the following result:

08:32:55up98days, 5:43, 2users, loadaverage: 0.05, 0.03, 0.00

USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT

Root pts/0 113.174.161.1 08:26 0.00s 0.03s 0.02ssshroot@coopeaa12

Root pts/1 78.31.109.1 08:26 0.00s 0.01s 0.00sw

The first IP is the UK IP and the second IP is the Vietnam IP. This is not a good sign.

Stop and take a deep breath, don't panic just kill their SSH connection. Unless you can prevent them from entering the server again, they will come in quickly and kick you off in case you go back again.

Please refer to the section "What to do after the invasion" at the end of this article to see what should be done after finding evidence of the invasion.

The whois command can pick up an IP address and tell you all the information about the organization that the IP is registered with, of course, the information about the country.

Check 2 – Who has logged in?

The Linux server keeps track of which users, which IPs, when they logged in, and how long they logged in. Use the last command to view this information.

The output looks like this:

Root pts/1 78.31.109.1 ThuNov3008:26 still loggedin

Root pts/0 113.174.161.1 ThuNov3008:26 still loggedin

Root pts/1 78.31.109.1 ThuNov3008:24-08:26 (00:01)

Root pts/0 113.174.161.1 WedNov2912:34-12:52 (00:18)

Root pts/0 14.176.196.1 MonNov2713:32-13:53 (00:21)

Here you can see that the UK IP and Vietnam IP alternate, and the top two IPs are still logged in. If you see any unauthorized IP, please refer to the last chapter.

The history after login will be recorded in the binary /var/log/wtmp file (LCTT translation: the author should be wrong here, modified according to the actual situation), so it is easy to delete. Usually the attacker will delete this file directly to cover up their attack behavior. Therefore, if you run the last command and only see your current login, then this is a bad signal.

If you don't have a login history, be careful and continue to pay attention to other clues about the invasion.

Check 3 – Review Command History

Attackers at this level usually don't pay attention to masking the history of commands, so running the history command will show everything they have done. Be sure to use the wget or curl commands to download unconventional software like spam bots or mining programs.

The command history is stored in the ~/.bash_history file, so some attackers will delete the file to cover what they are doing. As with the login history, if you run the history command without outputting anything, it means that the history file has been deleted. This is also a bad signal, you need to check the server very carefully. (LCTT, if there is no command history, it may be your configuration error.)

Check 4 – Which processes are consuming CPU?

The type of attackers you often encounter are usually not very likely to cover up what they do. They will run some processes that are particularly CPU intensive. It is easy to find these processes. Just run top and look at the first few processes.

This also shows those attackers who are not logged in. For example, someone might be using an unprotected mail script to send spam.

If you don't know the top process, then you can Google to see the process name, or use losf and strace to see what it does.

Using these tools, the first step is to copy the PID of the process from top and then run:

strace-pPID

This will show all system calls called by the process. It produces a lot of content, but this information can tell you what the process is doing.

Lsof -pPID

This program will list the files opened by the process. By looking at the files it accesses, you can understand what it is doing.

Check 5 – check all system processes

An unauthorized process that consumes less CPU may not appear in the top, but it can still be listed by ps. The command ps auxf will display enough clear information.

You need to check every process you don't know. Running ps often (this is a good habit) can help you find strange processes.

Check 6 – Check the network usage of the process

Iftop functions like top, it arranges the processes that display network data and their source and destination addresses. Processes like DoS attacks or spam bots are easy to display at the very top of the list.

Check 7 – Which processes are listening for network connections?

Usually the attacker will install a backdoor program to listen to the network port to accept the command. The process waits for no CPU and bandwidth consumption, so it is not easy to find through commands like top.

The lsof and netstat commands list all networked processes. I usually let them take the following parameters:

Lsof-i

Netstat-plunt

You need to be aware of processes that are in the LISTEN and ESTABLISHED states, which are either waiting for a connection (LISTEN) or are already connected (ESTABLISHED). If you encounter a process you don't recognize, use strace and lsof to see what they are doing.

First of all, don't be nervous, especially when the attacker is logged in. You need to take control of the machine before the attacker is alert to the fact that you have found him. If he finds that you have already discovered him, then he may lock you out of the server and then start destroying the corpse.

If your skills are not so good then shut down directly. You can run one of the two commands shutdown -h now or systemctl poweroff on the server. You can also shut down the server by logging into the host provider's control panel. After shutting down, you can start to configure the firewall or consult the supplier's comments.

If you are confident in yourself and your hosting provider also provides an upstream firewall, then you only need to create and enable the following two rules:

Only allow login to SSH from your IP address.

Anything other than this is banned, not just SSH, but also any protocol on any port.

This will immediately shut down the attacker's SSH session, leaving only the server you can access.

If you are unable to access the upstream firewall, then you will need to create and enable these firewall policies on the server itself, then use the kill command to shut down the attacker's SSH session after the firewall rules have taken effect. (LCTT: Local firewall rules may not block an established SSH session, so for insurance purposes, you need to kill the session manually.)

Finally, there is a way to log in to the server via an out-of-band connection such as a serial console, and then stop the network function via systemctl stop network.service. This will shut down the network connections on all servers so you can slowly configure those firewall rules.

After regaining control of the server, don't think that everything will be fine.

Don't try to fix this server and then use it. You never know what the attacker has done, so you can never guarantee that this server is still secure.

The best way is to copy all the data and reinstall the system. (LCTT Annotation: Your program is not trusted at this time, but the data is generally fine.)