Our developer sent me a message in our
Lync Skype for Business and she told me that some of our customer apps are not responding. It seems that the back-end server (A Windows Server 2008 R2 with IIS and SQL 2012) is causing the lag. When I mean lag, I mean it is not responding. When I mean not responding, I mean it crashed.
So I asked our backend developers to check it out, they confirmed that The SQL Server process and full text died. Least of my worries, then MMC is not launching. Then you try it again and again, then this server shows everything that you opened earlier *toinks*.
Lets just restart the server *grin* I really don’t have time for this *grin*. Server did not recover, had to ask the client to start the machine from their VM consoles. All went back to normal, well after a few hours.
Its already past 6PM and I really got a bad feeling about this. Worse, its not our server per se, its in a hosted facility somewhere, a lot of developers / groups had used this server from before and it’s the clients machine. Ouch.
Anyway, I still have to diagnose what is happening or tomorrow we have no server. I miss Azure, sigh. So opening up “MSTSC” and logging into the server:
1. First thing that you do in this situation is to open up your instrumentations, you can use task manager but I use Performance Monitor first so that I can see what is really happening to the entire server in detail . So on your run command, type PERFMON then press enter.
Go to Performance > Data Collector Sets > System > System Performance. Right click and start this baby up. Once finished you will have this report:
2. Now we use Task Manager:
You see a certain “SVCHOST.EXE” is running and is taking a lot of CPU from the server. Something is really odd in this server, its not using a lot of memory but look at the CPU, also check the description.
Hmmmm… The plot thickens…You can actually right click the process and then choose properties, and here is mine:
Okay so its not the real SVCHOST! I now have evidence that this is an actual malware! Copy that location, screenshot it. We will delete it later. Lets go to the details, maybe there’s more info there.
None, so lets proceed.
3. Kill it before it lays eggs!!!
Okay, if you can stop yours great! Delete the file but mine wont die. Nothing is like good ol’fashioned manual malware killing I see.
Experience tells me I cant install an antivirus now and convenience suggest its much harder work to do it, downloading from subscription, running updates yada yada yada.
So download the Process Explorer from Technet: https://technet.microsoft.com/en-us/sysinternals/bb896653.aspx
Lets find it and check what it is doing using process explorer: Sort this list by CPU so that we know which SVC Host.
Go to the properties and navigate to each tab. On the networking “TCP/IP” tab I just found out that this connects to 126.96.36.199 and its using port 52324 and 7777. Checked it out it’s a “Cryptonyte mining pool”. I really don’t care what it is, I know its not for the use of the server so lets kill it.
Then refresh it a couple of times so that we can see if it is still there. Another tool that you can use is the commandline for tasklist.
4. To check if its still a running process using another tool, open an elevated command prompt and type tasklist /SVC /FI “IMAGENAME eq svchost.exe
Not running anymore? Run it a couple of times, just to be sure.
5. Delete it. AKA Zombie Rule #2
While it is not running, we can now delete the actual executable file. Told you you need the address / path / folder where it is. In my case its in the temp directory of the administrator. Which by the way should have been disabled thru policy, but I gotta check later, for now, we just need to delete this. In my case delete the entire temp folder. Deleting it should be our double tap.
So do a couple more checks, lets see if its still there. If not, try to restart and do a couple of checks including performance monitor. Then we can give it a good bill of health, hope I can now back to
steam blizzard origin TFS. Nah who am I kidding – going back to outlook.
6. What we learned so far:
> Never install anything that is not needed in the server: Text editors, freewares, winrar, etc. If you had to edit, edit it outside of the server. If you need to zip it, use the built-in file compressor by Windows.
- > Always turn on firewall: As you can see, if only necessary ports are opened, the malware wouldn’t be able to connect to any host.
> Never use the server for downloading: Shame on you if you do this, never, ever use the server to download anything. Even it is EF updates, no. You do this on your machine, not in the server
> Never put server outside of DMZ and in the internet: There’s a reason why we do not do this. Ever. Use hardware firewalls or internet delivery or content delivery appliances. Don’t put your server bare naked in the internet
> Harden your servers. Security is a must. Get qualified consultants if you cant do this yourself. Test your hardening, create policies.
> Never use the default “ADMINISTRATOR”: Create another administrator account and use a different less administrative account. Disable Administrator if possible.
> GPO: Always manage servers with Group Policies.
> Common Sense
7. Resources that we used:
Process Monitor: https://technet.microsoft.com/en-us/sysinternals/bb896653.aspx
> To the Systinternal Guys, up till today, I use this. Thank you.
Performance Monitor: https://technet.microsoft.com/en-us/library/cc749249.aspx
Data Collector Sets: https://technet.microsoft.com/en-us/library/cc722148.aspx
Zombieland Rule #2 reference : http://www.zombielandrules.com/zombieland-rule-2-double-tap/