A case of svchost.exe high CPU usage

Our developer sent me a message in our Lync Skype for Business and she told me that some of our customer apps are not responding. It seems that the back-end server (A Windows Server 2008 R2 with IIS and SQL 2012) is causing the lag. When I mean lag, I mean it is not responding. When I mean not responding, I mean it crashed.

So I asked our backend developers to check it out, they confirmed that The SQL Server process and full text died. Least of my worries, then MMC is not launching. Then you try it again and again, then this server shows everything that you opened earlier *toinks*. 

Lets just restart the server *grin* I really don’t have time for this *grin*. Server did not recover, had to ask the client to start the machine from their VM consoles. All went back to normal, well after a few hours.

Its already past 6PM and I really got a bad feeling about this. Worse, its not our server per se, its in a hosted facility somewhere, a lot of developers / groups had used this server from before and it’s the clients machine. Ouch.

Anyway, I still have to diagnose what is happening or tomorrow we have no server. I miss Azure, sigh. So opening up “MSTSC” and logging into the server:

1. First thing that you do in this situation is to open up your instrumentations, you can use task manager but I use Performance Monitor first so that I can see what is really happening to the entire server in detail . So on your run command, type PERFMON then press enter.

Go to Performance > Data Collector Sets > System > System Performance. Right click and start this baby up. Once finished you will have this report:

SVCHOSTSo as you can see, I have a busy CPU. At 100% utilization, something is running or should I say hogging the CPU of this server.

2. Now we use Task Manager:

SVCHOST2

You see a certain “SVCHOST.EXE” is running and is taking a lot of CPU from the server. Something is really odd in this server, its not using a lot of memory but look at the CPU, also check the description.

Hmmmm… The plot thickens…You can actually right click the process and then choose properties, and here is mine:

SVCHOST3

Okay so its not the real SVCHOST! I now have evidence that this is an actual malware! Copy that location, screenshot it. We will delete it later. Lets go to the details, maybe there’s more info there.

 SVCHOST4

None, so lets proceed.

3. Kill it before it lays eggs!!!

Okay, if you can stop yours great! Delete the file but mine wont die. Nothing is like good ol’fashioned manual malware killing I see.

Experience tells me I cant install an antivirus now and convenience suggest its much harder work to do it, downloading from subscription, running updates yada yada yada.

So download the Process Explorer from Technet: https://technet.microsoft.com/en-us/sysinternals/bb896653.aspx

SVCHOST5

Lets find it and check what it is doing using process explorer: Sort this list by CPU so that we know which SVC Host.

SVCHOST6

Go to the properties and navigate to each tab. On the networking “TCP/IP” tab I just found out that this connects to 77.72.133.157 and its using port 52324 and 7777. Checked it out it’s a “Cryptonyte mining pool”. I really don’t care what it is, I know its not for the use of the server so lets kill it.

SVCHOST7

Then refresh it a couple of times so that we can see if it is still there. Another tool that you can use is the commandline for tasklist.

4. To check if its still a running process using another tool, open an elevated command prompt and type tasklist /SVC /FI “IMAGENAME eq svchost.exe

SVCHOST9

Not running anymore? Run it a couple of times, just to be sure.

5. Delete it. AKA Zombie Rule #2

While it is not running, we can now delete the actual executable file. Told you you need the address / path / folder where it is. In my case its in the temp directory of the administrator. Which by the way should have been disabled thru policy, but I gotta check later, for now, we just need to delete this. In my case delete the entire temp folder. Deleting it should be our double tap.

So do a couple more checks, lets see if its still there. If not, try to restart and do a couple of checks including performance monitor. Then we can give it a good bill of health, hope I can now back to steam blizzard origin TFS. Nah who am I kidding – going back to outlook.

6. What we learned so far:

> Never install anything that is not needed in the server: Text editors, freewares, winrar, etc. If you had to edit, edit it outside of the server. If you need to zip it, use the built-in file compressor by Windows.

  • > Always turn on firewall: As you can see, if only necessary ports are opened, the malware wouldn’t be able to connect to any host.

> Never use the server for downloading: Shame on you if you do this, never, ever use the server to download anything. Even it is EF updates, no. You do this on your machine, not in the server

> Never put server outside of DMZ and in the internet: There’s a reason why we do not do this. Ever. Use hardware firewalls or internet delivery or content delivery appliances. Don’t put your server bare naked in the internet

> Harden your servers. Security is a must. Get qualified consultants if you cant do this yourself. Test your hardening, create policies.

> Never use the default “ADMINISTRATOR”: Create another administrator account and use a different less administrative account. Disable Administrator if possible.

> GPO: Always manage servers with Group Policies.

> Common Sense

 

7. Resources that we used:

MSTSC: https://technet.microsoft.com/en-us/library/cc753907.aspx

Process Monitor: https://technet.microsoft.com/en-us/sysinternals/bb896653.aspx

    > To the Systinternal Guys, up till today, I use this. Thank you.

Performance Monitor: https://technet.microsoft.com/en-us/library/cc749249.aspx

Data Collector Sets: https://technet.microsoft.com/en-us/library/cc722148.aspx

Tasklist: https://technet.microsoft.com/en-us/library/bb491010.aspx

Zombieland Rule #2 reference : http://www.zombielandrules.com/zombieland-rule-2-double-tap/

Advertisements

Windows Server 2008 R2 DHCP Error 1046–not authorized

Its late last night and I am doing my usual labs and I just rebuilt my entire lab using Server Core. It’s a temporary lab for a customer POC that I will ship out tomorrow and with that I have combined AD + DNS + DHCP role in one VM. So here’s the story:

  1. Successfully Installed AD and DNS, thru DCPromo /unattend:c:\DCUnnattend.txt – good.
  2. Installed DHCP server role using my trusted OCsetup and it installed flawlessly.
  3. Used netsh exec to load my usual DHCP configuration, same one that I have been using so far on any server install that I have for labs. Great.
  4. Now its not giving any IP addresses to other client VM’s – now why?
    I did some troubleshooting and found an error on the event log: Error 1046! (yea used wevtutil and viewing it thru event viewer but that’s for another story Smile)
      So what is Event 1046? TechNet says:

    Event ID 1046 — DHCP General Availability
    Applies To: Windows Server 2008

    General availability of the Dynamic Host Configuration Protocol (DHCP) server refers to its ability to service clients. General availability depends on:

    1. Proper authorization of the DHCP server
    2. Presence of Active Directory Domain Services
    3. Successful loading of the DHCP dynamic-link libraries (DLLs)

    http://technet.microsoft.com/en-us/library/cc726914(WS.10).aspx

    Wait I say to myself, isn’t this server core installation is also the AD Server of this domain? Because I know for a fact that if DHCP role is installed in a computer running the AD also, it does not have to be Authorized!

    Okay, lets authorize it via CMD using this:

    netsh dhcp server serverID initiate auth

    or

    netsh dhcp server initiate auth

    Oh this will be a long night. So saved the server VM, snapshot, shutdown and then I tried it again now using a template that is not from Core (Windows Server 2008 R2 Full + AD + DNS installed from WDS, another story for later). Tried same steps and there, Then me going to the DHCP snap-in on the server manager, I confirmed that the DHCP server still appears unauthorized. Out of desperation, I Restarted my server and while doing so reading this: http://support.microsoft.com/kb/279908 Not very helpful though. Ok out of frustration, I authorized my DHCP on the MMC Snap-in and everything works fine!

    Having that, I suspect I am using the wrong command to initiate the authorization. Okay going back to basics, as one of my very very dark mentor before when I was still on the ISP business “RTFM”!

    So going to http://technet.microsoft.com/en-us/library/dd379483(WS.10).aspx

    image

    Wait, what?! Nah, lets see the complete manual of netsh dhcp here: http://technet.microsoft.com/en-us/library/bb490941.aspx

    image

    So lets try it out, shut down the Server Fulls and restored the Server Core VM’s fired up this command:

    Netsh DCHP add server <fqdn> 192.168.1.2

    and it works, its alive! its alive!

    Okay lesson learned, if you are installing DHCP role with ADDS + DNS role, make sure that you add your fqdn and ip to the list of authorized servers in active directory. Snap-in authorization does this for us (I think) but if you are now using and adopting Server Core for the entire enterprise make sure that you fire up that command. Thus a new entry in my step-by-step commands on deploying Windows Server.

    Now back to my servers! Cheers! And oh Good morning!

    Enabling Audit Events for Windows Firewall with Advanced Security

    If you are following the TechNet Article http://technet.microsoft.com/en-us/library/ff428143(WS.10).aspx you may notice that if you use :

    auditpol.exe /list /category:"Policy Change"  or any category, this throws an error 0x00000057 that the parameter is incorrect when used with Windows Server 2008 R2 and Windows 8 Beta (I have not checked with Vista and XP).

    When /get is used rather than /list, there is no error and it displays correctly. Full command used is as follows:

    auditpol.exe /get /category:"Policy Change"

    image

    Using Remote Server Administration Tool for Hyper-V on a WorkGroup Environment

    I have a couple of emails for a few months now since I last posted about using RSAT for Windows 7 to remotely administer Windows Server 2008 R2 with Hyper-V – mostly asking how to configure this on a WorkGroup environment (or without a Domain).

    So first we must download and install RSAT

    Bits are here: http://www.microsoft.com/download/en/details.aspx?id=7887

    image

    After downloading, run the update:

    image\\

     

    image

    Accept the license terms:

    image

    image

    Click close when the installation is complete:

     

    image

    To use RSAT, go to the control panel, Programs and Features and turn Windows Features on or off and check hyper-v tools.

    image

    image

    Check if the Hyper-v Manager is installed on your administrative tools

    image

    In order to use RSAT on workgroup, I use the Hyper-V Remote Management Configuration Utility available at MSDN.

     

    image

    http://archive.msdn.microsoft.com/HVRemote/Release/ProjectReleases.aspx?ReleaseId=3084

    Agreed!

    image

    On download of the script,

    On the Client Machine, open your Command Prompt in elevated context.

    image

    Then run cscript hvremote.swf /mmc:enable

    image

    then cscript hvremote.wsf /anondcom:grant

    image

    Now on the server that has the Hyper-V role, run also the Command Prompt as an Administrator.

    image

    add a user that would be used to connect from the client to the server. The user must have the same username in the Client and in this server. Another description is that we have to clone the user (username and password) in the client to the server.  Use Net User command:

    net user <user> <password> /add

    image

    Then execute the “cscript hvremote.swf /add:<user>:”

    image

    Reboot both the client and server!

    After rebooting.. Open your Hyper-V manager on the client and then click Connect to Server:

    image

    Connect to another computer and type the server name of the Hyper-V server, click OK.

    image

    That’s it. You now have your Remote Hyper-V manager:

    image

    To summarize:

    1. Download and install RSAT

    2. Clone Users

    3. Download and use hvremote.wsf on both Server and Client

    4. Enjoy!

    Hyper-V Windows clients not using Dynamic Memory on fresh install

    1

    Ok if you encounter a freshly installed Windows 7 client and are not using dynamic memory on hyper-V even though you have already set it in options that it should – try upgrading the Integration Services.

    Just go to actions then choose  “Insert Integrations Services Setup Disk”.

    1b

    Then when autoplay comes, just run.

    1c

    Click ok on UAC

    1d

    Click okay.

    2

    While installing, grab a beer and walk away.

    3

    Restart when appropriate.

    4

    Will install some updates….

    5

    After booting.. WHALAHH!

    All clients are now running dynamic memory!

    ADFS_SP2010_DN

     

     

     

    This post is dedicated to the one that called me earlier while I am at work asking about this so I figured, lets just post this overdue draft! Good luck tomorrow!

    SharePoint 2010 Stand Alone Installation – Error on Step 2: Microsoft.SharePoint.SPException: User cannot be found

    A thing worth posting, Note to SELF: If you are installing SharePoint 2010 on a standalone configuration, make sure that the machine where you are installing MUST HAVE ACCESS TO THE DOMAIN CONTROLLER where it is a member of.

    For the one that needs more info, here is the stack trace from the PSDiagnosticsLog:

    Exception: Microsoft.SharePoint.SPException: User cannot be found.
       at Microsoft.SharePoint.Administration.SPSiteCollection.Add(SPContentDatabase database, SPSiteSubscription siteSubscription, String siteUrl, String title, String description, UInt32 nLCID, String webTemplate, String ownerLogin, String ownerName, String ownerEmail, String secondaryContactLogin, String secondaryContactName, String secondaryContactEmail, String quotaTemplate, String sscRootWebUrl, Boolean useHostHeaderAsSiteName)
       at Microsoft.SharePoint.Administration.SPSiteCollection.Add(SPSiteSubscription siteSubscription, String siteUrl, String title, String description, UInt32 nLCID, String webTemplate, String ownerLogin, String ownerName, String ownerEmail, String secondaryContactLogin, String secondaryContactName, String secondaryContactEmail, Boolean useHostHeaderAsSiteName)

    P.S. No screenshot for now, as I have fixed it and would not be possible to replicate, besides its 1AM already, geez!

    Windows 7 and Server 2008 R2 Service Pack 1

    As you may already knew, the Service Pack 1 was already released for Windows Server 2008 R2 and Windows 7 clients as well.

    I had a few discussions with the other IT people and asking me, why am I installing the SP1. Generally speaking, why I want to install from a disk that has the service pack in it already is because I wanted to keep my PC as updated as possible and instead of me doing an upgrade to a machine that I will update anyways, I’ll just start from an image that has the service pack – minus the head ache of updating one’s machine.

     

    On the server side, I had a chance before to demo one of its feature the “Dynamic Memory” over a TechNet session here in Makati and if you are looking into Virtualization, you may want that feature – and its free with the update, why not yea?

    More info at this TechNet link: http://technet.microsoft.com/en-us/library/ff817622(WS.10).aspx and here http://www.microsoft.com/oem/en/downloads/pages/windows_7_sp1.aspx