Microsoft ATA: Encryption downgrade activity

If you have Microsoft Advanced Threat Analytics (ATA) within your environment, you may have seen the following warning:

Encryption downgrade activity
The encryption method of the ETYPE_INFO field of KRB_ERR message from x computers has been downgraded based on previously learned behavior. This may be a result of a Skeleton Key on x domain controllers.

I work for quite a large firm, and we were getting these for a handful of computers and accounts.

Initially we looked through Microsoft’s somewhat useful Advanced Threat Analytics suspicious activity guide. However under the heading “Encryption downgrade activity” it somewhat puts the fear of God into you. Indeed, after we did our investigations with the tools provided in that article, we found nothing. So what was going on?

We raised an incident with Microsoft and eventually found that these accounts have passwords that had not been updated for a very long time (before 2010).

Essentially:

When a client negotiates with the KDC (a Domain Controller), it will send the client principal name and realm (eg username/domain.local)
The KDC will look up the client in Active Directory, specifically the supplementalCredentials field in AD.
Should the user be missing the Primary:Kerberos-Newer-Keys property, then the KDC will negotiate a lower encryption with the client, and the warning will be raised in ATA.

If your domain functional level is Windows 2008 or higher (which quite frankly it should be by now!), reset the users password, as 2008 DFL introduces the Primary:Kerberos-Newer-Keys property. By resetting the users password, this *should* set the Primary:Kerberos-Newer-Keys property, and the error should go away in ATA.

If you want to test if you have this issue, just try logging on with one of these impacted user accounts to a domain joined PC, or any other system that authenticates via Kerberos. Just use a bogus password for the account, don’t worry about it if you don’t know the accounts password because the Kerberos negotiation happens before authentication. If your failed logon raises an alert in ATA, then that account needs to change it’s password.

TL;DR: Just reset the password of the impacted users.

Filtering the Windows Security Log

Filtering the Windows Security Log can be a pain in the arse if you don’t know how to use the XML filtering.

Here are two filtering options I use a lot in order to glean some information…

Filter Security Log by username:

Find all successful and unsuccessful logons from a particular user…

<QueryList>
<Query Id="0" Path="Security">
<Select Path="Security">* [EventData[Data[@Name='TargetUserName']='USERNAME']]</Select>
</Query>
</QueryList>

Get NTLM Logons

Find all logons that were done with NTLM (good for finding the old crap on your network)

<QueryList>
<Query Id="0" Path="Security">
<Select Path="Security">* [EventData[Data[@Name='AuthenticationPackageName']='NTLM']]</Select>
</Query>
</QueryList>

Visio org chart: entries not in the organization

If you are trying to create an organisation chart in Visio using Exchange, and getting the error:

Your organization data contains 1 entries that are not in the organization. Do you want to include them in your drawing?

To fix this, turn off caching in Outlook.

No idea why this works, haven’t looked much into it. If you do know of a more elegant way to solve this issue, leave a comment!

2012R2 Core and Hyper-V on Intel Nuc

As I am currently between jobs (read: Unemployed), I thought I would build my own mini lab to help practice the timeless art of being a Windows System Administrator. To that end I have purchased a Intel i5 NUC (D54250WYKH). I chose this as a lab computer because my current place doesn’t have much space, and while it may not have incredible performance, it’s reasonably powerful enough to run a number of virtualised servers, provided you have enough ram and an SSD.

This post will cover installing Windows Server 2012 R2 Core with Hyper-V on an Intel NUC in a workgroup environment (non domain joined).

Initial Notes

According to Intel the D54250WYKH does not come with AMT, so unfortunately you’ll have to connect a keyboard and monitor at least initially as Windows Server 2012 R2 does not come with the nic drivers ootb.
The steps below rely on using both the command prompt and PowerShell. To enter PowerShell from the command prompt in Windows Server Core, just type: powershell. To exit PowerShell back to the command prompt, just type: exit.
The client I am using is a Windows 8.1 Pro computer

Ingredients

You’ll need the following:

An Intel NUC (D54250WYKH), and the following drivers:
1. Ethernet Adapter (you will need to modify the driver to allow installation on Windows Server 2012 R2, follow these steps)
2. Chipset Device Software (you will need to extract the driver from the installer, you can do that by running SetupChipset.exe with the -extract argument)
3. Management Engine Driver
Windows 2012 R2 on a bootable USB
Hyper-V Remote Management Configuration Utility (HVRemote)

Steps

Install Windows Server 2012 R2 Core.
Give the server a name, in PowerShell run:
Rename-Computer -NewName server-name -Restart
Because we are installing modified drivers, we need to disable Windows restrictions around unsigned drivers, run the following commands in command prompt:
bcdedit /set LOADOPTIONS DISABLE_INTEGRITY_CHECKS
bcdedit /set TESTSIGNING ON
bcdedit /set nointegritychecks ON
You will now need to reboot the server
Install the drivers using pnputil in command prompt:
pnputil.exe -i -a D:intel-driversnicRarSFX0PRO1000Winx64NDIS64e1d64x64.inf
You will be prompted with Windows can’t verify the publisher of this driver software, just Install this driver software anyway
pnputil.exe -i -a D:intel-driverssmbusChipset_SMBus.inf
pnputil.exe -i -a D:intel-driversintelMEDriversMEIheci.inf
Now you should have network connectivity, lets enable remote desktop connections, file sharing and ping with powershell:
Set-ItemProperty -Path ‘HKLM:SystemCurrentControlSetControlTerminal Server’ -Name “fDenyTSConnections” -Value 0
Enable-NetFirewallRule -DisplayGroup “Remote Desktop”
Enable-NetFirewallRule –DisplayGroup “File and Printer Sharing”
Install Hyper-V and allow remote management (this is where you will need the hvremote script):
Install HyperV: Install-WindowsFeature –Name Hyper-V -IncludeManagementTools –Restart
cscript hvremote.wsf /mode:server /add:administrator
cscript hvremote.wsf /mode:server /firewallhypervmgmt:enable
cscript hvremote.wsf /mode:client /anondcom:grant
If you are remoting to this server you will need to whitelist clients using the following command:
winrm set winrm/config/client @{TrustedHosts=”client-name”}
On your Hyper-V client
1. Make sure you have PowerShell and the HyperV Management Tools installed
2. Open Component Services
3. Expand Component Services, then Computers
4. Right click on My Computer and click on Properties.
5. In the COM Security tab, click Edit Limits under Access Permissions.
6. Enable Remote Access for Anonymous Logon
7. In PowerShell run:
  Set-Item WSMAN:LocalhostClientTrustedHosts -Value * -Force # *
8. In command prompt (replace the server-name and password):
  cmdkey /add:server-name /user:ADMINISTRATOR /pass:password

Closing Notes

You should now be able to:

Connect your client HyperV manager to the server you created
Enter remote PowerShell sessions via Enter-PSSession -ComputerName
Ping and File Share with your server

There will be some devices that aren’t installed, these are 2 Audio Devices and the Infrared sensor on the front of the Nuc, I have not bothered to install those drivers.

If you have any comments, please leave them! I would appreciate the feedback, and I would also love to know if I have missed anything or whether I should have done something differently.

user profile not loading

Have you got a windows user profile that isn’t loading correctly?

Have you checked:

Is the profile loaded into the registry already? (check regedit under HKEY_USERS)
Has the user got write access to the registry hive? (can be an issue in default profile / roaming profile scenarios)

Well if it’s not those things! Open regedit and head to:

HKLM\Software\Microsoft\Windows NT\CurrentVersion\ProfileList

Find the folder that contains your user profile information (it’ll be in the key ProfileImagePath) and then delete this folder.

32bit ODBC drivers

Not seeing all the ODBC drivers you thought should be installed? You may be using the 64bit ODBC Data Source Administrator.

Try the 32bit one. It’s located here:

C:\WINDOWS\SysWOW64\odbcad32.exe

rds and citrix printing issues

The major issue with printing in Citrix boils down to the Client Side Rendering Print Provider!

Do you add printers to your remote sessions via group policy / have users add printers into their sessions using the standard windows add printer wizard? Well, if you haven’t experienced many issues yet, just you wait!

Here are some of the issues you will see that all stem from the Client Side Rendering Print Provider in RDS / Citrix (Though I can’t see why this also wouldn’t happen on a PC either).

UAC Prompt when trying to delete a session printer

Here’s a video of a standard user trying to delete a printer they added to their session.

As you can see they receive a UAC prompt! (That shouldn’t happen). So how do I resolve it?

Go to the Client Side Rendering Print Provider registry key: HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Print\Providers\Client Side Rendering Print Provider
In the users key, expand Printers then Connections. You will need to find out the SID of the user that has the issue, there are many scripts online that can resolve a username to a SID.
Find the print connection that is having the issue, and delete the key. As you can see this key can store many connections that the user does not currently have. This can cause this part of the registry to become really large!
Restart the print spooler and try to delete the connection again.
Done!

In the video above, I don’t give the spooler enough time to restart before deleting the printer, hence why I receiver the error when trying to delete the printer the first time after I restart the spooler.

I understand you could also remove the connection from HKCUPrintersConnections, however that would just be a workaround, I’m trying to tell you why you actually get the issue in the first place!

Printer properties cannot be displayed. Operation could not be completed (error 0x00000002)

Here’s a video of the x00000002 error. Because the Client side rendering print provider caches the printers by ,,printservername,name, you can just connect to your print server using an IP or a fully qualified name and printing will work. Same printer, same print server, different results!

This is what lead to confusion for the communications staff where I work, as they thought it was an issue with the cname to the print server, because: cnameprinter didn’t work but fullyqualifiednameprintername did! This had nothing to do with the cname, and nothing to do with communications, but it just had to do with the cached entries in the Client Side Rendering Print Provider.

Case-sensitive printing

There are some applications we use internally that seem to care what case a printer is added. Think of the following scenario:

User adds a printer with lower case, and logs off server 1. The connection will be cached in the Client Side Rendering cache in lower case on server 1.
User deletes the lower case printer on server 2.
User adds the printer this time in upper case on server 1. The client side rendering key will not update to upper case, it will retain the lower case connection.

Now this is no big issue, as most programs don’t care, well… some do!

I’ve seen enough! How do I fix it?

Download and install Microsoft KB2778831

It’s not part of regular windows updates, so you have to request it (at least at the time of this writing). It resolves the issues!
Also, if you use the Citrix Printer Server / Client this issue also doesn’t appear (and I plan to move to it ASAP after testing)

The pain I experienced

It took me some time to realise that this was the issue. I turned off client side rendering in both Group Policy and on the Print Server, neither of these fixed the issue.

The communications department blamed my cname for the print server as the issue, and recommended putting the cname into the hosts file of all computers (terrible idea by the way, and I had to untangle that mess).

In the end this hotfix fixed the issues above. Install it if you want to keep your sanity!

no default mail client

Have you been getting these errors?

NoEmailClientError

Either there is no default mail client or the current mail client cannot fulfill the messaging request. Please run Microsoft Outlook and set it as the default mail client.

Well! I get it a lot as well. Let me tell you why it happens and how to fix it!

Different bitness

If the program you are trying to email from is a different bitness from the Outlook you have installed; you will get this error. For example:

If you have 64bit Outlook installed and try to email from a 32bit application.
If you have 32bit Outlook installed and try to email from a 64bit application.

In the above scenarios you will see the default mail client error. There are some work arounds though!

Workarounds

Uninstall your current version and install the other version.
1. If you have 64bit Office, uninstall it and install 32bit Office
2. If you have 32bit Office, uninstall it and install 64bit Office
If you have 64bit Outlook installed I believe it is possible to have 32bit applications email correctly. Follow the steps here on this webpage.

I have not tried the instructions but from a cursory glance they look reasonable.

Unfortunately I have not found a way of emailing from 64bit applications with 32bit Outlook installed. Except there is a hack to do if for .NET applications!

.NET Applications

When .NET applications are built in Visual Studio, most of the time they will be built with platform of Any CPU. .NET applications have the advantage of running across multiple CPU types. If you run a .NET application on a 32bit computer it will be 32bit, run it on 64bit and it will be 64bit. Sounds Amazing!

Well, it’s not that amazing because we have 32bit Office installed on a 64bit computer. These .NET applications are deciding to run as 64bit applications and because of this they are throwing this error around like monkeys flinging poop. There is a solution!

Grab corflags.exe off MSDN, and then run it on the .NET application you are having trouble with:

corflags.exe dotnetexecutable /32BIT+ /Force

This tells the application to be 32bit in 64bit environments. I have tried this on several applications and it’s fixed the issues I was having.

Email Client registry settings are incorrect

Just so you know, I have very VERY rarely actually found that the client registry settings are incorrect. But hell, the steps to fix the registry settings can be found in Microsoft KB813745.

Hopefully the above helps you out!

issues with RDS virtual IPs

We run Cisco Ironports to monitor internet usage and block websites to certain staff etc. When authentication is enabled with the Ironports, and you are using Internet Explorer; Internet Explorer will silently authenticate using the current users credentials against the Ironport device. The Ironport will then save the IP address of the Computer using Internet Explorer, and assume all internet requests from that IP are coming from the user that originally authenticated.

This presents a problem in Citrix XenApp, as multiple users may be using the one server, which only has one IP address. If the first user to log onto the XenApp Server is a company director with full internet access, the Ironport will grant all subsequent users that log onto the server the same internet access. Hey all the traffic is from the same IP!

To get around this issue the Ironports offer a different method than using an IP address for remembering internet access, using a session cookie. From within Identities, under Authentication in the Ironports you can create an Identity, provide a list of IP addresses (your XenApp servers) and set the surrogate type to Session Cookie. From now on, when users try to access websites from servers that have an IP in the identity created above, Internet Explorer will be given a cookie for each website the user visits. Ironports will now refer to the session cookie rather than the IP address of the server for access.

Now internet monitoring and website blocking will work for users on RDS and Citrix XenApp servers!

Well… it’s not all rainbows and unicorns…

Session cookie based authentication is… well… shithouse. It’s terrible and I have no idea why they even offer it as a solution it’s so woefully inadequate. Here are some of the issues we have encountered:

Applications that require Internet Access, and don’t know how to handle a proxy asking for authentication… don’t work
- As now the Ironports authenticate you on a site by site basis, if you are running an application that requires an external resource, the ironport will ask the application for credentials, as most applications don’t know how to handle these credential requests, they usually complain that they can’t access the internet.
- If you were using IP based authentication then once the Ironport knows you are using a certain IP, it won’t bug you for credentials for some time. This allows applications that require internet access to work!
- You could also set the websites to require no authentication to get around the issue, but do you really want to figure out the URLs of each website
Websites with urls with less than 3 characters in before the domain (such as t.co) just don’t work at all unless you want to also put them into your no authentication website list.
Websites that have dynamic content that pull from other locations will fail to work.
Sometimes websites won’t work until you have the user clear their cookies.

What has this got to do with Virtual IPs?

Well, if we could somehow give each Citrix User a unique IP, the issues above would go away! By enabling Virtual IPs on your RDS server each user will get their own unique IP address. People on Cisco forums also recommend enabling Virtual IPs to get around the issues stated above (although I seem to be unable to find official documentation on it).

So it should just be as simple as getting an IP range from your network guys and enable Virtual IPs right? Well… you would think so!

Enabling Virtual IPs

There are good resources on this already out there, I followed the following articles:

After following the above and enabling Per Session IP Virtualization, I got it working on my test sever! But then I started to do some testing, and boy… Virtual IPs break a lot of things!

IP Virtualization randomly does not work

I’m not quite sure of what causes it, but if you run the following command I have had quite a lot of success with it beginning to work again:

netsh winsock reset

IP Virtualization fails on Servers provisioned in VMWare ESX/ESXi

I did not encounter this as we are running a newer version of VMWare, but I thought I would mention the issue if you are having it. Go here for a resolution: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1038232.

Sessions with Virtual IPs cannot LDAP

This effects so many things, but mainly Outlook. Outlook won’t connect to Exchange if it can’t connect to the domain, and how does it connect to the domain? LDAP of course! You will need to install Microsoft KB2619880

Group Policy fails to apply when Virtual IPs are enabled

Since enabling Virtual IPs I have seen Group Policy fail to apply far more often. You will need to install Microsoft KB2647582

Cannot shadow users with Microsoft Remote Assistance Tool on servers with Virtual IPs enabled

I could not find anything on the internet about this, so I created a technet forum post about it here. In the end the issue is: if you don’t have an administrative user logged onto the servers console session, you will not be able to shadow users on that server.

I contacted Microsoft Support about this issue, they worked on it for a few weeks before contacting me saying this is “by design”. I have requested Microsoft update their documentation and create a KB article to reflect this issue. I have not heard back yet but I will keep you posted.

I was still having difficulty with shadowing staff, in the end I created the following registry key:

HKLM\SYSTEM\CurrentControlSet\Control\Terminal Server\TSAppSrvVirtualIPPerSessionraserver.exe

It just has to be an empty key, seemed to fix the issue for me. I found that RAServer.exe was looking for this key when I ran procmon. I created the key and the issue went away!

Conclusion

Sure you can enable Virtual IPs, just be prepared to deal with all the issues associated with it. I can also recommend that you stay away from Cisco Ironports… they are rubbish.

find space with LDAP query

I looked everywhere and couldn’t really find an answer to this!

The company I work for had some legacy system that interfaced with the Active Directory, when a users attribute was emptied in the software, the software did not clear the attribute in AD, it just put a space character. 😦

This has been annoying for us as applications such as Exclaimer Mail Signatures have a if attribute has a value, write the attribute into the email signature. When peoples Mobile numbers are just a space in AD, it does technically have a value!

So I needed to find a query to fix this.

Find users with spaces in attributes

(objectCategory=user)(homePhone=20)

You can use ascci characters in an LDAP query. The ascci character for space (in hexadecimal) is 20. The backslash is an LDAP query escape character. The query above should find all the users that just have the space character in the homePhone attribute.

How to clear them all

Now that the offending accounts have been found, it’s time to fix em! Use PowerShell, and run this command:

Get-ADUser -LDAPFilter "(objectCategory=user)(homePhone=20)" | Set-ADUser -Clear homePhone

Done!