Import a 3rd Party Certificate Into XenMobile

ActiveSync, Blackberry, Xenmobile

To replace the default certificate that comes with XenMobile with a 3rd party one, from say Digicert, there are a couple of steps that you need to take.  None of these are hard, but are not well documented anywhere.  First you need to have a PFX file that has your root CA, intermediate CA and regular certificate included in it.  It should also be protected by a password.  The easy way to create this is to use an existing windows server that has the certificate installed.  Open of the Certificate snapin and browse to the local computer.

The following was done with XenMobile 8.6 but also applies to XenMobile 8.5.  It was also done with a wildcard certificate but the process should be the same for a SAN or regular certificate.

 

Navigate to Personal->Certificates.  As you can see I have the following certificate that I want to export:

 

first 

Click on the Details tab, and select copy to file

second to last

Select Yes, export the private key

image

 

Select the option to Include all certificates in the certification path if possible

image

Enter in a password to secure the file, and finally export:

image

Copy that file to the XenMobile Device Manager Server.  In our example we will copy it to a folder on the C drive called ExternalSSL.  Rename the extension to be p12 instead of pfx:

image

Next, to make your life easy, download the certificate utility from Digicert at https://www.digicert.com/util/.  When you run it you’ll get the following screen, and select Import in the upper right:

image

Browse to, and select the certificate we copied over earlier:

image

Next you need to enter the same password you put in when you exported the certificate above:

image

Enter a friendly name for the certificate.  This is simply so you can better label it:

 

digit

Select Finish, and you should get a message that the import was successful:

image

 

Now on the device manager server, navigate to the tomcat directory, which if your on an x64 server is the following path:

C:\Program Files (x86)\Citrix\XenMobile Device Manager\tomcat

We have to edit two files.  The first one is C:\Program Files (x86)\Citrix\XenMobile Device Manager\tomcat\webapps\zdm\WEB-INF\classes\pki.xml

Open the file in WordPad.  At the bottom of the file, but before the </beans> section paste the following info:

<bean id=”externalSslCert” class=”com.sparus.nps.pki.def.KeyStoreParams”

p:keyStoreType=”PKCS12″

p:keyStorePath=“C:\ExternalSSL\xenmobile.p12”

p:entryAlias=””

p:keyStorePass=“password”

p:publiclyTrusted=”true”

/>

Note the highlighted sections.  The first is the path to the certificate file, the next is the password from when we exported it. 

Save and close the file.

The next file we are going to edit is C:\Program Files (x86)\Citrix\XenMobile Device Manager\tomcat\conf\server.xml

Again open the file in wordpad. 

Find the following section, and replace the highlighted section to match the same as above

<Connector
            port=”443″
            maxHttpHeaderSize=”8192″
            maxThreads=”400″
            enableLookups=”false”
            redirectPort=”-1″
            acceptCount=”100″
            connectionTimeout=”30000″
            disableUploadTimeout=”true”
            maxKeepAliveRequests=”-1″
            protocol=”org.apache.coyote.http11.Http11NioProtocol”
            scheme=”https”
            secure=”true”

            clientAuth=”want”
            SSLEnabled=”true”
            truststoreFile=”C:\Program Files (x86)\Citrix\XenMobile Device Manager\tomcat\conf\cacerts.pem.jks”
            truststoreType=”JKS”
            truststorePass=”notMeaningFul”
            keystoreFile=”C:\ExternalSSL\xenmobile.p12″
            keystorePass=”password”
            keystoreType=”PKCS12″
            ciphers=”TLS_RSA_WITH_AES_256_CBC_SHA,SSL_RSA_WITH_RC4_128_SHA,SSL_RSA_WITH_RC4_128_MD5,TLS_ECDHE_RSA_WITH_RC4_128_SHA,TLS_ECDH_RSA_WITH_RC4_128_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_RSA_WITH_AES_256_CBC_SHA256,TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384″

        />

 

Still in server.xml, do the same with the following section:

<Connector
            port=”8443″
            maxHttpHeaderSize=”8192″
            maxThreads=”20″
            enableLookups=”false”
            redirectPort=”-1″
            acceptCount=”100″
            connectionTimeout=”30000″
            disableUploadTimeout=”true”
            maxKeepAliveRequests=”-1″
            protocol=”org.apache.coyote.http11.Http11NioProtocol”
            scheme=”https”
            secure=”true”

            clientAuth=”false”
            SSLEnabled=”true”
            truststoreFile=”C:\Program Files (x86)\Citrix\XenMobile Device Manager\tomcat\conf\cacerts.pem.jks”
            truststoreType=”JKS”
            truststorePass=”notMeaningFul”
           keystoreFile=”C:\ExternalSSL\xenmobile.p12″
            keystorePass=”password”

            keystoreType=”PKCS12″
            ciphers=”TLS_RSA_WITH_AES_256_CBC_SHA,SSL_RSA_WITH_RC4_128_SHA,SSL_RSA_WITH_RC4_128_MD5,TLS_ECDHE_RSA_WITH_RC4_128_SHA,TLS_ECDH_RSA_WITH_RC4_128_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_RSA_WITH_AES_256_CBC_SHA256,TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384″

        />

Save and close the file.

Now, restart the XenMobile Device Manager service:

image

After browse to https://localhost/zdm on the Device Manager and you should be able to validate that your certificate was installed.  Note that the tomcat service does spend some time with heavy CPU after a restart and it may be a minute or two until the page comes up:

image

Now all thats left is to publish it in DNS!

How to Delete Emails from Mailboxes in Exchange 2010 Using Search-Mailbox

Exchange 2010, Exchange 2013, Scripting, Security

Quick post today.  If you have an criteria of emails that you want to delete from all mailbox’s you can use Search-Mailbox to delete the message from users mailbox.  For example, you get hit with a virus email that has the subject “I am a virus” from the user virussender@virus.com.

The below search goes through all mailbox’s and deletes any email with the subject “I am a virus”:

 

get-mailbox | Search-Mailbox -SearchQuery ‘subject:”I am a virus.” ‘ -TargetMailbox “adminmailbox” -TargetFolder “DeleteVirus” -LogLevel Full –DeleteContent

 

That above command will delete the emails from the users mailbox’s and copy them to the adminmailbox in the folder DeleteVirus

The next search does it based on the sender of the email:

get-mailbox | Search-Mailbox -SearchQuery ‘from:virussender@virus.com ‘ -TargetMailbox “adminmailbox” -TargetFolder “DeleteVirus” -LogLevel Full –DeleteContent

 

The above search does the same thing as the first, except this query does it based on the from address of the sender versus the subject.

If you log into the admin mailbox, you can see the results of the search and log files for it:

 

image

You can also run the same commands without the –DeleteContent switch to copy the messages to the admin mailbox for review before running the commands to delete.

Remember, do this at your own risk and test test test before you run it!

How to Use Managed Availability in Exchange 2013 with your Load Balancer

Exchange 2013, High Availability, Managed Availability, Netscaler

One of the major changes in Exchange 2013 is the concept of Managed Availability.  I wont go too deep into it, but it is the ability of Exchange 2013 to monitor itself, detect problems and attempt to resolve them.  One of the added bonuses of this, is that Managed Availability then knows when a particular application is working and able to serve data.  One of these specific instances where we can use it with third party tools is with hardware based load balancers.

One of the jobs of the hardware load balancer is to detect the health of the server that it is load balancing, something that Managed Availability is already doing itself!  Hardware load balancers can detect the health through a variety of different ways.  The basic is ping, which just checks if the host is responding to ping.  The obvious problem here is that the host could be up, but none of the services!  The next would be to check if a port is accessible.  Here you configure the load balancer to check if say port 443 is alive.  This is better than ping, but doesn’t check if the application is actually working behind the scenes, just that it can telnet to 443.  We can use Managed Availability with our hardware load balancer to check if the application itself is actually healthy.

How do we do that?  Say we want a HLB to check if OWA is healthy on a server.  The normal http path would be https://servername/owa right?  Well, if you navigate to https://servername/owa/healthcheck.htm, you will get a page generated on the stop indicating if OWA is working on that server.

For example, say we have two Exchange 2013 servers:

PHDC-SOAE13CAS1 – 10.220.10.3

PHDC-SOAE13CAS2 – 10.220.10.4

And we want to publish OWA through a HLB to email.company.com at ip address 10.220.10.1

If we navigate to https://PHDC-SOAE13CAS1/owa/healthcheck.htm on this server with working OWA, we get the following page:

image

Not a lot too it, but essentially its returning a 200 OK message indicating the service is working. If the service was not working, this page would not generate.  So we can have our HLB check to see if it gets a 200 OK response from a particular server.

We want to configure these in our load balancer for services such as OWA, Activesync, Outlook Anywhere etc.  So, we will configure Exchange 2013 using Citrix’s Netscaler in this example.  The configuration will be similar for other HLB, but we’ll go through the steps here.

On the Netscaler go to Load Balancing->Monitors and click add to create a new monitor.  Here, we will create a custom monitor for the Netscaler, so that it can poll that web page.  Name the monitor MONITOR-EXCHANGE2013_OWA and set the type to HTTP-ECV.  Ensure to select the Secure check box at the bottom as this will be over SSL.  Leave the other options default:

image

Next, click on the Special Parameters tab.  In the Send String box enter in GET /owa/healthcheck.htm.  In the Receive String box, enter in 200 OK:

image

Click create to save, and navigate over to Load Balancing –>Service Groups

Add a new Service Group, and name it Exchange2013-OWA and set the protocol to SSL.  Enter in the IP addresses of your CAS servers, and set their ports to 443

image

Next, click on the Monitors tab.  Find the Monitor_Exchange2013_OWA monitor we created above and add it to the configured Monitors selection

image

Click on the SSL Settings tab and select the SSL certificate that you will use to publish the Netscaler service to the internet.  I have a preloaded one named Lab-2013 that I will be using:

image

Click Create to save the Service Group.

If we check, our service group should be reporting up, that’s good, it means our monitor is working!

image

Next lets go to Load Balancing->Virtual Servers

Create a new Virtual Server and name it Exchange2013_OWA, Set the Protocol to SSL, and assign it an IP, in our case 10.220.10.1. Leave the SSL Port at 443.

image

Select the Service Groups tab and select the service group Exchange2013-OWA we created earlier:

image

Then click on the SSL Settings tab and select the same certificate as you did on the service group, in our case Lab-2013:

image

Click Create to save the virtual server.

Next, make sure your DNS address is pointing to the IP address of the virtual server and lets try to login:

image

There we go!  There is are OWA page!  But, the question is how do we test that our monitor is working.  That’s easy.  On PHDC-SOACAS2, lets go into IIS Manager and stop the MsExchangeOWAAppPool:

image

If we check our Netscaler, we see that one of the servers is now being reported as down:

image

If we try to telnet to that server on port 443:

image

We can see it works fine:

image

I know it doesn’t show much, but it shows that the server is still listening on port 443.  This also proves that using Managed Availability for your HLB is much better.  Here, the standard checks would have said the server was working fine, sent user requests to it, but in fact OWA isn’t working.  But since we are using Managed Availability, we are passing that knowledge on an application layer to our HLB.

If we try to go back to OWA:

image

The HLB sees that one server is down, and runs everything to the server that’s still up.  But, if both servers have their application pools stopped:

We get a HTTP Error, The Service is unavailable:

image

This works for all of the Exchange web services.  So that means you can create separate monitors just be appending healthcheck.htm at the end of the URL.  So for ActiveSync its https://servername/Microsoft-Server-ActiveSync/healthcheck.htm.  The only one that has a stipulation is OWA, which requires Forms Based Authentication to be enabled for it to provide a HealthCheck.htm page.

I hoped you have found this helpful, and hopefully it will save you some configuration steps (and some uptime!) on your hardware load balancer

5.3.2 STOREDRV.Deliver: Missing or bad StoreDriver MDB Properties When Using Address Book Policy Routing in Exchange 2013

Exchange 2013, Hosting, hub transport

 

If your using the new Address Book Policy Routing Agent to assist in separating your tenant organizations, you may run into a situation where a user is not able to receive email sent to them.  If you check the message tracking logs with:

Get-MessageTrackingLog –Recipients emailaddress@company.com

You may get a result similar to this one below:

9

As we can see, our original email to SOA-User1@accessabac.us generated an Undeliverable.

Of particular note is the Recipient Status where it lists 532 5.3.2 STOREDRV.Deliver: Missing or bad StoreDriver MDB properties.

There are two causes of this.  The first seems to be a bug where the mailbox is hidden from address lists:

image

This appears to be fixed in Exchange 2013 CU2.  A workaround is to uncheck the hide from address lists box.  See this forum post by Greg Taylor of Microsoft:

http://social.technet.microsoft.com/Forums/exchange/en-us/69a3e303-f848-4729-b818-ea1acaeb43d2/exchange-2013-address-book-policy-routing-agent-issue-with-mailboxes-hidden-from-the-address-lists

The second cause, is not a bug but expected behavior. If the user you are trying to email to, is not a member of the GAL that is in their Address Book Policy, the email routing will fail.  I’ll give you an example.

In our case, we tried to email SOA-User1@accessabac.us.

The user has an Address Book Policy applied to him called ABP-Soa

image

If we check that address book policy with Get-AddressBookPolicy ABP-Soa we can see that the Default Global Address List is GAL-SOA

image

If we check the filter of that Global Address list with Get-GlobalAddressList GAL-SOA | select RecipientFilter:

image

We can see that the GAL is filtering for all recipients that are of the type UserMailbox that also has CustomAttribute1 set to value SOA.

If we check the mailbox soa-user1@accessabac.us:

image

So, there is our problem.  SOA-User1 has the Address Book Policy ABP-SOA assigned to him, which forces him to get the Global Address List GAL-SOA.  But since we failed to set CustomAttribute1 to SOA on his mailbox, he is actually not even showing up in his own GAL.  This is causing the routing issue.  We can simply fix this by ensuring the user is presented in his own GAL, in this case adding CustomAttribute1 with a value of SOA:

image

And you should be all set. 

User Cannot Set an Out of Office in Exchange 2010/2013 or EventID 3004 Appears in your Event Log

exchange 2007, Exchange 2010, Exchange 2013

Recently had an issue where a user was setting an out of office in their Outlook client, but when a different user emailed them, no Out of Office was sent.  We did the traditional troubleshooting steps, check rules on the far side, check junk box, check OWA to ensure the Out of Office was set.  Everything looked good.  Strange enough, if an internal user went to email the person going on vacation, they would receive a MailTip to that effect:

image

But, went you sent the email to Paul Ponzeka, you would not get an Out of Office back.  So what gives?

One of the steps we took was to check if Microsoft Exchange Mailbox Assistants service was running.  It was, we restarted it, but still same effect.  One of the responsibilities of the Microsoft Exchange Mailbox Assistants service is to handle enabling the Out of Office.  In this case, it didn’t resolve anything.

The next step we took was to try disabling, and re-enabling the Out of Office message, and then checking the event log on the Mailbox server.  Low and behold, we found our answer:

Untitled

The above is EventID 3004 from the MsExchangeMailboxAssistants.  As you can see, there is an error stating the rules quota of the mailbox has been reached and the automatic reply rules can’t be enabled or updated.

Well, we found our issue, how do we fix it?  There are two ways.

The first the user can go through their Outlook rules and edit or delete existing rules.  Keep in mind that the length of both the rule actions, as well as the NAMES of the rules themselves will affect the size of the rule

The second is, the Exchange admin can increase the size of the rules quota.  By default, all mailboxes have a 64 KB quota, think of this as mailbox limits for Outlook rules.  We can increase this up to 256 KB on Exchange 2007 and later.  Exchange 2003 we cannot increase the rule size unfortunately, but that’s okay, because you should be off Exchange 2003 by now!

So, how do we increase the quota?  Easy.  Open Exchange Management Shell.  We can check the existing quota of the user pponzeka by running the command:

Get-Mailbox –Identity pponzeka | select RulesQuota

image

As we can see, its set to 64 KB.  To increase it, we run:

Set-Mailbox –Identity pponzeka –RulesQuota 256KB

image

And there we go, we can check by running the same Get-Mailbox as above to confirm:

image

Now, you cannot go past 256KB, this is the error you get, in this example I tried to set it to 512KB:

Untitled2

(UPDATED) How to Get a Report of Active Sync Devices in Exchange 2010/Exchange 2013

ActiveSync, Exchange 2010, Exchange 2013, Scripting

***UPDATE***

 

There was a lot of feedback from people who wanted extra items and fields added in so I edited the original script.  It will now also include the users Primary SMTP Address, LastSyncAttemptTime, and LastSyncSuccessTime.  Also edited it for environments with over a thousand mailboxes. Below is the updated script code:

The new output will look like this:

image

image

 

Ever had the need to get a nice report of all active sync devices in your Exchange organization?  Well then I have the script for you!

This script will get through all active sync devices, and match them up with their respective owners.  It will also output the Device Type, Device Model, and most importantly the Device OS.

Why is this important?  As some of you remember shortly back, there was an issue with Apple iOS devices causing excessive logging on Exchange Mailbox servers.  As part of the way to fix that you could block or quarantine those devices.  You most likely want to be able to see who you are blocking though, so you you know who your going to annoy and warn them preemptively.

The script requires 4 parameters to run, and should be run from an Exchange Management Shell:

  1. SMTPServer = SMTP server as the report will send you a copy of the report
  2. SMTPFrom = The FROM address of the email
  3. SMTPTo = The recipient of the email
  4. ExportPath = The folder location where you want the CSV export of the report. 

For example, to send the report to admin@port25guy.com using the SMTP server relay.port25guy.com, have the from address be reports@port25guy.com and export the CSV to C:\Reports:

Get-ActiveSyncReport.ps1 –exportpath C:\Reports –smtpserver relay.port25guy.com –smtpfrom reports@port25guy.com –smtpto admin@port25guy.com

After running that, if we check the C:\reports folder we should have a nice CSV export:

image

And a nice email report in your inbox:

 

image

You can download the script from the link below, just rename the file to extension .ps1

Here is the ***UPDATED** script block as well:

 

### BEGINNING OF SCRIPT

#####
#
# Get-ActiveSyncDeviceReport
# Author: Paul Ponzeka
# Website: port25guy.com
# email ponzekap2 at gmail dot com
#
######
param(
    [Parameter(Mandatory = $true)]
    [string] $SMTPServer = “”,
    [Parameter(Mandatory = $true)]
    [string] $SMTPFrom = “”,
    [Parameter(Mandatory = $true)]
    [string] $SMTPTo = “”,
    [Parameter(Mandatory = $true)]
    [string] $exportpath = “”
    )

#######
#
# HTML Formatting Section
# Thanks to Paul Cunningham at http://exchangeserverpro.com/
#
#######
#
#
#
######
$style = “<style>BODY{font-family: Arial; font-size: 10pt;}”
$style = $style + “TABLE{border: 1px solid black; border-collapse: collapse;}”
$style = $style + “TH{border: 1px solid black; background: #dddddd; padding: 5px; }”
$style = $style + “TD{border: 1px solid black; padding: 5px; }”
$style = $style + “</style>”

$messageSubject = “ActiveSync Device Report”

$message = New-Object System.Net.Mail.MailMessage $smtpfrom, $smtpto
$message.Subject = $messageSubject
$message.IsBodyHTML = $true

####  Get Mailbox

$EASDevices = “”
$AllEASDevices = @()

$EASDevices = “”| select ‘User’,’PrimarySMTPAddress’,’DeviceType’,’DeviceModel’,’DeviceOS’, ‘LastSyncAttemptTime’,’LastSuccessSync’
$EasMailboxes = Get-Mailbox -ResultSize unlimited
foreach ($EASUser in $EasMailboxes) {
$EASDevices.user = $EASUser.displayname
$EASDevices.PrimarySMTPAddress = $EASUser.PrimarySMTPAddress.tostring()
    foreach ($EASUserDevices in Get-ActiveSyncDevice -Mailbox $EasUser.alias) {
    $EASDeviceStatistics = $EASUserDevices | Get-ActiveSyncDeviceStatistics
    $EASDevices.devicetype = $EASUserDevices.devicetype
    $EASDevices.devicemodel = $EASUserDevices.devicemodel
    $EASDevices.deviceos = $EASUserDevices.deviceos
    $EASDevices.lastsyncattempttime = $EASDeviceStatistics.lastsyncattempttime
    $EASDevices.lastsuccesssync = $EASDeviceStatistics.lastsuccesssync
    $AllEASDevices += $EASDevices | select user,primarysmtpaddress,devicetype,devicemodel,deviceos,lastsyncattempttime,lastsuccesssync
    }
    }
$AllEASDevices = $AllEASDevices | sort user
$AllEASDevices
$AllEASDevices | Export-Csv $exportpath\ActiveSyncReport.csv

######
#
# Send Email Report
#
########

$message.Body = $AllEasDevices | ConvertTo-Html -Head $style

$smtp = New-Object Net.Mail.SmtpClient($smtpServer)
$smtp.Send($message)

##END OF SCRIPT

Also, special thanks to Paul Cunningham at http://exchangeserverpro.com.  He wrote the HTML formatting section in the script that makes this look nice and pretty, versus my junky plain text (http://exchangeserverpro.com/powershell-html-email-formatting).  If you haven’t check out Paul’s site you should, he has great information on there. 

Hope you find the script helpful!

How to Install a Certificate in Exchange 2013

Uncategorized

 

Log into the Exchange Admin Center by going to your CAS server at https://CASSERVERNAME/ECP:

image

Now navigate to Servers->Certificates

image

Select the CAS server you want to push it to, in our case we will select PHDC-E15CAS01.E15.corp

image

Now, select the + sign which will bring up the New Exchange Certificate wizard:

image

Create a friendly name for the certificate:

image

At the next screen you can decide to request a wildcard certificate, where you would enter the root domain.  For example, if you wanted a wildcard certificate for exchange15.com, your screen would look like the following:

image

If you want to create a SAN certificate, leave this unchecked and select next.

Select the server to store certificate on, in our case, the same server we are requesting it for PHDC-E15CAS01:

image

Next, you need to select the services that you want to assign to the external domain, and the FQDN of that service.  In our case, everything will be to email.exchange15.com.  Select each service that does NOT say (when accessed from the intranet) and click the pencil icon to edit the domain:

image

image

image

When you click next, it will show you the domains that will be added to the certificate.  If you have any accepted domains in your organization, it will add the autodiscover.accepteddomain.com entry to the certificate:

image

When you click Next, you will need to fill out the information for the organization requesting the certificate:

image

Select the location to save the certificate.  If you don’t have a network share pre-configured (with the exchange trusted subsystem as an administrator), then you can store it on the C drive of the CAS server with \\phdc-e15cas01.e15.corp\c$\newcertreq.req

image

Now when you see the request, it will be pending:

 

image

Now we need to submit this request to a certificate authority to complete the request.  In our case, we will use a Windows 2008 R2 CA to do so.

Log into your certificate authority at https://CA/certsrv

Select Request a Certificate-> Advanced Certificate Request-> Submit a Certificate Request by using…

Open the request you saved before in notepad:

image

Copy and past that into the Base-64-Encoded…field, and set the Certificate Template to Web Server:

image

Hit submit to finalize, and you should see the option to Download Certificate or Download the Certificate Chain.  Select Download the certificate and save the file to the shared location that you saved the request file to.  Next, download the Certificate Chain to the same location, as we will need to import the CA certificate to the host to ensure it trusts the certificate.  certnew.cer is the exchange servers certificate, certnew.p7b is the CA certificate.

image

To import the Certificate Authority certificate, RDP into PHDC-E15CAS01.  Open up a blank MMC console and add the certificates snapin for the local account:

image

Expand and select Certificates underneath Trusted Root Certification Authorities

image

Right click Certificates select Import->All Tasks->Import

image

Select the Certificate Authority certificate you downloaded before:

\\phdc-e15cas01.e15.corp\c$\certnew.p7b

image

Select Next and Finish.

Return to Exchange Admin Center, select the pending request certificate, and on the right hand side select Complete

image

image

A new dialog box will open up, enter the path to the certnew.cer file, in our example this would be:

\\phdc-e15cas01.e15.corp\c$\certnew.cer

image

Now we need to assign this certificate to the specific services we want, select the certificate and click the pencil icon.  Then click services, and lets check off which services we want.  We are going to want to add SMTP and IIS:

image

You will receive a warning about overwriting the existing certificate, just select yes:

image

That’s it, you are all set! When we go to the site and check the certificate:

image

We are now utilizing the new cert!

Witness Server Boot Time, GetDagNetworkConfig and the pain of Exchange 2010 DR Tests

Exchange 2010, High Availability

 

So we recently had a client who wanted to perform a DR test of their Exchange 2010 DAG.  The DAG consisted of a single, all in one server in production, and a single all in one server in DR.  The procedure for this test was to disconnect all network connectivity between prod and DR, shutdown the exchange server and the domain controller, snapshot them, and then start them back up.

Now, we can all agree that snapshots and domain controllers are inherently dangerous, so its up to you to ensure that you have your ducks in a row to ensure that this doesn’t replicate back to production.  That discussion is outside this article.

Now, initially they had trouble bringing up the databases in DR, as well as many components of the DAG.  This article will walk through an example, and try to make sense of what’s causing these issues.

So, here is our setup, we have a two node DAG cluster, stretched across two sites. 

Production

PHDC-SOAEXC01 – Prod all in one Exchange Server

PROD-DC01 – Prod domain controller

PHDC-SOADC01 – Primary witness server

DR

SFDC-SOAEXC01 – DR all in one Exchange Server

DR-DC01 – DR domain controller

SFDC-SOADC01 – Alternate witness server

The DAG name is SOA-DAG-01 and the Active Directory Sites are:

Prod = PH

DR = SF

So in our scenario, we shutdown both PHDC-SOAEXC01 and PHDC-SOADC01.  This will cause the databases in DR to dismount because quorum has been lost by the DR server.

Now, in a DR “test”, we would shutdown the DR exchange server, and the DR domain controller to snapshot them.  I just want to warn you, DO NOT EVER roll a domain controller back to a snapshot in a production environment.  This is a purely hypothetical setup.  Rant over.

Now, in our case, we have snapshotted and rebooted DR-DC01 and SFDC-SOAEXC01.  When we open the Exchange Management Console, we see that the DR servers databases is in a failed state:

image

Now, lets start running through the DR activation steps.  Here is what the process should normally be:

  1. Stop the mailbox servers in the prod site
  2. Stop the cluster service on all mailbox servers in the DR site
  3. Restore the mailbox servers in the DR site, evicting the prod servers from the cluster

After step 3, the database’s should mount, but as you will see, they wont, and I’ll try to explain why.

So, step 1, lets mark the prod servers as down:

   1: Stop-DatabaseAvailabilityGroup SOA-DAG-01 -ActiveDirectorySite PH -ConfigurationOnly

You should expect to see some errors, this is completed expected because the prod site is unable, hence the –configurationonly option:

image

Now, step 2, we will stop the clustering service on SFDC-SOAEXC01 with the powershell command:

   1: Stop-Service ClusSvc

Now, step 3, we will restore the dag with just the servers in DR:

   1: Restore-DatabaseAvailabilityGroup SOA-DAG-01 -ActiveDirectorySite SF

You may get an error stating

Server ‘PHDC-SOAEXC01’ in database availability group ‘SOA-DAG-01’ is marked to be stopped, but couldn’t be removed fro

m the cluster. Error: A server-side database availability group administrative operation failed. Error: The operation f

ailed. CreateCluster errors may result from incorrectly configured static addresses. Error: An error occurred while attempting a cluster operation. Error: Cluster API ‘"EvictClusterNodeEx(‘PHDC-SOAEXC01.SOA.corp’) failed with 0x46.

Simply re-run the command again and it should complete:

image

So now, we should have the databases mounted, and we should be able to see the prod servers as stopped by running the following command:

Get-DatabaseAvailabilityGroup -Status | FL

But, behold, we get an error stating GetDagNetworkConfig failed on the server.  Error: the NetworkManager has not yet been initialized

image

So, here is the first road block, what happened is that since the DR server is one node, it uses the boot time of the alternate file share witness to determine if it is allowed to form quorum.  This is due to a one node cluster, always having cluster, and it trying to prevent split brain.  Tim McMichael does a great job of explaining it Tim McMichael Blog Post.  Essentially the boot time is stored in the registry of the Exchange Server under:

HKEY_LOCAL_MACHINESoftwareMicrosoftExchangeServerv14ReplayParameters

The Exchange Server checks if it was rebooted more recently than the AFSW, it will not form quorum.  So how do we fix?  We can start by rebooting the AFSW to see what behavior changes.

After we do so, we can re-run:

Get-DatabaseAvailabilityGroup -Status | FL

Now, we get the network and stopped servers info, but there are some entries that are in a broken state, and we get the message that the DAG witness is in a failed state:

image

Note the WitnessServerinUse field reports InvalidConfiguration

We have to re-run our Restore-DatabaseAvailabilityGroup command to resolve this:

Restore-DatabaseAvailabilityGroup SOA-DAG-01 -ActiveDirectorySite SF

Now if we re-run Get-DatabaseAvailabilityGroup –Status | FL we get an expected output:

image

Now, we see that the WitnessShareInUse is set to the alternate.

So, are the databases mounted!? If we check, they are no longer failed, but are “Disconnected and Resyncing”

image

We need to force the server in DR to start because of the single node quorum issue.  This can be done with the following command:

Start-DatabaseAvailabilityGroup SOA-DAG-01 -ActiveDirectorySite SF

Now the database is mounted:

image

So, you can see, the testing can affect what occurs with the DR test, but also the setup with the single node cluster can cause this issue.  The boot time of the alternate file share witness is also extremely important to what the node can do when it restarts.

Hopefully you find the info useful!  Happy Holidays to all!

Configure Application Impersonation for Exchange 2010 in Resource Forest

Uncategorized

 

With the new Exchange 2010 RBAC model, one of the configuration changes is regards to EWS and Application Impersonation.  Instead of defining the ACL’s directly, you configure roles for the appropriate permissions.

If your in a resource forest setup, things are a little different.  Here are the steps.

Your service account, named ServiceAccount needs to be assigned Application Impersonation rights to all the accounts in the Accounting OU.  The user accounts are in client.corp and the Exchange mailboxes are stored in exchange.corp and there is a forest trust between the two.

Step 1:

Create a Universal Security Group in client.corp named UG-ExchangeImpersation.

Step 2:

Create a new linked role group with the Application Impersonation rights bound to this group.  Run the following from an Exchange Management Shell in exchange.corp:

$remotecred = get-credential
Put in a user name of an admin account for client.corp

New-RoleGroup ROLEGROUP-ExchangeImpersonation –LinkedForeignGroup “UG-ExchangeImpersation” –LinkedDomainController DC01.client.corp –RecipientOrganizationalUnitScope ‘exchange.corpAccounting’

Step 3:

Add serviceaccount to the UG-ExchangeImpersonation group in Client.corp.

Ensure that serviceaccount has a linked mailbox in exchange.corp.

Once AD replication finishes, you should have impersonation rights on all users in that Organizational Unit!