Office 365 Error - stuck at "We are preparing a mailbox for this user"

Background: Customer has an Office 365 tenant with DirSync enabled with OnPremise active directory. DirSync has been setup successfully and no errors are reported on synchronization. Users correctly appear in the office 365 portal and are correctly displayed as "Synced with Active Directory"

Users were then assigned a license using the Office 365 portal. Around 40% of the users were provisioned successfully and the Office 365 portal successfully displayed the email addresses and mail properties for the user. The remaining users had the message "We are preparing a mailbox for this user" under the mail properties.

In the Exchange Admin Center, a mailbox was successfully created for all users, including the ones showing the above error. However certain attributes such as (First Name, Last Name, proxyaddresses) that were set in the On Premise AD were not being populated in EAC. The only proxy address displayed was the primary "user@default-tenantdomain.com" address.

At this point, the user mailboxes were successfully receiving email and we were able to configure outlook 2016. However if a user logged in to the Office 365 Portal, the mail tile was grayed out and displayed "Setting up…".

Having found very little useful information on forums, a case was logged with Microsoft support and 12 hours had lapsed with no response. Below are some of the troubleshooting steps followed to identify the cause of the issue.

Troubleshooting Process

Remove the license assigned to the user, wait for the mailbox to disappear from EAC, reassign license to use. The same issue persisted.

Connect to Office 365 with Powershell (http://o365info.com/connecting-to-office-365-with-remote/)

$Cred = Get-Credential

Import-Module MSOnline

Connect-MsolService -Credential $cred

$Session = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri https://ps.outlook.com/powershell/ -Credential $Cred -Authentication Basic -AllowRedirection

Import-PSSession $Session

Idenitfy user accounts with errors. The command below listed all the accounts that were showing the above error.

Get-MsolUser -HasErrorsOnly

Check the detailed status of a user account with the error

Get-MsolUser -UserPrincipalName user@domain.com | fl

The user accounts that displayed the issue above had the Errors property populated with an exception and the ValidationStatus property listed as Errors.

The user accounts that did not have this issue did not have any values in the Errors property and the ValidationStatus property was listed as Healthy.

Then I came across a Microsoft KB article on how to troubleshoot validation errors on Office 365.
https://support.microsoft.com/en-au/kb/2741233

The following commands were run to retrieve the errors on user object

$errors = (get-msoluser -userprincipalname "johnsmith@contoso.com").Error

$errors | foreach-object {"`nService: "+ $_.ErrorDetail.Name.split("/")[0]; "Error Message: "+ $_.ErrorDetail.ObjectErrors.ErrorRecord.ErrorDescription}

The output displayed an error with the Country code.

This lead me to investigate the user accounts in Active Directory.

All users did not have the country code populated in the Address tab. However, looking at the attribute editor, I found the users with the issue had the "C" attribute set to the value "EN" whereas the users who did not have the issue did not have a value set for the "C" attribute.

Solution: Clear the country code attribute or set the Country in the Address tab for each user. This can also be done by selecting multiple users.

Once the country code is set, force a DirSync to replicate.

Start-ADSyncSyncCycle -PolicyType initial
 

Once replication completed, the user mailboxes were completed successfully within about 5 mins.