Firstly, I need to set the background of this, many years ago, a simple and username and password was acceptable with an autodiscover record and SCP entry - both of these work together, XML file and SRV record.
Exchange/Exchange Online Outline
If you have a local connection to your Exchange server that usually Outlook does not need to talk out to the Internet, then you can usually pass your credentials through to your Exchange server with minimal configuration.
However, if you have both Exchange locally and one in the cloud, that would be Exchange online you have what’s known as a hybrid configuration,
This configuration keeps your mailbox record in locally on Exchange as a remote mailbox and you have your custom routing address to get your mail to EXO mailbox.
Hybrid Mailflow
In this instance, we have a hybrid configuration so it’s slightly more complicated But that doesn’t mean you can’t troubleshoot issues, So the issue here seems to be outlook is unable to automatically configure itself in the set of circumstances we have, and rather than automatically configure itself and then start Outlook you get an error about please confirm your email address, therefore, obviously, something has broken down with the autodiscover process.
Outlook : Modern Authentication
It goes without saying you need to use modern authentication, which is the authentication method that supports MFA for added security, depending on your version of Outlook, you’re trying to use this will be natively, supported, or you will need to apply additional Registry keys to turn Modern authentication on, Obviously, it goes without saying you will need to keep your office installs up-to-date and patched because all the versions of office can be quite buggy when it comes to Modern authentication, and they work consistently.
Outlook : Version and Registry keys
In this example Outlook is on the latest version and the registry keys are applied, but we are unable to connect Outlook successfully using autodiscover which will deliver your mail to your Outlook client, The mailbox is located in exchange online so if we use the Web version of Outlook, it works absolutely fine, it’s worth noting that the web version of Outlook uses websites to get your mail not the autodiscover process
Internet Articles and 🐇 Rabbit holes
If you try to research this on the Internet, there will be many helpful articles, but unfortunately many of them will lead you down rabbit holes that may or may not fix the problem, and what is worse sometimes it will cause you to change settings that don’t need to be changed, When I was trying to troubleshoot this, the rabbit holes I went down included:
- Ensure outlook was patched and on the latest version
- Ensure autodiscover registry key required modern authentication
- Ensure identity profile for Outlook was enabled for modern authentication
- Confirm browsers were enabled for TLS 1.2
- Confirm system was enabled for TLS 1.2
- Confirm IEtoEdge BHO (browser helper object) was Set up correctly
- Ensure Edge IEMode was not enabled for exchange services
- Ensure compatibility mode was not turned on for exchange services
- Check enable third-party browser extensions in Internet Explorer (same as IEtoEdge)
- Checking System event log for schannel errors
- Checking enabled cipher suites and protocols system wide
🐇 Rabbit holes failing to fix the issue
Interestingly, none of these ideas or fixes where applied were working, so now is the time to think to yourself are you getting far too technical without taking a look at the bigger picture, is now the time to step back and rather than hypothesis and predict what’s going on and actually get the valid data to tell you what’s going on.
Taking a step back : Overview
First, we need to be clear on how the Autodiscover process works, so let’s get into that right now because knowing how stuff works is just as important as fixing it, seriously.
When does Autodiscover process run?
- During account creation
- At certain intervals to collect Exchange and EWS data
- Response to certain connectivity failures
- Manually invoked by other applications
Now we need to know when this process is involved how it actually works, obviously, there are many steps to this process, which are not relevant to this guide, but I will outline the important steps below.
Check 0365 as a priority
If Outlook determines confidently that you are an O365 user, an attempt is made to retrieve the Autodiscover payload from the known O365 endpoints (typically https://autodiscover-s.outlook.com/autodiscover/autodion scover.xml or https://autodiscover-s.partner.outlook.cn/autodiscover/autodiscover.xml). If this step does not retrieve a payload, Outlook moves to checking SCP data
Check SCP data
If the computer is domain-joined, Outlook performs an LDAP query to retrieve Service Connection Point data that returns a path of the Autodiscover XML. An attempt is then made to each URL that's returned by the SCP lookup to try to retrieve the Autodiscover payload. If this step does not retrieve a payload, Outlook moves to checking root domain
Checking root domain
For this step, Outlook builds a URL from the domain name of the initial address in the format of https://<domain>/autodiscover/autodiscover.xml and tries to retrieve the payload from the resulting URL. Because many root domains aren't configured for Autodiscover, Outlook purposefully silences any certificate errors that occur during the attempted retrieval. If this step does not retrieve a payload, Outlook moves check autodiscover domain
Check Autodiscover domain
Outlook builds a URL from the domain name of the initial address in the format of https://autodiscover.<domain>/autodiscover/autodiscover.xml and tries to retrieve the payload from the resulting URL. Because this is the primary URL typically for Autodiscover data, Outlook does not silence any certificate errors that occur during the attempted retrieval. If this step does not retrieve a payload, Outlook moves to checking for local XML file
Checking for local XML
Outlook checked whether the administrator had deployed a policy to specifically check for the Autodiscover payload as a preference. If there's no policy in place, but the previous steps didn't retrieve a payload, Outlook now attempts to retrieve a payload from the local file even without the PreferLocalXML setting in place. If this step does not retrieve a payload, Outlook checks for HTTP redirects
Checking for HTTP redirects
Outlook sends a request to the Autodiscover domain URL (http://autodiscover.domain>/autodiscover/autodiscover.xml) and test for redirect responses. If an actual Autodiscover XML payload is returned and not a redirect, Outlook ignores the actual Autodiscover XML response because it was retrieved without security (http). If the response is a valid redirect URL, Outlook follows the redirect and tries to retrieve a payload XML from the new URL. Outlook will also perform certificate checks to prevent redirection to potentially harmful URLs in this step. If this step does not retrieve a payload, Outlook then checks for SRV data
Note : Outlook only considers http 301 or 302 as a valid redirect, or a HTTP 200 that tells Outlook XML to redirect elsewhere and finally a HTTP 200 in the form of an XML file that tells outlook to use a different target address.
Check for SRV data
Outlook makes a DNS query for "_autodiscover._tcp.<domain name>" and loops through the results looking for the first record that uses https as its protocol. Outlook then tries to retrieve the payload from that URL. If this step does not retrieve a payload, Outlook moves to check O365 as a failsafe
Check O365 as a failsafe
If all the previous steps didn't return a payload, Outlook uses a less restrictive set of heuristics to decide whether a final attempt to the O365 endpoints are potentially helpful. If outlook decides that an attempt is worthwhile, it tries the known O365 Autodiscover endpoints in case the account is an O365 account. This attempt uses the same target URLs as the “check for O365 as a priority” and differs only in the fact that it's tried as a last resort and not earlier in the Autodiscover process.
Auto configuration failure
If you get through this list above and nothing has been satisfied correctly, then Outlook will fail to configure itself automatically and require you to do it manually, however, setting up Outlook manually means that something isn’t working as it should be and it should be a red flag symptom and not a workaround situation.
Autodiscover Summary : Failed
We now have the requirements for when Autodiscover starts and processes data, and we also now know how the process works and what needs to happen for a successful connection.
Autodiscover Failed : Check your logic!
Confirm it has failed in Outlook by creating a profile with no email account, then when Outlook is started on the taskbar you should see an Outlook icon, hold down Ctrl and right click that icon and then choose "Test E-mail AutoConfiguration" as below:
Then you need to enter your e-mail address and only choose the "Use Autodiscover" option as below, then click test:
This will then fail like this, this should be another "red flag" that something is setup correctly withy our configuration:
Then you need to enter your e-mail address and only choose the "Use Autodiscover" option as below, then click test:
This will then fail like this, this should be another "red flag" that something is setup correctly withy our configuration:
Fiddler : Wireshark but not!
If you’re looking to analyze network traffic the usual go to is Wireshark, however, you are not really interested about other networking protocols that will be active on your network like DHCP requests and offers, and while you have an interest in DNS, you don’t need to know every DNS look up being requested from your network card because you’re only interested in a certain small envelope for Outlook.
Yes, you can absolutely filter all this information, however, there is another way to accomplish this with a tool called Fiddler,
Fiddler Introduction
This particular tool is more focused around web requests than networking packets - so for the purpose of fixing this problem it’s also run in the User context, this means if the current User session is not requesting anything web-based, you won’t get anything logged in Fiddler - therefore Fiddler is like the F12 developer to when it’s not a web browser.
Fiddler Installation
All your need for this section is the download link which you can get from here
Once downloaded, install the application, of which you will require administrative credentials and then the install is complete.
Fiddler Configuration
Once you have installed Fiddler startup the application, you will be asked about AppContainers you want to say "No" to this:
If you try to look at the data it will look like this, all grey and just a "tunnel to" address with no details, this is due to the fact that's its all encrypted, which is not good to anymore.
Therefore need to turn on HTTPS decryption to make this traffic useful, so complete that follow the steps in the next section.
Therefore need to turn on HTTPS decryption to make this traffic useful, so complete that follow the steps in the next section.
Enable HTTPS Decryption
Then when the application starts navigate to Tools then Options as below:
Then navigate to the HTTPS section as below:
You will need to put a tick in the box "Decrypt HTTPS traffic" you will then see a alert box, do not be scared, but you need to click "Yes" to this:
That will then ask you if you wish to trust the FiddletRoot certificates which is required for this software to work as it should, so click yes:
You will then be asked to confirm this action again like this, and "Yes" is what you need to say to this as well:
Then navigate to the HTTPS section as below:
You will need to put a tick in the box "Decrypt HTTPS traffic" you will then see a alert box, do not be scared, but you need to click "Yes" to this:
That will then ask you if you wish to trust the FiddletRoot certificates which is required for this software to work as it should, so click yes:
You will then need to approve on the secure desktop or enter your administrator password to get this stored, if all works well you should see the "added" message as below:
Check HTTPS Decryption (or MiTM)
Now it has been enabled, visit the same site again and this time you should get more relevant data in the Fiddler console as below, this will show you more data than before like HTTP status, HTTP codes and content type - much better!
Fiddler capturing data
When you start Fiddler it will automatically capture data, however the F12 key with Fiddler as the window in focus will either stop or start this capture, once you have you target data captured then you can stop the capture with F12 and then save the file as a SAZ file.
If you wish to save the trace you can do that using File>Save>All Session as below:
If you wish to save the trace you can do that using File>Save>All Session as below:
Fiddler : Capturing Traces Advice
In order to troubleshoot a problem you need a working version and then a non-functional version so you can do some comparisons, trying to troubleshoot with only a broken trace can sometimes be quite hard when connectivity fails or becomes faulty Outlook won’t try other websites or services, so if you only have the broken trace attempt to analyse you won’t get a true picture of the full communication path.
Capture : Working Outlook
Once Outlook has started and you start to receive email leave it a couple of seconds then stop the capture then you will need to save all the sessions as a SAZ file for later analysis.
Note : If you think this data may have sensitive information like passwords and usernames, you will have the option to save as a password protected SAZ file, Just don’t forget the password!
Capture : None-working Outlook
You need to do the same capture with the non-functional Outlook, however, ensure after it failed, you leave a couple of seconds or to get more accurate data try the operation again so you get a couple of attempts then you can scrutinize the data later.
Fiddler : Analysis of the functional Outlook
The working flow looks like this:
This shows that the path is:
1. autodiscover.grizzly.me
2. <redirection using 302 shown in image>
2. <redirection using 302 shown in image>
3. autodiscover.magicalbear.onmicrosoft.com
4. autodiscover-s.outlook.com
5. outlook.office365.com
Fiddler : Analysis of the non-functional Outlook
This path is very broken, here we do not get beyond the first autodiscover, so in this example the path goes:
autodiscover.grizzly.me
<broken and failed>
That means we never get as far as these lookups, so autodiscover will fail, the errors will be shown in the next section.
autodiscover.magicalbear.onmicrosoft.com
autodiscover-s.outlook.com
outlook.office365.com
Autodiscover : Investigate the failures
We can see the start of the troubles when we seem to get a HTTP 401 (Unauthorized) then move on a HTTP 502 (Bad Gateway) as below:
Note : In this particular instance, as outlined above in the workflow, the first request is. HTTPS that failed, then we move along to HTTP in this particular set up this will also fail, we will get to that in the moment.
If we then look at the HTTP 502 error, which at this point is not using the secure channel, if we click on the web view option, we can get the error in all its glory which you can see below:
So the server is requiring a secure connection and without that secure connection, it’s not accepting any requests which explains the error, but it doesn’t quite fully explain the error as the service should be sufficiently able to authenticate correctly with secured autodiscover.
Autodiscover Failure : Proxy solution
In this case of this example if would appear that when you tried to communicate with the HTTP 302 redirection server which was utodiscover.magicalbear.onmicrosoft.com that it never got this far.
This would indicate that something was playing around with the HTTPS traffic with Man-in-the-middle (MiTM) or we are breaking the session with authentication policies, just to be clear Outlook does not support MiTM or authentication that is not direct internet.
if you have a security proxy, that can also apply additional authentication and other security measures remember that Outlook is one of those clients that does not support additional inspection or authentication rules or policies.
if you’re requirements are to inspect this traffic and apply additional security filtering you will need to use Outlook web access and not Outlook.