I was dealing with a problem the other day, where in this case Absolute which is a VPN solution, was trying to authenticate computers using a device authentication certificate, but that process was failing.
Note : In this particular example, network outages were preventing internal and external communication communications with infrastructure for extended timeframes.
The VPN software simply reported that EAP has failed, this is not actually the fault of the VPN software but the upstream authentication service.
It is particular example the Network Policy Server (NPS) was not accepting any requests from computers, or more technically was not authorizing the EAP request, this caused the VPN connection to fail.
This is how the process should work:
Client > VPN > Authentication Policy > EAP request > NPS > EAP Sucess > VPN > Approve Connection > VPN Active
However, what was going on was more like this, The changes here are in bold:
Client > VPN > Authentication Policy > EAP request > NPS > EAP Fail > Deny VPN connection
We now have to figure out why NPS denied the connection, as with many things, the event log is the first place to start, We need to look for the NPS events for authorization and rejection, which you can find in the following event log source:
I observed immediately that every time a computer was trying to authenticate. It was immediately being rejected, annoyingly you will not get a red cross for the rejection it will still be an informational alert - however, when troubleshooting errors like this, you should not be looking for the pretty red crosses or yellow exclamation marks.
This is how it should look when "normal":
However we did not get Event ID 6272 we were getting Event ID 6273 which you can see below:
You should say a whole screen of these errors because every device authenticating seems to be being rejected it’s EAP token, that entry should start with the text:
Network Policy Server denied access to a user.
The NPS server does not differentiate computer/users on the failure event so everything will be "user" regardless of what you are trying to authenticate.
The first order of troubleshooting is to make sure it is receiving the correct policies which can be viewed from this section of the event log data, the same section of data will also give you the error as well, that is all under the Authentication Details section as below:
Authentication Details:
Connection Request Policy Name: VPN : CRP Validator
Network Policy Name: VPN : Device Based Authentication
Authentication Provider: Windows
Authentication Server: nps1.bear.local
Authentication Type: EAP
EAP Type: Microsoft: Smart Card or other certificate
Account Session Identifier: -
Logging Results: Accounting information was written to the local log file.
Reason Code: 259
Reason: The revocation function was unable to check revocation because the revocation server was offline.
NPS uses a connection request policy (CRP) and then a network policy (NP) - The connection request policy filters, basic information about the type of connection and what’s being sent in the EAP token, whereas the network policy defines what type of devices can connect under a certificate being used to verify them.
The stage of the check confirms you are receiving the correct policies and you’ve not defaulted to the inbuilt connections that are usually called “Deny all other connections” - if you have not matched your policy criteria, then something different to this article has changed, which requires further investigation, In this example, the policies match exactly what they should be.
If all your policies match, you can then move onto the error details at the bottom of the event data, This will usually give a disconnection code and then a reason, in this example, we were getting the disconnection reason of:
Reason Code: 259
Reason: The revocation function was unable to check revocation because the revocation server was offline.
This problem would indicate the NPS is not able to check the revocation certificate which is causing the issues with the disconnections.
If your VPN is critical then the first order is business is to fix the issue and allow people to connect which means you can then investigate this issue after restoring service to the end users, in order to fix this we need to check the "Revocation Check" so far that on the NPS server navigate to this registry location:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\RasMan\PPP\EAP\13
Then from here you want to set a new key called NoRevocationCheck as a DWORD (32bit) to the Value of 0 so in summary that is:
Key Type : DWORD (32bit) value
Key Name : NoRevocationCheck
Key Type : DWORD (32bit) value
Key Name : NoRevocationCheck
Value : 0 (Zero)
When this key has been added you can then restart the NPS service with the command
Restart-Service IAS
Restart-Service IAS
This will fix you issue with the disconnection, but now we need to look at why it occurred in the first place, this is the more interesting side of the issue.
🦈 Deep Dive
If you are talking about CRL (certificate, revocation list) the first eight checks is the local cache, if the information here is not available try to retrieve the new CRL configuration, which will come from external sources, this mechanism fail during network use because it cannot reach the CRL distribution point.
If we are talking about certificate chains, then if one of the validation chains fail, the whole chain fails by default because it requires the food chain to be validated before it can be class as healthy and available.
if either of these systems fail, or cannot talk to the end points required, then NPS will not approve EAP tokens, which is exactly the problem we are experiencing here.
this means the cash data is no longer valid and to get the service back online. We need to flush the cash data for CRL distribution points and certificate chains to accomplish this we need the following command:
certutil -urlcache * delete
This should confirm that the local cache has indeed been flushed as below:
This will get the service back online however, remember you will need to restart NPS but if you're curious as to why this problem has happened, we can do a little bit more of a deep deep dive.
🦑 Deeper Dive
Note : You may find not all these commands are required, but you need to adapt the commands to your unique situation.
The first come out will check the event log for certificate based failures that will be relevant to Certificate/CRL events:
Get-WinEvent -LogName "Application" | Where-Object { $_.Source -like "*Certificate*" } | Select-Object TimeCreated, Message -First 20
Then we need to check the certificate operation log with this:
Get-WinEvent -LogName "Microsoft-Windows-CertificateServices*" -ErrorAction SilentlyContinue | Where-Object { $_.Level -eq 2 -or $_.Level -eq 3 } | Select-Object TimeCreated, Message -First 20
Then we need to check The status of historical download attempts:
certutil -urlcache * history
Then finally, we need to check the event log for certificate validation, failures with this command:
Get-WinEvent -LogName "Application" | Where-Object { $_.Message -like "*CRL*" -or $_.Message -like "*Certificate*" } | Select-Object TimeCreated, Message -First 20
After analyzing the data, I could see issues with the CRL distribution points, not being accessible during The network outage has mentioned earlier, so this got me, wondering let’s look at the default timeout values:
CRL Download Timeout: 15 seconds
CRL Validity Period: 1 week (168 hours)
CRL Overlap Period: 10% of validity period (16.8 hours)
CRL Period Units: 1 (hours)
CRL Delta Period: 24 hours
The one that stands out at me is to CRL download timeout, Now we need to check for values for The retry interval and the retry count:
# Check Timeout settings
$configPath = "HKLM:\SOFTWARE\Microsoft\Cryptography\OID\EncodingType 0\CertDllCreateCertificateChainEngine\Config"
Get-ItemProperty -Path $configPath |
ForEach-Object {
Write-Host "Chain Cache Settings:" -ForegroundColor Green
Write-Host "===================="
$_.PSObject.Properties | Where-Object { $_.Name -notlike 'PS*' } | ForEach-Object {
Write-Host "$($_.Name): $($_.Value)"
}
}
# Check URL timeout
$timeoutValue = Get-ItemProperty -Path $configPath -Name "ChainUrlRetrievalTimeoutSeconds" -ErrorAction SilentlyContinue
if ($timeoutValue) {
Write-Host "CRL Timeout: $($timeoutValue.ChainUrlRetrievalTimeoutSeconds) seconds"
} else {
Write-Host "CRL Timeout: Not configured (using system default)"
}
When Windows will attempt to try and get the CRL distribution points on its next cycle immediately, when this fails, it will do the next retries in two minutes, four minutes, eight minute intervals - if it gets to the fifth attempt and fails, it will fail the whole CRL distribution list.
I’ve been thought about looking at best practices for these timeout values because defaults seem quite aggressive, and the recommended values are as follows:
Production environments (for high availability)
CRL Download Timeout: 30-60 seconds
CRL Validity Period: 336 hours (2 weeks)
CRL Overlap Period: 48 hours
CRL Period Units: 1
CRL Delta Period: 48 hours
Unstable network sites (bandwidth saturated sites)
CRL Download Timeout: 120 seconds
CRL Validity Period: 504 hours (3 weeks)
CRL Overlap Period: 72 hours
CRL Period Units: 1
CRL Delta Period: 72 hours
It is always worse, optimizing settings, using in your infrastructure to make it more resilient against future outages, However, remember increasing the time out, has other side effects, like not requiring more frequent updates of the CRL, which could be a security problem and if you incorrectly set your overlap times, you can generate your own errors in trying to optimize these values.
I have therefore created a script that I’ve used quite successfully on MPS servers to set these values from the default where it deemed applicable, You had the option to use a couple of preset values or manually set the values yourself, If you are going to use manual mode, please be careful about what you’re setting because like I said, you can cause further issues with settings thesde incorrectly.
Script : CRLTimeout.ps1
# Define presets
$global:Presets = @{
'Standard' = @{
Description = "Enterprise Settings - Balanced security and performance"
Settings = @{
CRLDownloadTimeout = 30
CRLValidityPeriod = 168 # 1 week
CRLOverlapPeriod = 24
CRLPeriodUnits = 1
CRLDeltaPeriod = 24
RetryInterval = 30
RetryCount = 5
}
}
'HighAvailability' = @{
Description = "Critical Systems - Maximum uptime and resilience"
Settings = @{
CRLDownloadTimeout = 60
CRLValidityPeriod = 336 # 2 weeks
CRLOverlapPeriod = 48
CRLPeriodUnits = 1
CRLDeltaPeriod = 48
RetryInterval = 60
RetryCount = 8
}
}
'UnstableNetwork' = @{
Description = "Unreliable Networks - Extended timeouts and retries"
Settings = @{
CRLDownloadTimeout = 120
CRLValidityPeriod = 504 # 3 weeks
CRLOverlapPeriod = 72
CRLPeriodUnits = 1
CRLDeltaPeriod = 72
RetryInterval = 120
RetryCount = 10
}
}
}
function Write-Log {
param(
[string]$Message,
[string]$Level = "Info"
)
$timestamp = Get-Date -Format "yyyy-MM-dd HH:mm:ss"
$logMessage = "[$timestamp] [$Level] $Message"
# Add logging to file if needed
Write-Host $logMessage -ForegroundColor $(
switch ($Level) {
"Error" { "Red" }
"Warning" { "Yellow" }
"Success" { "Green" }
default { "White" }
}
)
}
function Get-CRLTimeoutSettings {
try {
$settings = @{
'CRLDownloadTimeout' = Get-ItemProperty -Path 'HKLM:\SYSTEM\CurrentControlSet\Services\CertSvc\Configuration' -Name CRLDownloadTimeout -ErrorAction SilentlyContinue
'CRLValidityPeriod' = Get-ItemProperty -Path 'HKLM:\SYSTEM\CurrentControlSet\Services\CertSvc\Configuration' -Name CRLValidityPeriod -ErrorAction SilentlyContinue
'CRLOverlapPeriod' = Get-ItemProperty -Path 'HKLM:\SYSTEM\CurrentControlSet\Services\CertSvc\Configuration' -Name CRLOverlapPeriod -ErrorAction SilentlyContinue
'CRLPeriodUnits' = Get-ItemProperty -Path 'HKLM:\SYSTEM\CurrentControlSet\Services\CertSvc\Configuration' -Name CRLPeriodUnits -ErrorAction SilentlyContinue
'CRLDeltaPeriod' = Get-ItemProperty -Path 'HKLM:\SYSTEM\CurrentControlSet\Services\CertSvc\Configuration' -Name CRLDeltaPeriod -ErrorAction SilentlyContinue
'RetryInterval' = Get-ItemProperty -Path 'HKLM:\SOFTWARE\Microsoft\Cryptography\OID\EncodingType 0\CertDllCreateCertificateChainEngine\Config' -Name RetryInterval -ErrorAction SilentlyContinue
'RetryCount' = Get-ItemProperty -Path 'HKLM:\SOFTWARE\Microsoft\Cryptography\OID\EncodingType 0\CertDllCreateCertificateChainEngine\Config' -Name RetryCount -ErrorAction SilentlyContinue
}
return $settings
}
catch {
Write-Log "Error retrieving CRL settings: $_" -Level "Error"
return $null
}
}
function Show-CurrentSettings {
$settings = Get-CRLTimeoutSettings
if ($null -eq $settings) {
Write-Log "Unable to retrieve current settings." -Level "Error"
return
}
Write-Host "`nCurrent CRL Settings:" -ForegroundColor Green
Write-Host "================================"
Write-Host "1. CRL Download Timeout: $($settings.CRLDownloadTimeout.CRLDownloadTimeout) seconds"
Write-Host "2. CRL Validity Period: $($settings.CRLValidityPeriod.CRLValidityPeriod) hours"
Write-Host "3. CRL Overlap Period: $($settings.CRLOverlapPeriod.CRLOverlapPeriod) hours"
Write-Host "4. CRL Period Units: $($settings.CRLPeriodUnits.CRLPeriodUnits)"
Write-Host "5. CRL Delta Period: $($settings.CRLDeltaPeriod.CRLDeltaPeriod) hours"
Write-Host "6. Retry Interval: $($settings.RetryInterval.RetryInterval) seconds"
Write-Host "7. Retry Count: $($settings.RetryCount.RetryCount) attempts"
Write-Host "================================`n"
}
function Test-ValidSettingRange {
param (
[string]$Setting,
[int]$Value
)
$validRanges = @{
'CRLDownloadTimeout' = @{Min = 15; Max = 120}
'CRLValidityPeriod' = @{Min = 24; Max = 504}
'CRLOverlapPeriod' = @{Min = 12; Max = 72}
'CRLPeriodUnits' = @{Min = 1; Max = 1}
'CRLDeltaPeriod' = @{Min = 24; Max = 72}
'RetryInterval' = @{Min = 30; Max = 120}
'RetryCount' = @{Min = 3; Max = 10}
}
$range = $validRanges[$Setting]
if ($Value -lt $range.Min -or $Value -gt $range.Max) {
Write-Log "Warning: $Setting value $Value is outside recommended range ($($range.Min)-$($range.Max))" -Level "Warning"
return $false
}
return $true
}
function Set-CRLTimeout {
param (
[string]$Setting,
[int]$Value,
[string]$Path
)
try {
if (-not (Test-ValidSettingRange -Setting $Setting -Value $Value)) {
$confirmation = Read-Host "Do you want to proceed with this value outside the recommended range? (yes/no)"
if ($confirmation -ne 'yes') {
Write-Log "Operation cancelled by user" -Level "Warning"
return
}
}
# Backup current value
$currentValue = (Get-ItemProperty -Path $Path -Name $Setting -ErrorAction SilentlyContinue).$Setting
Set-ItemProperty -Path $Path -Name $Setting -Value $Value -Type DWORD
Write-Log "Successfully updated $Setting from $currentValue to $Value" -Level "Success"
# Verify the change
$newValue = (Get-ItemProperty -Path $Path -Name $Setting).$Setting
if ($newValue -ne $Value) {
throw "Verification failed: New value ($newValue) does not match intended value ($Value)"
}
}
catch {
Write-Log "Error updating setting: $_" -Level "Error"
# Attempt to rollback
if ($null -ne $currentValue) {
try {
Set-ItemProperty -Path $Path -Name $Setting -Value $currentValue -Type DWORD
Write-Log "Rolled back $Setting to previous value: $currentValue" -Level "Warning"
}
catch {
Write-Log "Failed to rollback setting! Manual verification required." -Level "Error"
}
}
}
}
function Show-ManualConfigWarning {
Write-Host "`n⚠️ WARNING ⚠️" -ForegroundColor Red
Write-Host "===========================================" -ForegroundColor Red
Write-Host "MANUAL CONFIGURATION RISK ALERT" -ForegroundColor Red
Write-Host "===========================================" -ForegroundColor Red
Write-Host "- This operation should only be performed by qualified administrators" -ForegroundColor Red
Write-Host "- Incorrect values can cause system-wide certificate validation issues" -ForegroundColor Red
Write-Host "- Changes may impact all certificate operations in your environment" -ForegroundColor Red
Write-Host "- Consider using presets if you're unsure about specific values" -ForegroundColor Red
Write-Host "- Document your changes for future reference" -ForegroundColor Red
Write-Host "===========================================" -ForegroundColor Red
$confirmation = Read-Host "`nDo you understand the risks and want to proceed with manual configuration? (yes/no)"
return $confirmation.ToLower() -eq 'yes'
}
function Apply-Preset {
param (
[string]$PresetName
)
$preset = $global:Presets[$PresetName]
if ($null -eq $preset) {
Write-Log "Invalid preset name: $PresetName" -Level "Error"
return
}
Write-Log "Applying $PresetName preset ($($preset.Description))..." -Level "Info"
# Backup current settings
$currentSettings = Get-CRLTimeoutSettings
$backupFile = "CRL_Settings_Backup_$(Get-Date -Format 'yyyyMMdd_HHmmss').json"
$currentSettings | ConvertTo-Json | Out-File $backupFile
Write-Log "Current settings backed up to $backupFile" -Level "Info"
try {
foreach ($setting in $preset.Settings.GetEnumerator()) {
$path = if ($setting.Key -in @('RetryInterval','RetryCount')) {
'HKLM:\SOFTWARE\Microsoft\Cryptography\OID\EncodingType 0\CertDllCreateCertificateChainEngine\Config'
} else {
'HKLM:\SYSTEM\CurrentControlSet\Services\CertSvc\Configuration'
}
Set-CRLTimeout -Setting $setting.Key -Value $setting.Value -Path $path
}
Write-Log "Preset applied successfully!" -Level "Success"
}
catch {
Write-Log "Error applying preset: $_" -Level "Error"
Write-Log "Settings backup available in: $backupFile" -Level "Warning"
}
}
function Show-PresetMenu {
Write-Host "`nAvailable Presets:" -ForegroundColor Cyan
Write-Host "===================="
foreach ($preset in $global:Presets.GetEnumerator()) {
Write-Host "`n$($preset.Key):" -ForegroundColor Yellow
Write-Host $preset.Value.Description
Write-Host "Settings:"
foreach ($setting in $preset.Value.Settings.GetEnumerator()) {
Write-Host " $($setting.Key): $($setting.Value)"
}
}
$choice = Read-Host "`nEnter preset name to apply (or 'back' to return)"
if ($choice -ne 'back') {
Apply-Preset -PresetName $choice
}
}
function Show-Menu {
do {
Show-CurrentSettings
Write-Host "`nSelect an option:"
Write-Host "1. Apply Preset Configuration (Recommended)"
Write-Host "2. Modify CRL Download Timeout (Manual Configuration)"
Write-Host "3. Modify CRL Validity Period (Manual Configuration)"
Write-Host "4. Modify CRL Overlap Period (Manual Configuration)"
Write-Host "5. Modify CRL Period Units (Manual Configuration)"
Write-Host "6. Modify CRL Delta Period (Manual Configuration)"
Write-Host "7. Modify Retry Interval (Manual Configuration)"
Write-Host "8. Modify Retry Count (Manual Configuration)"
Write-Host "Q. Quit"
$choice = Read-Host "`nEnter your choice"
switch ($choice.ToLower()) {
'1' { Show-PresetMenu }
'2' {
if (Show-ManualConfigWarning) {
Write-Host "`nRecommended range: 15-120 seconds" -ForegroundColor Yellow
Write-Host "Default value: 15 seconds" -ForegroundColor Yellow
$value = Read-Host "Enter new CRL Download Timeout (in seconds)"
Set-CRLTimeout -Setting "CRLDownloadTimeout" -Value $value -Path 'HKLM:\SYSTEM\CurrentControlSet\Services\CertSvc\Configuration'
}
}
'3' {
if (Show-ManualConfigWarning) {
Write-Host "`nRecommended range: 24-504 hours (1-21 days)" -ForegroundColor Yellow
Write-Host "Default value: 168 hours (7 days)" -ForegroundColor Yellow
$value = Read-Host "Enter new CRL Validity Period (in hours)"
Set-CRLTimeout -Setting "CRLValidityPeriod" -Value $value -Path 'HKLM:\SYSTEM\CurrentControlSet\Services\CertSvc\Configuration'
}
}
'4' {
if (Show-ManualConfigWarning) {
Write-Host "`nRecommended range: 12-72 hours" -ForegroundColor Yellow
Write-Host "Default value: 16.8 hours" -ForegroundColor Yellow
$value = Read-Host "Enter new CRL Overlap Period (in hours)"
Set-CRLTimeout -Setting "CRLOverlapPeriod" -Value $value -Path 'HKLM:\SYSTEM\CurrentControlSet\Services\CertSvc\Configuration'
}
}
'5' {
if (Show-ManualConfigWarning) {
Write-Host "`nRecommended value: 1 (hours)" -ForegroundColor Yellow
Write-Host "Default value: 1" -ForegroundColor Yellow
$value = Read-Host "Enter new CRL Period Units"
Set-CRLTimeout -Setting "CRLPeriodUnits" -Value $value -Path 'HKLM:\SYSTEM\CurrentControlSet\Services\CertSvc\Configuration'
}
}
'6' {
if (Show-ManualConfigWarning) {
Write-Host "`nRecommended range: 24-72 hours" -ForegroundColor Yellow
Write-Host "Default value: 24 hours" -ForegroundColor Yellow
$value = Read-Host "Enter new CRL Delta Period (in hours)"
Set-CRLTimeout -Setting "CRLDeltaPeriod" -Value $value -Path 'HKLM:\SYSTEM\CurrentControlSet\Services\CertSvc\Configuration'
}
}
'7' {
if (Show-ManualConfigWarning) {
Write-Host "`nRecommended range: 30-120 seconds" -ForegroundColor Yellow
Write-Host "Default value: 30 seconds" -ForegroundColor Yellow
$value = Read-Host "Enter new Retry Interval (in seconds)"
Set-CRLTimeout -Setting "RetryInterval" -Value $value -Path 'HKLM:\SOFTWARE\Microsoft\Cryptography\OID\EncodingType 0\CertDllCreateCertificateChainEngine\Config'
}
}
'8' {
if (Show-ManualConfigWarning) {
Write-Host "`nRecommended range: 3-10 attempts" -ForegroundColor Yellow
Write-Host "Default value: 5 attempts" -ForegroundColor Yellow
$value = Read-Host "Enter new Retry Count"
Set-CRLTimeout -Setting "RetryCount" -Value $value -Path 'HKLM:\SOFTWARE\Microsoft\Cryptography\OID\EncodingType 0\CertDllCreateCertificateChainEngine\Config'
}
}
'q' {
Write-Host "Exiting..."
return
}
default {
Write-Host "Invalid choice. Please try again." -ForegroundColor Yellow
}
}
Write-Host "`nPress any key to continue..."
$null = $Host.UI.RawUI.ReadKey('NoEcho,IncludeKeyDown')
Clear-Host
} while ($true)
}
# Check if running as administrator
if (-NOT ([Security.Principal.WindowsPrincipal][Security.Principal.WindowsIdentity]::GetCurrent()).IsInRole([Security.Principal.WindowsBuiltInRole] "Administrator")) {
Write-Warning "You need to run this script as an Administrator!"
Exit
}
# Create log directory if it doesn't exist
$logDir = "C:\Logs\CRLManager"
if (-not (Test-Path $logDir)) {
New-Item -ItemType Directory -Path $logDir -Force | Out-Null
}
# Start the script
try {
Write-Log "Starting CRL Management Script" -Level "Info"
Show-Menu
}
catch {
Write-Log "Critical error in script execution: $_" -Level "Error"
}
finally {
Write-Log "Script execution completed" -Level "Info"
}
This script when run first you can see we have no values set as they are on defaults:
If we then choose the presets we can select which option we would like:
If you choose manual configuration we get the warning then we can update the value:
If we then choose the presets we can select which option we would like:
If you choose manual configuration we get the warning then we can update the value: