Crash Dumps 101
Crash dump basically contains the current working state of the program which has terminated abnormally.
Crash dumps can also give us a complete state picture of the state of what is in memory as well this can be used for analyzing the problem. The simplest way to take the crash dump is "procdump" which is a sysinternals tools which you can get from here
User and Kernel Mode address space
When talking about crashing you have two options you have User mode and Kernel mode, when you have a process in user mode Windows creates a process for it with private virtual address space and a private handle table, since this address is private, one application cannot modify other applications address space.
If the same applications runs in kernel mode it shares a single virtual address space. As a result, a kernel-mode driver isn't isolated from other drivers or the operating system. If a kernel-mode driver mistakenly writes to the wrong virtual address, it could compromise data belonging to the operating system or another driver. If a kernel-mode driver crashes, it causes the entire operating system to crash.
WinDbg - Hello World, welcome
Now that we have got the dump, we need to analyze the dump. for this we need windbg (Windows Debugging) which is part of the ADK or SDK when you install that the option is "Debugging Tools" as below, here we see the SDK:
PDB File / Symbols Server
Once you have Debugging tools installed you need the PDB file to figure out why the application has crashed, however you can only get the PDB files if you have the source code for the application was built, however Microsoft have a "symbols" servers that store these PDB/symbols.
This means once you have WinDbg installed and its started up, you first need to configure the symbols search path to this location, this will tell WinDbg to connect to the symbols server and download the PDB files you do not have, magic!
Types of Dump Files (and descriptions)
You can specify a couple types of dump files depending how "deep" you want to go into the finding the fault these are defined in the "System" section of Windows to get there type in:
control sysdm.cpl
control sysdm.cpl
Then you wish to navigate to Advance > Startup and Recovery > Settings > System Failure to see you options as below, if you change any these options a reboot is required for this to take effect.
Types of Dump file are outlined now below, usually for basic diagnostics and if space in an issue the "small dump files" can lead you to some very helpful insights.
Complete memory dump
A complete memory dump records all the contents of system memory when your computer stops unexpectedly. A complete memory dump may contain data from processes that were running when the memory dump was collected.
If you select the Complete memory dump option, you must have a paging file on the boot volume that is sufficient to hold all the physical RAM plus 1 megabyte (MB).
Kernel memory dump
A kernel memory dump records only the kernel memory. It speeds up the process of recording information in a log when your computer stops unexpectedly. You must have a pagefile large enough to accommodate your kernel memory. For 32-bit systems, kernel memory is usually between 150 MB and 2 GB.
This dump file doesn't include unallocated memory or any memory that's allocated to User-mode programs. It includes:
- Memory that's allocated to the kernel and hardware abstraction layer (HAL) in Windows 2000 and later.
- Memory that's allocated to Kernel-mode drivers and other Kernel-mode programs.
- For most purposes, this dump file is the most useful. It's smaller than the complete memory dump file. But it omits only those parts of memory that are unlikely to have been involved in the problem.
Small memory dump
A small memory dump records the smallest set of useful information that may help identify why your computer stopped unexpectedly. This option requires a paging file of at least 2 MB on the boot volume and specifies that Windows 2000 and later create a new file every time your computer stops unexpectedly. A history of these files is stored in a folder.
This dump file type includes the following information:
- The Stop message and its parameters and other data
- A list of loaded drivers
- The processor context (PRCB) for the processor that stopped
- The process information and kernel context (EPROCESS) for the process that stopped
- The process information and kernel context (ETHREAD) for the thread that stopped
- The Kernel-mode call stack for the thread that stopped
Windows Debugging (WinDbg)
Now you have installed WinDbg via the SDK (in our case) start up the application, but you need to start the correction application, you will have a couple of choices here and that depends on the application you are debugging, it is highly recommended to get the right version of WinDbg
WinDbg x64 - 64bit applications
WinDbg x86 - 32 but applications
WinDbg ARM - ARM 32bit applications
WinDbg ARM64 - ARM 64bit applications
In this example I will be using WinDbg x64 as Edge is a x64 application:
Once WinDbg has started you need to click File > Symbols File Path as below:
Then in the symbols path you need to enter the command below, and then you have the visual as well:
SRV*C:\SymCache*http://msdl.microsoft.com/download/symbols
That should look like this, when you can click OK.....
Symbols Folder
The folder you specify here before the URL is where WinDbg will store the PDB/symbols file from the symbols server, when you open a DMP file it will download all the PDB files it requires to complete the trace:
To give an example when I was tracing issues with crypt32.dll the symbols server provided me with the crypt32.pdb so that I could complete the stack trace with WinDbg, as you can see below there is the PDB file:
Once you have got to this stage where the user symbols are loaded and the module list is ready, you then need to enter this into the debugger command window:
!analyze -v
Well if you go the debugger command line and enter this command
lmvm nvlddmkm
That should look like this:
This will then show you information about the problematic component, which in this case appears to be a nVidia driver......
This driver Nvlddmkm.sys is a system file associated with NVIDIA graphics cards. It is responsible for managing the communication between the operating system and the graphics card. This file is crucial for the proper functioning of your graphics card and any issues with it can lead to system instability and crashes
Provide the DMP (Small Dump File)
Once you have the DMP file and WinDbg is configured as per the instructions above then you need to open the DMP file in WinDbg from the File > Open Crash Dump
Then you need to choose that file, for this example I have some DMP files in a folder I have created in C:\DMP-Files this is my working folder if you like...
So from WinDbg lets point it at that folder and choose the first DMP file as you can see below:
When you open the file it will show you some data as below, firstly the path validation then the OS version and build, you will also see the symbol search path in the debugger as that is what you should have specified:
Then you need to choose that file, for this example I have some DMP files in a folder I have created in C:\DMP-Files this is my working folder if you like...
So from WinDbg lets point it at that folder and choose the first DMP file as you can see below:
When you open the file it will show you some data as below, firstly the path validation then the OS version and build, you will also see the symbol search path in the debugger as that is what you should have specified:
Once you have got to this stage where the user symbols are loaded and the module list is ready, you then need to enter this into the debugger command window:
!analyze -v
This is shown below:
When you press enter it will get to work and you will see the Bug check analysis and you will notice it will say *BUSY* at the debugger command prompt:
After a short wait you will get clues to the issue as you can see below, we seem to have a problem with the driver nvlddmkm.sys this is causing a critical event that is halting Windows and causing the BSOD (Blue Screen of Death) but what is this file???
When you press enter it will get to work and you will see the Bug check analysis and you will notice it will say *BUSY* at the debugger command prompt:
After a short wait you will get clues to the issue as you can see below, we seem to have a problem with the driver nvlddmkm.sys this is causing a critical event that is halting Windows and causing the BSOD (Blue Screen of Death) but what is this file???
Well if you go the debugger command line and enter this command
lmvm nvlddmkm
That should look like this:
This will then show you information about the problematic component, which in this case appears to be a nVidia driver......
This driver Nvlddmkm.sys is a system file associated with NVIDIA graphics cards. It is responsible for managing the communication between the operating system and the graphics card. This file is crucial for the proper functioning of your graphics card and any issues with it can lead to system instability and crashes
Yes, indeed in this case this system instability has a BSOD as it got very unstable, in the instance the problem has easily fixed by downloading the latest drivers from NVidia which actually fixed the problem, many of the problems here are drivers out of date and not kept up to date.
Decision Point - Application or Operating System
Which type fo crash as you need to follow the correct option, if you have something crashing within Windows then you need to look at the application crash which is below, however if you have no operating system as you are looking at one of these, which is a BSOD then you need the "operating system" guide which is below the application one.
However I did prefer a BSOD when it was a little more technical abd gave you something on the screen rather than nothing but a QR code that is not helpful at all!
Application Crashes (not Windows Crash Dumps)
When applications crash in Windows in user mode they crash out the application, so you see an application error occur like this, this is a crash in a user application in this instance msedge.exe or Microsoft Edge:
This will then be followed up in the "Application" eventlog with an Event ID 1000 or an application error as below:
However if take a look at the error more closely you will notice you get a report ID as shown below, on the new versions of Windows you will get a path to this report, but here we do not:
However, if we navigate to the path C:\ProgramData\Microsoft\Windows\WER you will notice you have three folders in that path as below:
Now if you enter the ReportArchive folder you will see all the "reports" from crashes:
If you the search for msedge which is the faulty process you will see them all groups like this and you need to go with the crash time
If you correlate the date and time this correct trace for this is the one shown below:
When you enter that folder you will see the Report.wer
However that file tells us that it crashed but did not create a DMP file, so there is not analyses to do there....
FriendlyEventName=Stopped working
ConsentKey=APPCRASH
AppName=Microsoft Edge
AppPath=C:\Program Files (x86)\Microsoft\Edge\Application\msedge.exe
NsPartner=windows
We do not have the dump (DMP) file so we need to create a process in procdump to catch the next crash, and yes the application needs to crash, to complete this action procdump should be configured before the application crashes like this:
procdump -ma -x c:\dumps "C:\Program Files (x86)\Microsoft\Edge\Application\msedge.exe"
procdump -ma -x c:\dumps "C:\Program Files (x86)\Microsoft\Edge\Application\msedge.exe"
Then it is a waiting game for a crash, and once it crashes you will then get a dump file that we can later analyse for juicy goodness, when you do get a crash procdump take the full memory dump and save it to c:\dumps - as per the command we used.
Windows crashes (Known working cause)
First, it’s always best to start with a crash you can easily trace so you get the hang of it, this is not about knowing what caused the crash as that’s obvious. It will be the utility advise in a bit, the reason for doing it this way is so you can learn how to read debug files with a known variable that’s consistent.
The tool we will be using is called “not my fault” by sysinternals and the purpose of this tool is to crash or in this particular instance blue screen the server you’re working on, so without further ado, download the tool from this link
Then, when you run the application, that’s quite stunningly simple you are after the tablet says crash and you will have a choice of crashes you would like to generate, this is shown below:
⚠️ WARNING : Please be aware that when you click the button marked crash, the server you are currently using will blue screen and fail, once it has failed, it will generate the relevant dump file and then restart, It goes without saying, do not run command like this on a production system, really you should be using a development machine
⛔️ Any unsaved Work on this device will be permanently lost, ensure you save any unsaved work!!!
When you have read the warning above and you have accepted it, click the button labeled “Crash” to have your windows device, spectacularly blue screen, and reboot.
Now your device is rebooted we can get to work on analyzing that dump file which will be in the location C:\Windows and it will be called memory.dmp and here it is:
SRV*C:\SymCache*http://msdl.microsoft.com/download/symbols
Then you want to File > Open Crash Dump and choose the memory.dmp in out folder as below:
Then it will analyse the DMP file and get the symbols as required give this a moment that will look like this, maybe not exactly but something similar:
Then as before in the debugging command section we need to enter this:
!analyze -v
This is shown below:
When you press enter it will get to work and you will see the Bug check analysis and you will notice we get an DRIVE_IRQL_NOT_LES_OR_EQAL error of a "D1"
When you press enter it will get to work and you will see the Bug check analysis and you will notice we get an DRIVE_IRQL_NOT_LES_OR_EQAL error of a "D1"
The stack text is then starting the bug check as you would expect, we caused one, but the first entry is the KeBugCheckEx call then a couple of lines down you see myfault which is intresting.....
STACK_TEXT:
ffff810d`cc213678 fffff803`602356a9 : 00000000`0000000a ffffb404`54fdd660 00000000`00000002 00000000`00000000 : nt!KeBugCheckEx
ffff810d`cc213680 fffff803`60230e4c : ffffa097`b40bd4c7 fffff240`0657b990 00000000`00000fff 00000000`00000f4d : nt!KiBugCheckDispatch+0x69
ffff810d`cc2137c0 fffff803`727812d0 : 00000000`00000000 fffff240`04e367e0 00000000`00000000 000001cb`aba25c40 : nt!KiPageFault+0x44c
ffff810d`cc213950 fffff803`7278168e : 00000000`00000000 fffff240`0476e010 00000000`00000000 ffff8281`ac071080 : myfault+0x12d0
ffff810d`cc213980 fffff803`727817f1 : 00000000`00000002 00000000`00000000 00000000`000000f0 fffff803`604a5f21 : myfault+0x168e
ffff810d`cc213ac0 fffff803`60049025 : 00000000`00000002 00000000`00000001 ffff810d`cc213ea0 00000000`00000000 : myfault+0x17f1
ffff810d`cc213b20 fffff803`604bc1ee : 00000000`00000001 00000000`00000000 00000000`00000001 fffff214`00000000 : nt!IofCallDriver+0x55
ffff810d`cc213b60 fffff803`604a6fdd : ffff8281`00000000 ffff810d`cc213ea0 00000000`00010000 00000000`00000000 : nt!IopSynchronousServiceTail+0x33e
ffff810d`cc213c00 fffff803`604a7096 : 00000000`00050200 00000000`00000000 00000000`00000000 00000000`00000000 : nt!IopXxxControlFile+0xc7d
ffff810d`cc213d40 fffff803`60234d85 : 00000000`00000000 00000000`00000000 00000000`00000000 fffff214`319aee10 : nt!NtDeviceIoControlFile+0x56
ffff810d`cc213db0 00007ffb`2b7602b4 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiSystemServiceCopyEnd+0x25
00000013`e66fea98 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x00007ffb`2b7602b4
Then when you navigate to the summary you see this, which confirms myfault.sys is the cause of the blue screen
SYMBOL_NAME: myfault+12d0
MODULE_NAME: myfault
IMAGE_NAME: myfault.sys
STACK_COMMAND: .cxr; .ecxr ; kb
BUCKET_ID_FUNC_OFFSET: 12d0
FAILURE_BUCKET_ID: AV_myfault!unknown_function
OS_VERSION: 10.0.20348.859
BUILDLAB_STR: fe_release_svc_prod2
OSPLATFORM_TYPE: x64
OSNAME: Windows 10
FAILURE_ID_HASH: {9745090a-9bce-ccba-c096-ca6e9ca04c64}
This is then confirms the issue when you use "lmvm" on the system driver name like this:
lmvm myfault
lmvm myfault
This will show you the details of the faulty system driver, which is our case is expected as that is part of NotMyFault.
start end module name
fffff803`72780000 fffff803`72789000 myfault (no symbols)
Loaded symbol image file: myfault.sys
Image path: \??\C:\Windows\system32\drivers\myfault.sys
Image name: myfault.sys
Browse all global symbols functions data
Timestamp: Thu Sep 29 17:17:31 2022 (6335C51B)
CheckSum: 00010CED
ImageSize: 00009000
Translations: 0000.04b0 0000.04e4 0409.04b0 0409.04e4
If you have no operating system then you have encountered a Windows "safe" shutdown to protect itself or a BSOD, obviously you need to get the file off the computer so if you can restart the computer and get some time before the crash then you can extract it off via the network.
Advanced Boot Menu (F8)
However if you have a BSOD that occurs on boot then you cannot get into the operating system, so as Windows boots press the F8 key for the boot menu and then choose Safe Mode as below:
If this options boots Windows then you likely have a driver or application issue causing the problem, in this particular example safe mode was not available, however you will need to get the file to be sure, so with this mode the networking will not be enabled but if safe mode works you can try another reboot with networking enabled:
Assuming the problem is not with your network drivers this will then allow you to access Windows with basic networking so you can copy the DMP off the device, if you have problems with networking and the devices does not boot you will need to copy this DMP file to a USB drive, alternatively if this is a servers you can use the Hypervisor or iLO access to copy it off.
Assuming the problem is not with your network drivers this will then allow you to access Windows with basic networking so you can copy the DMP off the device, if you have problems with networking and the devices does not boot you will need to copy this DMP file to a USB drive, alternatively if this is a servers you can use the Hypervisor or iLO access to copy it off.
Dump File Location
There are two places for the dump files depending on which one you have asked Windows to produce the locations are:
Full Dump is a file : C:\Windows\Memory.DMP
Full Dump is a file : C:\Windows\Memory.DMP
Kernel/Small will be in the folder : C:\Windows\Minidump\
Additional Troubleshooting Tasks
Usually the stack trace will actually tell the you the name of the system driver or file that is causing the issue, however when its ntkrnlmp.exe you have a corrupt that is not captured in the dump file, so you need to try other steps to narrow down the cause of the issue.
- Safe Mode: Boot into Safe Mode to see if the issue persists. If the issue is fixed, then that would indicate a problem with system drivers or applications.
- Disable Driver Verification Enforcement : This will confirm if you have unsigned drivers causing issues with Windows, if the server boots in this mode, then you have a drvier that is not behaving.
- System File Checker (SFC): Run SFC to scan and repair corrupted system files this can be initiated by sfc /scannow
- Driver Verifier: Use Driver Verifier to identify problematic drivers. It can stress-test drivers and detect issues, this will be covered here in this guide
Safe Mode
We briefly touched on this earlier, if safe mode allows the computer do boot, like it did in my case then the problem will be with drivers and system based drivers that are loading in Kernel mode and causing access violation which are known to cause a blue screen.
Disable Driver Signature Enforcement
This option which can got from the screenshot above, will prevent Windows enforcing drivers checks, if this then allows the computer to boot, then, like in this case, the compass is pointing towards system, drivers or 3rd party drivers.
System File Checker (SFC)
This will versify all the systems are consistent, these files that are critical to the Windows platform running stably and smoothly, to start of the this scan you need this:
sfc /scannow
That will then verify all the file that are required the stability of the Windows, however this scan complete with an issue:
Driver Verification Setup
It has found some files that are "corrupt" or "not valid" as has also had an issue repairing some times, obviously after this we need a reboot.
Driver Verification Pre-Check
First you need make sure everything is disabled before you set this up, so from the command prompt enter this command:
verifier /querysettings
verifier /querysettings
You are looking for the standard flags section, by default it should now have any selections made, like this:
Standard Flags:
[ ] 0x00000001 Special pool.
[ ] 0x00000002 Force IRQL checking.
[ ] 0x00000008 Pool tracking.
[ ] 0x00000010 I/O verification.
[ ] 0x00000020 Deadlock detection.
[ ] 0x00000080 DMA checking.
[ ] 0x00000100 Security checks.
[ ] 0x00000800 Miscellaneous checks.
[ ] 0x00020000 DDI compliance checking.
Driver Verification Setup
You now need to setup the driver verification to check all the drivers for issues and instability
Driver Verification Pre-Reboot
If you enter the same command as before as below:
verifier /querysettings
You are looking for the standard flags section, you should now notice that all the flags are enabled as below:
Standard Flags:
[X] 0x00000001 Special pool.
[X] 0x00000002 Force IRQL checking.
[X] 0x00000008 Pool tracking.
[X] 0x00000010 I/O verification.
[X] 0x00000020 Deadlock detection.
[X] 0x00000080 DMA checking.
[X] 0x00000100 Security checks.
[X] 0x00000800 Miscellaneous checks.
[X] 0x00020000 DDI compliance checking.
Driver Verification (Reboot)
Now this is setup and confirmed, reboot your device....