This is a simple script to monitor a log file in a specified file path and then send out a email notification when that string is found when it is seen more that four (4) time in the log file, that then need to fire off a remediation script to fix the issue and restore service.
Script : JavaRemediate.ps1
# Define parameters
$logFilePath = "C:\Quarantine\JavaHighCPU\Test.log" # Path to the log file
$errorText = "Caused by: java.io.IOException: HTTP/1.1 header parser received no bytes"
$smtpServer = "smtp.bear.local"
$alertRecipient = "lee@croucher.cloud"
$alertSender = "java.process@croucher.cloud"
$emailCooldownPeriod = 30 # Cooldown period in minutes
$htmlTemplatePath = "logmessage.html" # Path to the HTML template
$restartScriptPath = "RestartProcessandEmail.ps1" # Path to the restart script
# Get the server name
$serverName = [Environment]::MachineName
# Initialize last email sent time
$global:lastEmailSent = $null
# Initialize error counter
$global:errorCounter = 0
# Function to send an email alert
function Send-Alert {
param (
[string]$subject,
[string]$body
)
$currentTime = Get-Date
if ($global:lastEmailSent -eq $null -or ($currentTime - $global:lastEmailSent).TotalMinutes -ge $emailCooldownPeriod) {
$htmlContent = Get-Content -Path $htmlTemplatePath -Raw
$htmlContent = $htmlContent -replace "{{SUBJECT}}", $subject
$htmlContent = $htmlContent -replace "{{BODY}}", $body
$emailParams = @{
From = $alertSender
To = $alertRecipient
Subject = $subject
Body = $htmlContent
BodyAsHtml = $true
SmtpServer = $smtpServer
Encoding = [System.Text.Encoding]::UTF8
}
Send-MailMessage @emailParams
# Update the last email sent time
$global:lastEmailSent = $currentTime
# Run the restart script
Run-RestartScript
# Reset the error counter after sending the alert and running the script
$global:errorCounter = 0
}
}
# Function to run the restart script and wait for successful completion
function Run-RestartScript {
Write-Output "Running restart script: $restartScriptPath" # Debug output
$process = Start-Process -FilePath "powershell.exe" -ArgumentList "-ExecutionPolicy Bypass -File `"$restartScriptPath`"" -PassThru -Wait
# Check the exit code
if ($process.ExitCode -eq 0) {
Write-Output "Restart script completed successfully." # Debug output
} else {
Write-Output "Restart script failed with exit code $($process.ExitCode)."
}
}
# Monitor log file for specific text
Get-Content -Path $logFilePath -Wait -Tail 0 | ForEach-Object {
Write-Output "Reading log entry: $_" # Debug output
if ($_ -like "*$errorText*") {
# Increment the error counter
$global:errorCounter++
Write-Output "Trigger $global:errorCounter/4 found" # Output the trigger count
# Check if the counter has reached 4
if ($global:errorCounter -ge 4) {
Write-Output "Trigger threshold reached. Sending alert and running external script." # Debug output
$subject = "⚠️ Alert: Log file error detected on $serverName"
$body = "Error detected in log file: $logFilePath<br><br>Error Details:<br>$_"
Send-Alert -subject $subject -body $body
}
}
}
Provide the Log file path
This the the absolute path the log file you wish to monitor, that in this example is this variable:
$logFilePath = "C:\Quarantine\JavaHighCPU\Test.log" # Path to the log file
If you are not sure about gettting the full path then you can right click on the file (while holding down shift) and choose the "Copy as path option"
If you do not see the option for "Copy as Path" you are not holding down the sift key before you right click the file.
Format the logmessage.htmlThis is the content that will be send when the string is detected, this is what I have used, however you requirements may be different, this is simply HTML design in its most basic form.
If you wish to test this mechanism then you can do, and the script is pointing at a log file in this blog entry and its looking for this:
$errorText = "Error on http communication.java.io.IOException: HTTP/1.1 header parser
This means if you edit that file that "may" be empty and enter that into the file, you should get an email shortfall thereafter, that proves the logic works.
Multiple Triggers Required
I have required multiple triggers of the keyword phase to fire off the email and the remediation script, I have chosen four (4) as the trigger but that can be changed in the script, here you can see it waiting for 4 triggers before sending the email and then running the remediation script:
Script : RestartProcessandEmail.ps1
This will restart the process and then confirm the service is back online before returning the "success" command back the previous script, I have built in a timeout section as well as many services take time to restart.
$serviceName = "ServiceCode" # Service Name of the service you wish to restart
$timeout = 180 # Timeout in seconds
$stopTime = Get-Date
$stopTime = $stopTime.AddSeconds($timeout)
# Email settings
$smtpServer = "smtp.bear.local"
$smtpFrom = "java.process@croucher.cloud"
$smtpTo = "lee@croucher.cloud"
$subject = "$serviceName Service Restarted"
$htmlFilePath = "restartedservice.html"
# Stop the service
Write-Output "Attempting to stop the service $serviceName..."
Stop-Service -Name $serviceName -Force -ErrorAction Stop
# Wait for the service to stop
do {
Start-Sleep -Seconds 2
$service = Get-Service -Name $serviceName
if ($service.Status -eq 'Stopped') {
Write-Output "$serviceName service has stopped."
break
}
Write-Output "Waiting for $serviceName to stop..."
} while (($service.Status -ne 'Stopped') -and (Get-Date -lt $stopTime))
if ($service.Status -ne 'Stopped') {
Write-Output "Timeout: $serviceName service did not stop within $timeout seconds."
exit 1
}
# Restart the service
Write-Output "Starting the service $serviceName..."
Start-Service -Name $serviceName
# Confirm the service is running
$service = Get-Service -Name $serviceName
if ($service.Status -eq 'Running') {
Write-Output "$serviceName service is now running."
# Send confirmation email
if (Test-Path $htmlFilePath) {
$htmlBody = Get-Content -Path $htmlFilePath -Raw
Send-MailMessage -From $smtpFrom -To $smtpTo -Subject $subject `
-BodyAsHtml -Body $htmlBody -SmtpServer $smtpServer
Write-Output "Confirmation email sent to $smtpTo."
} else {
Write-Output "Error: HTML file not found at $htmlFilePath."
}
} else {
Write-Output "Failed to start the $serviceName service."
exit 1
}
This will also send another email this will confirm the script has restarted the service and the service is now back online, then once the e-mail is sent the script will default back to monitoring mode again.
The service name you give this script is the "Service Name" not the "Display Name" of the service, a good example of this is the Windows Update service as you can see below these names differ massively.
$serviceName = "ServiceCode" #Service Name of the service you wish to restart
Ensure that the $serviceName value you give in the script (shown above) is the Service Name value from the correct value.
What if the service fails to restart in the timeout period?
Well if the service does not restart in the timeout period then increase the value, this is that value:
$timeout = 180 # Timeout in seconds
If you increase this to more than 5 minutes and the service is not stopping then you either have something wrong with the service stop process or a hanging process.
Why is my service is not stopping!
However some services will get into "Stopping" and never recover depending on what they do and this will call for a process termination or killing the process "tree" and that can be accommodated with the script below.
Powershell : Restart-Timeout-ForceRestart,ps1
$serviceName = "TAGETIK_TDL"
$processName = "TAGETIK_TDL-wrapper"
$timeout = 180 # Timeout in seconds
$stopTime = Get-Date
$stopTime = $stopTime.AddSeconds($timeout)
$htmlFilePath = "forcedrestart.html" # Update with the correct path to your HTML file
$smtpServer = "smtp.bear.local" # Update with your SMTP server
$smtpFrom = "java.process@croucher.cloud"
$smtpTo = "lee@croucher.cloud"
$subject = "$serviceName Service Restarted"
# Step 1: Attempt to stop the service
Write-Output "Attempting to stop the service $serviceName..."
Stop-Service -Name $serviceName -Force -ErrorAction Stop
# Step 2: Wait for the service to stop within the timeout
do {
Start-Sleep -Seconds 2
$service = Get-Service -Name $serviceName
if ($service.Status -eq 'Stopped') {
Write-Output "$serviceName service has stopped."
break
}
Write-Output "Waiting for $serviceName to stop..."
} while (($service.Status -ne 'Stopped') -and (Get-Date -lt $stopTime))
# Step 3: Check if the service did not stop within the timeout
if ($service.Status -ne 'Stopped') {
Write-Output "Timeout: $serviceName service did not stop within $timeout seconds."
# Step 4: Find and kill the associated process
$processes = Get-Process -Name $processName -ErrorAction SilentlyContinue
if ($processes) {
Write-Output "Forcing the termination of the process $processName..."
Stop-Process -Name $processName -Force
Write-Output "$processName process has been terminated."
} else {
Write-Output "No process named $processName found."
}
# Step 5: Exit with a failure code since the service didn't stop gracefully
exit 1
}
# Step 6: Restart the service
Write-Output "Starting the service $serviceName..."
Start-Service -Name $serviceName
# Step 7: Confirm the service is running
$service = Get-Service -Name $serviceName
if ($service.Status -eq 'Running') {
Write-Output "$serviceName service is now running."
# Step 8: Send the HTML file via email
if (Test-Path $htmlFilePath) {
$htmlContent = Get-Content -Path $htmlFilePath -Raw
Write-Output "Sending restart notification email..."
Send-MailMessage -To $toAddress -From $fromAddress -Subject $subject -Body $htmlContent -BodyAsHtml -SmtpServer $smtpServer
Write-Output "Restart notification email sent."
} else {
Write-Output "HTML file not found at $htmlFilePath. Email not sent."
}
} else {
Write-Output "Failed to start the $serviceName service."
exit 1
}
Then with the new HTML template in the script you can then send a more information email, keeping people in the loop of the status and outcome.
