ULS Log Analyser with PowerShell

Infrastructure contacted me to complain that one of our SharePoint environments was logging too much data (via the ULS) and it was becoming unmanageable (an Operations Management tool like SCOM has not been configured). Looking through many gigabytes of text, even with a free tool like ULSViewer, it is difficult to be confident that you are correctly identifying the most common issues, it is an inaccurate art at best.

That is why I wrote a log analyser as a PowerShell script which will process ULS log files and, using fuzzy comparison, create a report of the most frequently occurring log entries.

I am very well aware that this is not necessarily useful information in many cases (hence I had to write this script myself). Nevertheless I found it useful in my scenario and I hope that some of you may as well.

Just in case you are interested: using this script I was able to declare with certainty that logging by the SPMonitoredScope class made up almost 30% of total log entries. This will be reduced by explicitly stating the log severity in the class constructor as verbose and maintaining a log level of Medium for the SharePoint Foundation : Monitoring diagnostic logging category.

A few things of note:

You may want to add to or remove from the set of replace statements in order to increase/decrease the ‘fuzziness’ of the comparison. Adding a replace statement for removing URLs may be a good candidate if you wish to increase matches.
The script loads entire files into memory at once. Be aware of this if you have very large log files or not much RAM.
The output file is a CSV, open it with Excel.
By default the script will identify and analyse all *.log files in the current directory.
If you cancel the script during processing (ctrl+c) it will still write all processed data up until the point at which it was cancelled.

param ( [string]$inputFile = ".\*.log", # The log file to process [string]$outputFile = ".\LogAnalyser_$(Get-Date -Format yyyyMMdd-HHmmss).csv", [bool]$confirm = $true ) # support relative paths if(![string]::IsNullOrEmpty($inputFile) -and $inputFile.StartsWith(".")) { $currentPath = Split-Path -parent $MyInvocation.MyCommand.Definition $inputFile = $currentPath + $inputFile.TrimStart(".") } if(![string]::IsNullOrEmpty($outputFile) -and $outputFile.StartsWith(".")) { $currentPath = Split-Path -parent $MyInvocation.MyCommand.Definition $outputFile = $currentPath + $outputFile.TrimStart(".") } $rowsPerDotLoading = 10000 # for display purposes only $rowsPerDotWriting = 100 # for display purposes only $rowCount = 0 $distintRowCount = 0; $hash = @{} $prevLine = "" $files = (Get-ChildItem $inputFile) if($files -eq $null -or $files.Length -lt 1) { Write-Host "No input files found!" return } $fileTotalCount = $files.Length if($confirm) { Write-Host "Files: " $files | Foreach { Write-Host $_.Name } Write-Host ("Press 'y' to process $fileTotalCount file(s)") $key = $host.UI.RawUI.ReadKey("IncludeKeyDown") if($key.Character -ne "y") { Write-Host "" return } } $startTime = Get-Date $fileCount = 0 try { foreach($file in $files) { $fileCount++ Write-Host "" Write-Host "Reading $fileCount of $fileTotalCount, $($file.Name) ..." -NoNewline $fileContent = Get-Content $file Write-Host " done" Write-Host "Processing " -NoNewline foreach ($_ in $fileContent) { # split line $lineArray = $_.Split(([char]9)) #$timestamp = $lineArray[0] $process = $lineArray[1] -replace "$.+$", "" #$tif = $lineArray[2] $area = $lineArray[3] $category = $lineArray[4] #$eventId = $lineArray[5] $level = $lineArray[6] $message = $lineArray[7] #$correlationId = $lineArray[8] $hashKey = "$process,$area,$category,$level" if(!$hash.ContainsKey($hashKey)) { $hash[$hashKey] = @{} } $msgsHash = $hash[$hashKey] # handle multi-line log messages if($message.EndsWith("...")) { $prevLine = $prevLine + $message.TrimEnd('.') } else { if(![string]::IsNullOrEmpty($prevLine)) { $message = $prevLine + $message.TrimStart('.') $prevLine = "" } # perform fuzzing on the message $msgHashKey = $message $msgHashKey = $msgHashKey -replace "(\{{0,1}([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}\}{0,1})", "GUID" # remove guids $msgHashKey = $msgHashKey -replace "$.+$", "(XXX)" # remove content of parentheses $msgHashKey = $msgHashKey -replace "\d*\.?\d+", "X" # remove numbers $msgHashKey = $msgHashKey -replace "\s+", " " # condense whitespace if(!$msgsHash.ContainsKey($msgHashKey)) { $msgsHash[$msgHashKey] = 0 $distintRowCount++ } $msgsHash[$msgHashKey]++ $rowCount++ if($rowCount%$rowsPerDotLoading -eq 0) { Write-Host "." -NoNewline } } } Write-Host " done" $fileContent = $null } Write-Host " done" -ForegroundColor Green } finally { Write-Host "" Write-Host "Writing $outputFile " -NoNewline Add-Content $outputfile "count,process,area,category,level,message" foreach($hashKey in $hash.Keys) { $msgHash = $hash[$hashKey] foreach($_ in $msgHash.Keys) { $count = $msgHash[$_] $message = '"' + ($_ -replace '"', '""') + '"' # csv escaping Add-Content $outputfile "$count,$hashKey,$message" if($distintRowCount%$rowsPerDotWriting -eq 0) { Write-Host "." -NoNewline } } } $hash = $null Write-Host " done" -ForegroundColor Green Write-Host "Distinct log rows: $distintRowCount of $rowCount" -ForegroundColor Black -BackgroundColor White $timeTaken = ((Get-Date) - $startTime) $timeSpanString = '{0:00}:{1:00}:{2:00}' -f ($timeTaken | % {$_.Hours, $_.Minutes, $_.Seconds}) Write-Host "Completed in $timeSpanString" -ForegroundColor Black -BackgroundColor White }

I quite enjoy PowerShell-ing so expect to see more utilities in the future.

ULS Log Analyser

Published by

Paul Ryan

Leave a Reply Cancel reply