Enforcing a Sysmon Archive Quota

Sysmon (System Monitor) is a well-known and widely used Windows logging utility providing valuable visibility into core OS (operating system) events. From a defender’s perspective, the presence of Sysmon in an environment greatly enhances detection and forensic capabilities by logging events involving processes, files, registry, network connections and more.

Since Sysmon 11 (released April 2020), the FileDelete event provides the capability to retain (archive) deleted files, a feature we especially adore during active compromises when actors drop-use-delete tools. However, as duly noted in Sysmon’s documentation, the usage of the archiving feature might grow the archive directory to unreasonable sizes (hundreds of GB); something most environments cannot afford.

This blog post will cover how, through a Windows-native feature (WMI event consumption), the Sysmon archive can be kept at a reasonable size. In a hurry? Go straight to the proof of concept!

Figure 1: A Sysmon archive quota removing old files.

The Challenge of Sysmon File Archiving

Typical Sysmon deployments require repeated fine-tuning to ensure optimized performance. When responding to hands-on-keyboard attackers, this time-consuming process is commonly replaced by relying on robust base-lined configurations (some of which open-source such as SwiftOnSecurity/sysmon-config or olafhartong/sysmon-modular). While most misconfigured events have at worst an impact on CPU and log storage, the Sysmon file archiving can grind a system to a halt by exhausting all available storage. So how could one still perform file archiving without risking an outage?

While searching for a solution, we defined some acceptance requirements. Ideally, the solution should…

Be Windows-native. We weren’t looking for yet another agent/driver which consumes resources, may cause compatibility issues and increase the attack surface.
Be FIFO-like (First In, First Out) to ensure the oldest archived files are deleted first. This ensures attacker tools are kept in the archive just long enough for our incident responders to grab them.
Have a minimal system performance impact if we want file archiving to be usable in production.

A common proposed solution would be to rely on a scheduled task to perform some clean-up activities. While being Windows-native, this execution method is “dumb” (schedule-based) and would execute even without files being archived.

So how about WMI event consumption?

WMI Event Consumption

WMI (Windows Management Instrumentation) is a Windows-native component providing capabilities surrounding the OS’ management data and operations. You can for example use it to read and write configuration settings related to Windows, or monitor operations such as process and file creations.

Within the WMI architecture lays the permanent event consumer.

You may want to write an application that can react to events at any time. For example, an administrator may want to receive an email message when specific performance measures decline on network servers. In this case, your application should run at all times. However, running an application continuously is not an efficient use of system resources. Instead, WMI allows you to create a permanent event consumer. […]
A permanent event consumer receives events until its registration is explicitly canceled.
Source: Microsoft

Leveraging a permanent event consumer to monitor for file events within the Sysmon archive folder would provide optimized event-based execution as opposed to the scheduled task approach.

In the following sections we will start by creating a WMI event filter intended to select events of interest; after which we will cover the WMI logical consumer whose role will be to clean up the Sysmon archive.

WMI Event Filter

A WMI event filter is an __EventFilter instance containing a WQL (WMI Query Language, SQL for WMI) statement whose role is to filter event tables for the desired events. In our case, we want to be notified when files are being created in the Sysmon archive folder.

Whenever files are created, a CIM_DataFile intrinsic event is fired within the __InstanceCreationEvent class. The following WQL statement would filter for such events within the default C:\Sysmon\ archive folder:

SELECT * FROM __InstanceCreationEvent
WHERE TargetInstance ISA 'CIM_DataFile'
    AND TargetInstance.Drive='C:'
    AND TargetInstance.Path='\\Sysmon\\'

Intrinsic events are polled at specific intervals. As we wish to ensure the polling period is not too long, a WITHIN clause can be used to define the maximum amount of seconds that can pass before the notification of the event must be delivered.

The beneath query requires matching event notifications to be delivered within 10 seconds.

SELECT * FROM __InstanceCreationEvent
WITHIN 10
WHERE TargetInstance ISA 'CIM_DataFile'
    AND TargetInstance.Drive='C:'
    AND TargetInstance.Path='\\Sysmon\\'

While the above WQL statement is functional, it is not yet optimized. As an example, if Sysmon came to archive 1000 files, the event notification would fire 1000 times, later resulting in our clean-up logic to be executed 1000 times as well.

To cope with this property, a GROUP clause can be used to combine events into a single notification. Furthermore, to ensure the grouping occurs within timely manner, another WITHIN clause can be leveraged. The following WQL statement waits for up to 10 seconds to deliver a single notification should any files have been created in Sysmon’s archive folder.

SELECT * FROM __InstanceCreationEvent
WITHIN 10
WHERE TargetInstance ISA 'CIM_DataFile'
    AND TargetInstance.Drive='C:'
    AND TargetInstance.Path='\\Sysmon\\'
GROUP WITHIN 10

To create a WMI event filter we can rely on PowerShell’s New-CimInstance cmdlet as shown in the following snippet.

$Archive = "C:\\Sysmon\\"
$Delay = 10
$Filter = New-CimInstance -Namespace root/subscription -ClassName __EventFilter -Property @{
    Name = 'SysmonArchiveWatcher';
    EventNameSpace = 'root\cimv2';
    QueryLanguage = "WQL";
    Query = "SELECT * FROM __InstanceCreationEvent WITHIN $Delay WHERE TargetInstance ISA 'CIM_DataFile' AND TargetInstance.Drive='$(Split-Path -Path $Archive -Qualifier)' AND TargetInstance.Path='$(Split-Path -Path $Archive -NoQualifier)' GROUP WITHIN $Delay"
}

WMI Logical Consumer

The WMI logical consumer will consume WMI events and undertake actions for each occurrence. Multiple logical consumer classes exist providing different behaviors whenever events are received, such as:

ActiveScriptEventConsumer which executes a predefined VBS script.
LogFileEventConsumer which writes a customized message to a text log file.
NTEventLogEventConsumer which logs a specific message in the Windows event log.
SMTPEventConsumer which sends an email using SMTP.
CommandLineEventConsumer which launches an arbitrary process.

The last CommandLineEventConsumer class is particularly interesting as it would allow us to run a PowerShell script whenever files are archived by Sysmon (a feature attackers do enjoy as well).

The first step on our PowerShell code would be to obtain a full list of archived files ordered from oldest to most recent. This list will play two roles:

It will be used to compute the current directory size.
It will be used as a list of files to remove (in FIFO order) until the directory size is back under control.

While getting a list of files is easy through the Get-ChildItem cmdlet, sorting these files from oldest to most recently archived requires some thinking. Where common folders could rely on the file’s CreationTimeUtc property, Sysmon archiving copies this file property over. As a consequence the CreationTimeUtc field is not representative of when a file was archived and relying on it could result in files being incorrectly seen as the oldest archives, causing their premature removal.

Instead of relying on CreationTimeUtc, the alternate LastAccessTimeUtc property provides a more accurate representation of when a file was archived. The following snippet will get all files within the Sysmon archive and order them in a FIFO-like fashion.

$Archived = Get-ChildItem -Path 'C:\\Sysmon\\' -File | Sort-Object -Property LastAccessTimeUtc

Once the archived files listed, the folder size can be computed through the Measure-Object cmdlet.

$Size = ($Archived | Measure-Object -Sum -Property Length).Sum

All that remains to do is then loop the archived files and remove them while the folder exceeds our desired quota.

for($Index = 0; ($Index -lt $Archived.Count) -and ($Size -gt 5GB); $Index++)
{
    $Archived[$Index] | Remove-Item -Force
    $Size -= $Archived[$Index].Length
}

Sysmon & Hard Links

In some situations, Sysmon archives a file by referencing the file’s content from a new path, a process known as hard-linking.

A hard link is the file system representation of a file by which more than one path references a single file in the same volume.
Source: Microsoft

As an example, the following snippet creates an additional path (hard link) for an executable. Both paths will now point to the same on-disk file content. If one path gets deleted, Sysmon will reference the deleted file by adding a path, resulting in the file’s content having two paths, one of which within the Sysmon archive.

:: Create a hard link for an executable.
C:\>mklink /H C:\Users\Public\NVISO.exe C:\Users\NVISO\Downloads\NVISO.exe
Hardlink created for C:\Users\Public\NVISO.exe <<===>> C:\Users\NVISO\Downloads\NVISO.exe
 
:: Delete one of the hard links causing Sysmon to archive the file.
C:\>del C:\Users\NVISO\Downloads\NVISO.exe
 
:: The archived file now has two paths, one of which within the Sysmon archive.
C:\>fsutil hardlink list Sysmon\B99D61D874728EDC0918CA0EB10EAB93D381E7367E377406E65963366C874450.exe
\Sysmon\B99D61D874728EDC0918CA0EB10EAB93D381E7367E377406E65963366C874450.exe
\Users\Public\NVISO.exe

The presence of hard links within the Sysmon archive can cause an edge-case should the non-archive path be locked by another process while we attempt to clean the archive. Should for example a process be created from the non-archive path, removing the archived file will become slightly harder.

:: If the other path is locked by a process, deleting it will result in a denied access.
C:\>del Sysmon\B99D61D874728EDC0918CA0EB10EAB93D381E7367E377406E65963366C874450.exe
C:\Sysmon\B99D61D874728EDC0918CA0EB10EAB93D381E7367E377406E65963366C874450.exe
Access is denied.

Removing hard links is not straight-forward and commonly relies on non-native software such as fsutil (itself requiring the Windows Subsystem for Linux). However, as the archive’s hard link does technically not consume additional storage (the same content is referenced from another path), such files could be ignored given they do not partake in the storage exhaustion. Once the non-archive hard links referencing a Sysmon-archived file are removed, the archived file is not considered a hard link anymore and will be removable again.

To cope with the above edge-case, hard links can be filtered-out and removal operations can be encapsulated in try/catch expressions should other edge-cases exists. Overall, the WMI logical consumer’s logic could look as follow:

$Archived = Get-ChildItem -Path 'C:\\Sysmon\\' -File | Where-Object {$_.LinkType -ne 'HardLink'} | Sort-Object -Property LastAccessTimeUtc
$Size = ($Archived | Measure-Object -Sum -Property Length).Sum
for($Index = 0; ($Index -lt $Archived.Count) -and ($Size -gt 5GB); $Index++)
{
    try
    {
        $Archived[$Index] | Remove-Item -Force -ErrorAction Stop
        $Size -= $Archived[$Index].Length
    } catch {}
}

As we did for the event filter, a WMI consumer can be created through the New-CimInstance cmdlet. The following snippet specifically creates a new CommandLineEventConsumer invoking our above clean-up logic to create a 10GB quota.

$Archive = "C:\\Sysmon\\"
$Limit = 10GB
$Consumer = New-CimInstance -Namespace root/subscription -ClassName CommandLineEventConsumer -Property @{
    Name = 'SysmonArchiveCleaner';
    ExecutablePath = $((Get-Command PowerShell).Source);
    CommandLineTemplate = "-NoLogo -NoProfile -NonInteractive -WindowStyle Hidden -Command `"`$Archived = Get-ChildItem -Path '$Archive' -File | Where-Object {`$_.LinkType -ne 'HardLink'} | Sort-Object -Property LastAccessTimeUtc; `$Size = (`$Archived | Measure-Object -Sum -Property Length).Sum; for(`$Index = 0; (`$Index -lt `$Archived.Count) -and (`$Size -gt $Limit); `$Index++){ try {`$Archived[`$Index] | Remove-Item -Force -ErrorAction Stop; `$Size -= `$Archived[`$Index].Length} catch {}}`""
}

WMI Binding

In the above two sections we defined the event filter and logical consumer. One last point worth noting is that event filters need to be bound to an event consumers in order to become operational. This is done through a __FilterToConsumerBinding instance as shown below.

New-CimInstance -Namespace root/subscription -ClassName __FilterToConsumerBinding -Property @{
    Filter = [Ref]$Filter;
    Consumer = [Ref]$Consumer;
}

Proof of Concept

The following proof-of-concept deployment technique has been tested in limited environments. As should be the case with anything you introduce into your environment, make sure rigorous testing is done and don’t just deploy straight to production.

The following PowerShell script creates a WMI event filter and logical consumer with the logic we defined previously before binding them. The script can be configured using the following variables:

$Archive as the Sysmon archive path. To be WQL-compliant, special characters have to be back-slash (\) escaped, resulting in double back-slashed directory separators (\\).
$Limit as the Sysmon archive’s desired maximum folder size (see real literals).
$Delay as the event filter’s maximum WQL delay value in seconds (WITHIN clause).

Do note that Windows security boundaries apply to WMI as well and, given the Sysmon archive directory is restricted to the SYSTEM user, the following script should be ran using the SYSTEM privileges.

$ErrorActionPreference = "Stop"
 
# Define the Sysmon archive path, desired quota and query delay.
$Archive = "C:\\Sysmon\\"
$Limit = 10GB
$Delay = 10
 
# Create a WMI filter for files being created within the Sysmon archive.
$Filter = New-CimInstance -Namespace root/subscription -ClassName __EventFilter -Property @{
    Name = 'SysmonArchiveWatcher';
    EventNameSpace = 'root\cimv2';
    QueryLanguage = "WQL";
    Query = "SELECT * FROM __InstanceCreationEvent WITHIN $Delay WHERE TargetInstance ISA 'CIM_DataFile' AND TargetInstance.Drive='$(Split-Path -Path $Archive -Qualifier)' AND TargetInstance.Path='$(Split-Path -Path $Archive -NoQualifier)' GROUP WITHIN $Delay"
}
 
# Create a WMI consumer which will clean up the Sysmon archive folder until the quota is reached.
$Consumer = New-CimInstance -Namespace root/subscription -ClassName CommandLineEventConsumer -Property @{
    Name = 'SysmonArchiveCleaner';
    ExecutablePath = (Get-Command PowerShell).Source;
    CommandLineTemplate = "-NoLogo -NoProfile -NonInteractive -WindowStyle Hidden -Command `"`$Archived = Get-ChildItem -Path '$Archive' -File | Where-Object {`$_.LinkType -ne 'HardLink'} | Sort-Object -Property LastAccessTimeUtc; `$Size = (`$Archived | Measure-Object -Sum -Property Length).Sum; for(`$Index = 0; (`$Index -lt `$Archived.Count) -and (`$Size -gt $Limit); `$Index++){ try {`$Archived[`$Index] | Remove-Item -Force -ErrorAction Stop; `$Size -= `$Archived[`$Index].Length} catch {}}`""
}
 
# Create a WMI binding from the filter to the consumer.
New-CimInstance -Namespace root/subscription -ClassName __FilterToConsumerBinding -Property @{
    Filter = [Ref]$Filter;
    Consumer = [Ref]$Consumer;
}

Once the WMI event consumption configured, the Sysmon archive folder will be kept at reasonable size as shown in the following capture where a 90KB quota has been defined.

Figure 2: A Sysmon archive quota of 90KB removing old files.

With Sysmon archiving under control, we can now happily wait for new attacker tool-kits to be dropped…