When a server backup program utilizes Volume Shadow Copy Service (VSS), sometimes you have to fix problems when one or more of the writers fails or disappears. This can usually be repaired by running a script that re-registers the various DLLs used by VSS. But we ran into a problem on one server that defied this traditional fix. Even though the script was executed and the server rebooted several times, the System Writer would disappear every time the backup was attempted. And it always failed in the same place — backing up the System State.
Since the tried and true fix of re-registering the DLLs wasn't working, we needed to look elsewhere. And the problem turned out to be something we hadn't seen before. Too many files in the Temporary ASP.NET Files directories.
The System State
According Microsoft TechNet, the System State is comprised of the following:
- Boot files, including the system files, and all files protected by Windows File Protection (WFP).
- Active Directory (on a domain controller only).
- Sysvol (on a domain controller only).
- Certificate Services (on certification authority only).
- Cluster database (on a cluster node only).
- The registry.
- Performance counter configuration information.
- Component Services Class registration database.
System files includes everything under the C:\Windows folder whether or not it's actually needed to restore the System State. (Since the majority of the files found in the C:\Windows folder are required for the operation of the system, it's much safer to simply back up everything rather than miss a critical file.) To create a System State backup, the VSS System Writer must first enumerate all the files and folders that make up the system files, and this where it ran afoul of the ASP.NET temporary files.
ASP.NET Temporary Files
One of the principal features of Microsoft.NET is that applications can be run on multiple operating systems without having to rebuild the program. (The open source Mono Project uses this very feature to bring the .NET Framework to Linux, Apple's OS X, and many other non-Microsoft platforms.) They are initially compiled into a machine-independent Intermediate Language (IL), and it's in this form that they are installed on the target system. Then when the program is executed, the program's IL code is compiled into machine code by an operating system-specific compiler.
Having to compile the IL code is an expensive operation, so to speed up subsequent executions of the program, .NET saves the compiled machine code in a temporary directory. When the program is run again, .NET checks to see if there is a cached copy of the compiled machine code. If there is, it skips compiling the IL code and runs the cached machine code instead. For ASP.NET programs, the compiled machine code is cached in a folder under C:\Windows\Microsoft.NET.
ASP.NET is different from other types of .NET programs in that each ASP.NET page is considered to be a separate program. This means that lots of compiled machine code gets cached. On a server that hosts hundreds of ASP.NET websites, there can be thousands, or even tens of thousands, of cached machine code files.
The System Writer Disappears
Because the default location of the ASP.NET temporary directories is under C:\Windows\Microsoft.NET, these files are considered to be system files, and therefore part of the System State. So when VSS is used to create a backup of the System State, the System Writer enumerates all the system files, including the ASP.NET temporary files. But there is a limit to the number of files that the System Writer can deal with. When this limit is exceeded, the System Writer aborts with an error, causing the System State backup to fail.
Once the System Writer has aborted, it disappears from the list of VSS writers until the service that controls it — which is the Cryptographic Services — is restarted. But even when the Cryptographic Services is restarted, the System Writer will simply abort again the next time it tries to enumerate all the ASP.NET temporary files.
Relocating the ASP.NET Temporary Directories
None of the ASP.NET temporary files are required to restore the System State from a backup. If they are deleted, they are recreated by compiling the IL code the next time the ASP.NET application is executed. But by virtue of their location under the C:\Windows\Microsoft.NET folder, they are considered to be part of the System State. So to get the ASP.NET temporary files out of the System State, we need to move these temporary directories to a different location.
Doing this requires that you perform a number of steps, which I describe below. The names and locations for the new folders are just my personal preference. Change them as desired to meet the needs of your system so long as they are not located under the C:\Windows folder.
Create New ASP.NET Temporary Directories
Despite the various versions of .NET Framework, only three versions of .NET actually have ASP.NET temporary directories. These are .NET Framework 1.1 (which is not included with Windows Server 2008 and higher and can largely be ignored), .NET Framework 2.0 (which includes .NET 3.0 and 3.5, as these are just extensions for .NET 2.0), and .NET Framework 4.0. (.NET Framework 4.5 has just been released. When installed, it replaces .NET Framework 4.0 if it's already installed. I didn't include it in the example here, but relocating its ASP.NET temporary directories will be similar to .NET 4.0.) And except for .NET Framework 1.1 (which is 32-bit only), each of these versions has a 32-bit and 64-bit temporary directory.
The first thing we need to do is to create new directories for the ASP.NET temporary files. These commands will create new directories for the ASP.NET 2.0 and 4.0 temporary files on the D drive.
md "D:\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files" md "D:\Microsoft.NET\Framework\v4.0.30319\Temporary ASP.NET Files" md "D:\Microsoft.NET\Framework64\v2.0.50727\Temporary ASP.NET Files" md "D:\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files"
Set Folder Permissions
Next we need to set the folder permissions to match the existing default directories. The easiest way to do this is with the ICACLS command. The first command removes the removes the inherited permissions and replaces them with a copy of the inherited permissions. Thus changes to the permissions of the drive (root directory) will not effect the new temporary directories. The remaining commands grant the required permissions.
icacls "D:\Microsoft.NET" /inheritance:d icacls "D:\Microsoft.NET" /grant:r "BUILTIN\Administrators:(OI)(CI)(F)" icacls "D:\Microsoft.NET" /grant:r "NT AUTHORITY\SYSTEM:(OI)(CI)(F)" icacls "D:\Microsoft.NET" /grant:r "CREATOR OWNER:(OI)(CI)(IO)(F)" icacls "D:\Microsoft.NET" /grant:r "BUILTIN\IIS_IUSRS:(OI)(CI)(M,WDAC,DC)" icacls "D:\Microsoft.NET" /grant:r "BUILTIN\Users:(OI)(CI)(RX)" icacls "D:\Microsoft.NET" /grant:r "NT SERVICE\TrustedInstaller:(CI)(F)" icacls "D:\Microsoft.NET" /grant:r "NT SERVICE\WMSvc:(OI)(CI)(M,DC)"
Add Attribute tempDirectory To The compilation Tag In web.config
For ASP.NET to use temporary directories anywhere other than the default location, the directory must be specified using the tempDirectory attribute of the <compilation> tag in the system web.config file. There is one file for each version of the .NET Framework. (Again, these are the same versions that have ASP.NET temporary directories, so there is no web.config file for .NET 3.0 and 3.5.) The tempDirectory attribute specifies the directory where the compiled machine code will be cached. The web.config file is a XML file that can be edited with Notepad.
For ASP.NET 2.0 32-bit, we would edit the web.config file and locate this tag:
and change it as follows to use the new temporary directory:
<compilation tempDirectory="D:\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files">
The web.config file is located in the CONFIG folder for the .NET Framework version. In our example, we will need to edit the following web.config files.
.NET Framework 2.0 – 32-Bit
.NET Framework 2.0 – 64-Bit
.NET Framework 4.0 – 32-Bit
.NET Framework 4.0 – 64-Bit
For the changes made to the web.config files to take effect, IIS has to be restarted. This is easily done from the command line.
Delete Files In The Old Temporary Directories
Now we need to delete the files in the old ASP.NET temporary directories so they are no longer part of the system state. These files are actually in a subfolder named root, so we'll actually delete this folder along with all it's files and subfolders. Again, this is easily done from the command line.
rmdir /s /q "C:\Windows\Microsoft.Net\Framework\v2.0.50727\Temporary ASP.NET Files\root" rmdir /s /q "C:\Windows\Microsoft.Net\Framework64\v2.0.50727\Temporary ASP.NET Files\root" rmdir /s /q "C:\Windows\Microsoft.Net\Framework\v4.0.30319\Temporary ASP.NET Files\root" rmdir /s /q "C:\Windows\Microsoft.Net\Framework64\v4.0.30319\Temporary ASP.NET Files\root"
Restart The Cryptographic Service
To get the VSS System Writer back, we must restart the service that controls it, which as previously mentioned, was the Cryptographic Service.
net stop cryptsvc net start cryptsvc
Verifying Everything Is Working
If you did everything correctly, you should see files created in the new ASP.NET temporary directories the next time the website is accessed. And to verify the System Writer has returned, run the vssadmin command to list the writers.
vssadmin list writers
Now that the ASP.NET temporary files are no longer considered to be system files, the System State backup should complete without errors.