Category Archives: Windows

Sizing The Windows Page File

A common question is how big should the Windows page file be?  In my opinion, as small as possible.

The Windows page file is analogous to the Linux swap file.  It's a file on the hard disk that serves as virtual memory.  If all running the applications won't fit inside physical RAM, Windows swaps out lesser-used segments of memory to the page file so it can make room for other applications.  But it's a relic from the days when computers had no where near the amount of RAM they have to today.

In practice, you never want to use the page file.  Why?  Because paging operations (swapping memory segments between physical RAM and the page file) is expensive.  Very expensive!  Disk I/O is orders of magnitude slower than physical RAM, and when Windows is doing a lot of paging operations, system performance takes a big hit.  It can slow the system to the point that the screen cursor needs minutes to respond when the mouse is moved.  And then the disk activity light and the power light are indistinguishable from each other — both shining with a steady glow.  Usually the only way to recover when this happens is a hard reboot of the system.

The problem is that the default for automatically allocating the page file hasn't changed since Windows NT Server 3.5.  It creates a page file that is 1.5 times the size of physical RAM.  A good value when servers had 512 MB of RAM, but extremely wasteful on a server with 16 GB of RAM.  That's a 24 GB page file!  And did I mention that the default location for the page file is the C drive?  With everything else that is vying for space on the C drive, the last thing you need is a gigantic file you never want to use.

You can have no page file, but this isn't a good idea.  There are a few rarely used things that Windows wants to keep in the page file, but these are never that big.  A good value for the page file is between 2 – 4 GB.  Resist the temptation to make it any bigger than 4 GB, regardless of what "best practices" say.  Remember, you never want to use the page file.  And you only ever need one page file.  Having multiple page files is even more wasteful than having a extremely large page file.  Don't do it!

Use a custom size for the page file and set the initial and maximum size to the same value.  If these values are not the same, then the page file will become fragmented as Windows shrinks and expands the page file.  You want to keep disk I/O operations for the page file to an absolute minimum, so let Windows create the page file once and keep it from constantly resizing it.

Posted in Windows.

Disappearing VSS System Writer and ASP.NET

When a server backup program utilizes Volume Shadow Copy Service (VSS), sometimes you have to fix problems when one or more of the writers fails or disappears.  This can usually be repaired by running a script that re-registers the various DLLs used by VSS.  But we ran into a problem on one server that defied this traditional fix.  Even though the script was executed and the server rebooted several times, the System Writer would disappear every time the backup was attempted.  And it always failed in the same place — backing up the System State.

Since the tried and true fix of re-registering the DLLs wasn't working, we needed to look elsewhere.  And the problem turned out to be something we hadn't seen before.  Too many files in the Temporary ASP.NET Files directories.

The System State

According Microsoft TechNet, the System State is comprised of the following:

  • Boot files, including the system files, and all files protected by Windows File Protection (WFP).
  • Active Directory (on a domain controller only).
  • Sysvol (on a domain controller only).
  • Certificate Services (on certification authority only).
  • Cluster database (on a cluster node only).
  • The registry.
  • Performance counter configuration information.
  • Component Services Class registration database.

System files includes everything under the C:\Windows folder whether or not it's actually needed to restore the System State.  (Since the majority of the files found in the C:\Windows folder are required for the operation of the system, it's much safer to simply back up everything rather than miss a critical file.)  To create a System State backup, the VSS System Writer must first enumerate all the files and folders that make up the system files, and this where it ran afoul of the ASP.NET temporary files.

ASP.NET Temporary Files

One of the principal features of Microsoft.NET is that applications can be run on multiple operating systems without having to rebuild the program.  (The open source Mono Project uses this very feature to bring the .NET Framework to Linux, Apple's OS X, and many other non-Microsoft platforms.)  They are initially compiled into a machine-independent Intermediate Language (IL), and it's in this form that they are installed on the target system.  Then when the program is executed, the program's IL code is compiled into machine code by an operating system-specific compiler.

Having to compile the IL code is an expensive operation, so to speed up subsequent executions of the program, .NET saves the compiled machine code in a temporary directory.  When the program is run again, .NET checks to see if there is a cached copy of the compiled machine code.  If there is, it skips compiling the IL code and runs the cached machine code instead.  For ASP.NET programs, the compiled machine code is cached in a folder under C:\Windows\Microsoft.NET.

ASP.NET is different from other types of .NET programs in that each ASP.NET page is considered to be a separate program.  This means that lots of compiled machine code gets cached.  On a server that hosts hundreds of ASP.NET websites, there can be thousands, or even tens of thousands, of cached machine code files.

The System Writer Disappears

Because the default location of the ASP.NET temporary directories is under C:\Windows\Microsoft.NET, these files are considered to be system files, and therefore part of the System State.  So when VSS is used to create a backup of the System State, the System Writer enumerates all the system files, including the ASP.NET temporary files.  But there is a limit to the number of files that the System Writer can deal with.  When this limit is exceeded, the System Writer aborts with an error, causing the System State backup to fail.

Once the System Writer has aborted, it disappears from the list of VSS writers until the service that controls it — which is the Cryptographic Services — is restarted.  But even when the Cryptographic Services is restarted, the System Writer will simply abort again the next time it tries to enumerate all the ASP.NET temporary files.

Relocating the ASP.NET Temporary Directories

None of the ASP.NET temporary files are required to restore the System State from a backup.  If they are deleted, they are recreated by compiling the IL code the next time the ASP.NET application is executed.  But by virtue of their location under the C:\Windows\Microsoft.NET folder, they are considered to be part of the System State.  So to get the ASP.NET temporary files out of the System State, we need to move these temporary directories to a different location.

Doing this requires that you perform a number of steps, which I describe below.  The names and locations for the new folders are just my personal preference.  Change them as desired to meet the needs of your system so long as they are not located under the C:\Windows folder.

Create New ASP.NET Temporary Directories

Despite the various versions of .NET Framework, only three versions of .NET actually have ASP.NET temporary directories.  These are .NET Framework 1.1 (which is not included with Windows Server 2008 and higher and can largely be ignored), .NET Framework 2.0 (which includes .NET 3.0 and 3.5, as these are just extensions for .NET 2.0), and .NET Framework 4.0.  (.NET Framework 4.5 has just been released.  When installed, it replaces .NET Framework 4.0 if it's already installed.  I didn't include it in the example here, but relocating its ASP.NET temporary directories will be similar to .NET 4.0.)  And except for .NET Framework 1.1 (which is 32-bit only), each of these versions has a 32-bit and 64-bit temporary directory.

The first thing we need to do is to create new directories for the ASP.NET temporary files.  These commands will create new directories for the ASP.NET 2.0 and 4.0 temporary files on the D drive.

md "D:\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files"
md "D:\Microsoft.NET\Framework\v4.0.30319\Temporary ASP.NET Files"
md "D:\Microsoft.NET\Framework64\v2.0.50727\Temporary ASP.NET Files"
md "D:\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files"

Set Folder Permissions

Next we need to set the folder permissions to match the existing default directories.  The easiest way to do this is with the ICACLS command.  The first command removes the removes the inherited permissions and replaces them with a copy of the inherited permissions.  Thus changes to the permissions of the drive (root directory) will not effect the new temporary directories.  The remaining commands grant the required permissions.

icacls "D:\Microsoft.NET" /inheritance:d
icacls "D:\Microsoft.NET" /grant:r "BUILTIN\Administrators:(OI)(CI)(F)"
icacls "D:\Microsoft.NET" /grant:r "NT AUTHORITY\SYSTEM:(OI)(CI)(F)"
icacls "D:\Microsoft.NET" /grant:r "CREATOR OWNER:(OI)(CI)(IO)(F)"
icacls "D:\Microsoft.NET" /grant:r "BUILTIN\IIS_IUSRS:(OI)(CI)(M,WDAC,DC)"
icacls "D:\Microsoft.NET" /grant:r "BUILTIN\Users:(OI)(CI)(RX)"
icacls "D:\Microsoft.NET" /grant:r "NT SERVICE\TrustedInstaller:(CI)(F)"
icacls "D:\Microsoft.NET" /grant:r "NT SERVICE\WMSvc:(OI)(CI)(M,DC)"

Add Attribute tempDirectory To The compilation Tag In web.config

For ASP.NET to use temporary directories anywhere other than the default location, the directory must be specified using the tempDirectory attribute of the <compilation> tag in the system web.config file.  There is one file for each version of the .NET Framework.  (Again, these are the same versions that have ASP.NET temporary directories, so there is no web.config file for .NET 3.0 and 3.5.)  The tempDirectory attribute specifies the directory where the compiled machine code will be cached.  The web.config file is a XML file that can be edited with Notepad.

For ASP.NET 2.0 32-bit, we would edit the web.config file and locate this tag:


and change it as follows to use the new temporary directory:

<compilation tempDirectory="D:\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files">

The web.config file is located in the CONFIG folder for the .NET Framework version.  In our example, we will need to edit the following web.config files.

.NET Framework 2.0 – 32-Bit

.NET Framework 2.0 – 64-Bit

.NET Framework 4.0 – 32-Bit

.NET Framework 4.0 – 64-Bit

Restart IIS

For the changes made to the web.config files to take effect, IIS has to be restarted.  This is easily done from the command line.


Delete Files In The Old Temporary Directories

Now we need to delete the files in the old ASP.NET temporary directories so they are no longer part of the system state.  These files are actually in a subfolder named root, so we'll actually delete this folder along with all it's files and subfolders.  Again, this is easily done from the command line.

rmdir /s /q "C:\Windows\Microsoft.Net\Framework\v2.0.50727\Temporary ASP.NET Files\root"
rmdir /s /q "C:\Windows\Microsoft.Net\Framework64\v2.0.50727\Temporary ASP.NET Files\root"
rmdir /s /q "C:\Windows\Microsoft.Net\Framework\v4.0.30319\Temporary ASP.NET Files\root"
rmdir /s /q "C:\Windows\Microsoft.Net\Framework64\v4.0.30319\Temporary ASP.NET Files\root"

Restart The Cryptographic Service

To get the VSS System Writer back, we must restart the service that controls it, which as previously mentioned, was the Cryptographic Service.

net stop cryptsvc
net start cryptsvc

Verifying Everything Is Working

If you did everything correctly, you should see files created in the new ASP.NET temporary directories the next time the website is accessed.  And to verify the System Writer has returned, run the vssadmin command to list the writers.

vssadmin list writers

Now that the ASP.NET temporary files are no longer considered to be system files, the System State backup should complete without errors.

Posted in Windows.

Disabling Large Send Offload – Windows

In an earlier post, I described the Large Send Offload (LSO) feature of modern Ethernet adapters and why it can cause havoc with network performance.  And since this is enabled by default, you have to manually disable it.  For Windows, this can be done in the Ethernet adapter properties (which I prefer) or in the TCP/IP network stack.  I'll start with disabling LSO in the TCP/IP network stack since Microsoft uses some confusing terms that you'll want to be familiar with.

Micosoft refers to LSO as "TCP Chimney Offload".  For Windows Server 2008 and later, it's described in MS Support Article 951037.  LSO was first supported in Windows Server 2003 with the release of the Scalable Networking Pack, which integrated into Service Pack 2, and is described in MS Support Article 912222.

Disabling LSO on Windows Server 2003

This has to be done by editing the registry.  Open regedit and locate this key:


Then right-click on EnableTCPChimney and select Modify from the pop-up menu.  Change the value to 0 and click on OK.  You can also use this REG command (this command is one line but shown here here on two lines for clarity):

reg add HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters /v EnableTCPChimney
    /t REG_DWORD /d 0


Windows must be restarted before the change will take effect.

Disabling LSO on Windows Server 2008 and higher

This is easily done using a NETSH command:

netsh interface tcp set global chimney=disabled


Disabling LSO on the Ethernet adapter

This works in all versions of versions of Windows Server since it's done at the driver level.  Go to where the network adapters are located in the Control Panel.  For Windows Server 2003, this will be under Network Connections.  For Windows Server 2008, this will be under Network and Sharing Center  –>  Change Adapter Settings.

Now right-click on the network adapter and choose Properties from the pop-up menu.  At the top of this windows will be a "Connect using" text field with the vendor and model of the network adapter.  For my example, I'm using an Intel 52575 Gigabit adapter.  Just below this text field, click on the Configure button.

Now click on the Advanced tab, which shows the configurable properties for the adapter.  Find the entry for Large Send Offload.  This is how it's labeled on Intel adapters, but will vary (sometimes wildly) for adapters from other other vendors.  If it's modern adapter like this one, there will be a setting for both IPv4 and IPv6.  For older adapters, there will only be a setting for IPv4.  Change the value for Large Send Offload from "Enabled" (or "On") to "Disabled" (or "Off") and click on OK.

Intel NIC Properties

W A R N I N G:  Changing any of the adapter policies causes the driver to be restarted.  There will be a brief (1 – 2 min) loss of network connectivity.

Posted in Networking, Windows.

Large Send Offload and Network Performance

One issue that I continually see reported by customers is slow network performance.  Although there are literally a ton of issues that can effect how fast data moves to and from a server, there is one fix I've found that will resolve this 99% of time — disable Large Send Offload on the Ethernet adapter.

So what is Large Send Offload (also known as Large Segmetation Offload, and LSO for short)?  It's a feature on modern Ethernet adapters that allows the TCP\IP network stack to build a large TCP message of up to 64KB in length before sending to the Ethernet adapter.  Then the hardware on the Ethernet adapter — what I'll call the LSO engine — segments it into smaller data packets (known as "frames" in Ethernet terminology) that can be sent over the wire. This is up to 1500 bytes for standard Ethernet frames and up to 9000 bytes for jumbo Ethernet frames.  In return, this frees up the server CPU from having to handle segmenting large TCP messages into smaller packets that will fit inside the supported frame size.  Which means better overall server performance.  Sounds like a good deal.  What could possibly go wrong?

Quite a lot, as it turns out.  In order for this to work, the other network devices — the Ethernet switches through which all traffic flows — all have to agree on the frame size.  The server cannot send frames that are larger than the Maximum Transmission Unit (MTU) supported by the switches.  And this is where everything can, and often does, fall apart.

The server can discover the MTU by asking the switch for the frame size, but there is no way for the server to pass this along to the Ethernet adapter.  The LSO engine doesn't have ability to use a dynamic frame size.  It simply uses the default standard value of 1500 bytes,or if jumbo frames are enabled, the size of the jumbo frame configured for the adapter.  (Because the maximum size of a jumbo frame can vary between different switches, most adapters allow you to set or select a value.)  So what happens if the LSO engine sends a frame larger than the switch supports?  The switch silently drops the frame.  And this is where a performance enhancement feature becomes a performance degredation nightmare.

To understand why this hits network performance so hard, let's follow a typical large TCP message as it traverses the network between two hosts.

  1. With LSO enabled, the TCP/IP network stack on the server builds a large TCP message.
  2. The server sends the large TCP message to the Ethernet adapter to be segmented by its LSO engine for the network.  Because the LSO engine cannot discover the MTU supported by the switch, it uses a standard default value.
  3. The LSO engine sends each of the frame segments that make up the large TCP message to the switch.
  4. The switch receives the frame segments, but because LSO sent frames larger than the MTU, they are silently discarded.
  5. On the server that is waiting to receive the TCP message, the timeout clock reaches zero when no data is received and it sends back a request to retransmit the data.  Although the timeout is very short in human terms, it rather long in computer terms.
  6. The sending server receives the retransmission request and rebuilds the TCP message.  But because this is a retransmission request, the server does not send the TCP message to the Ethernet adapter to be segmented.  Instead, it handles the segmentation process itself.  This appears to be designed to overcome failures caused by the offloading hardware on the adapter.
  7. The switch receives the retransmission frames from the server, which are the proper size because the server is able to discover the MTU, and forwards them on to the router.
  8. The other server finally receives the TCP message intact.

This can basicly be summed up as offload data, segment data, discard data, wait for timeout, request retransmission, segment retransmission data, resend data.  The big delay is waiting for the timeout clock on the receiving server to reach zero.  And the whole process is repeated the very next time a large TCP message is sent.  So is it any wonder that this can cause severe network performance issues.

This is by no means an issue that effects only Peer 1.  Google is littered with artices by major vendors of both hardware and software telling their customers to turn off Large Send Offload.  Nor is it specific to one operating system.  It effects both Linux and Windows.

I've found that Intel adapters are by far the worst offenders with Large Send Offload, but Broadcom also has problems with this as well.  And, naturally, this is a feature that is enabled by default on the adapters, meaning that you have to explicitly turn it off in the Ethernet driver (preferred) or server's TCP/IP network stack.

In the next article, I'll describe how to turn off Large Send Offload on both Linux and Windows systems.

Posted in Linux, Networking, Windows.

Time and the Single Windows Server

How Windows servers update, or not update, their clocks is something that might not be readily understood by the average hosting customer, but it can be very important if your web applications are time-sensitive.  In this article, I’m going to cover time on standalone (or single – couldn’t resist not using this title) Windows servers.  A standalone server is one that is not a member of an Active Directory domain.  I’ll cover time on domain servers in the another article.

Installed out of the box, every Windows server is set to update its time from, which is an NTP service maintained by Microsoft.  But as the number of Windows servers on the Internet increases, this service has rapidly become overloaded.  I’ve encountered countless errors in the System Event Log where the server was unable to contact  The key to good timekeeping on Windows, therefore, is to find another reliable time source (short of purchasing your own Stratum 2 time server).

Naturally, the Internet long ago solved this problem with the NTP Pool Project.  It describes itself as “a big virtual cluster of timeservers providing reliable easy to use NTP service for millions of clients.”  These are systems located around the world acting as highly accurate time servers that allow anonymous connections via the Network Time Protocol (NTP).  The load is distributed across all the servers, so every time you connect, it will be to a different pool server.  All of which is handled through the magic of DNS.

The host name used for the NTP Pool Project server allows you to use any of the servers located around the world, or to restrict youself to servers located in a particular country or in a particular region.  Naturally, there are more servers located in some countries than others (United States, Germany, France, and United Kingdom are the top four, in that order), so in the US or UK, you can safely restrict yourself to your country.  But in most other places, you’ll want to use the bigger regional pool or the entire world-wide pool.  This is especially true for Africa (15 servers), South America (43 servers), and Oceania (89 servers).  The country pools use the two-letter country code in the host name, and the regional pools use the region’s name in the host name.  If you exclude the country or region in the host name, all the servers in the world-wide pool are available to you.

Setting a Windows server to use the NTP Pool Project servers (or any other NTP server, for that matter) is best done using the W32TM command.  You can set the NTP server from the “Internet Time” tab of the Date and Time applet in the Control Panel, but the W32TM allows to set multiple time servers and configure other options.  This command, which works with Windows Server 2003 and higher, will set the server to update time from the NTP Pool Project US servers (this command is one line and broken into two lines here for clarity):

w32tm /config "/" /syncfromflags:manual /update

Here is what the option mean:

/config  –  queries or updates the Windows Time Service configuration

/manualpeerlist  –  specifies the NTP servers to use.  This is a space-delimited list of DNS and/or IP addresses and must be enclosed in quotes when more than one server is specified.

/syncfromflags  –  sets what sources should used.  Choices are MANUAL (use the manual peer list), DOMHIER (use an Active Directory domain controller), ALL (use both sources), or NO (don’t sync from an NTP server).  The DOMHIER option only makes sense on a server joined to a domain.

/update  –  commits the changes.

After you change the NTP server list, you have to restart the Windows Time service, either from the Services MMC or using the NET commands as shown below:

net stop w32time
net start w32time

NOTE:  On Windows Server 2008 and higher, the Windows Time service is not set to automatically run on standalone servers.  You need to change the Startup Type in the service properties.

You can verify that your server is updating its time from the NTP Pool Project servers by looking for Time-Service events (Event ID 37) in the System Event Log.

Time Update Event

Time on standalone Windows servers is not as critical as it is on domain member servers and workstations, as we will see in the next article.  But having the correct time makes everything that uses a timestamp (and there are a lot of them) much easier to understand, especially when troubleshooting problems that only happen during a particular time.

Posted in Windows.

Getting Version Information of a Microsoft SQL Server Instance

When working on a customer's Microsoft SQL Server, it often helpful to know what edition it is and what service packs are install.  This is a Transact-SQL query that will return (almost) everything one needs to know about the SQL Server instance:

SELECT SERVERPROPERTY('machinename') AS 'Server Name',
     SERVERPROPERTY('instancename') AS 'Instance Name',
     SERVERPROPERTY('productversion') AS 'SQL Server Version',
     SERVERPROPERTY('productlevel') AS 'Product Level',
     SERVERPROPERTY('edition') AS 'SQL Server Edition',
     SERVERPROPERTY('collation') AS 'Default Collation',
     SERVERPROPERTY('licensetype') AS 'License Type';

The results returned by this query are:

Server Name  –  The name of the computer on which SQL Server is installed.

Instance Name  –  If SQL Server was installed as the default instance, this value will be NULL.  Otherwise, it will name of the instance.

SQL Server Version  –  The version of the SQL Server instance.  This can be used to determine what updates have been applied to the instance.

Product Level  –  The current installed Service Pack.  If this value is RTM (Release To Manufacturing), then no service pack is installed.

SQL Server Edition  –  This is the edition of the SQL Server instance.  Possible values are “Express Edition”, “Workgroup Edition”, “Web Edition”, “Standard Edition”, or “Enterprise Edition”.  The edition determines what features are available in the SQL Server instance.

Default Collation  –  The default server collation value that was selected when the SQL Server instance was installed.

License Type  –  The type of client licenses used by the SQL Server instance.  If this value is DISABLED, then the license type is per-processor (which is anonymous access).

This query works with all editions of SQL Server 2005 and higher.

Posted in Windows.

FrontPage Server Extensions – The End Is Nigh

How many of you out there are still using FrontPage Server Extensions?  A few hands.  Most have moved off FrontPage, but there are still a few whose websites are totally dependent on FrontPage.  For whatever reason, they haven't revamped their code to get away from FrontPage. Microsoft officially killed off FrontPage Server Extensions with the release of Windows Server 2008 and IIS 7.  But despite everything that Microsoft has done, FrontPage continues to cling to life by its fingernails.  It's been a while, but just recently I ran into someone still using FrontPage.  A customer whose old server died and had to be rebuilt was facing the sudden and unplanned upgrade from Windows Server 2003 to Windows Server 2008.  And all of his web applications depended heavily on FrontPage Server Extensions.  As one who has helped out customers migrating to new servers, I've had to advise them about the current state of FrontPage and their options.

Official Status of FrontPage Server Extensions – The View from Redmond

Windows Server 2003 (and R2) was the last version to include support for FrontPage Server Extensions.  Microsoft FrontPage 2002 (the web designer application) was last version to be included with Microsoft Office, in the higher-end editions of Office XP.  FrontPage 2003 was released as a standalone product and was not included in any edition of Office 2003.  And with that, Microsoft hoped to move people along to ASP.NET, Expression Web (the web designer replacement for FrontPage), and SharePoint.  Realizing that people might still need to work with their FrontPage web applications for a while, Expression Web was able to open and edit, but not create, FrontPage web pages. The FrontPage Server Extensions that were released for Apache running on Unix/Linux were withdrawn by Microsoft in 2006, and though they can still be used on the servers where they were installed, it's technically illegal to install them on a new server.  But I seriously doubt many in the Linux community shed any tears over this. Windows Server 2008 was released without any support for FrontPage Server Extensions from Microsoft.  Legacy ASP code was fully supported, but FrontPage was officially dead.  (But what Microsoft wasn't telling anybody was that FrontPage Server Extensions were still being used to publish ClickOnce applications as late as Visual Studio 2008.)

Ready-to-Run Software to the Rescue

While Microsoft declared FrontPage dead, there were enough people out there still dependent on FrontPage Server Extensions that Ready-to-Run Software created a package that could be installed on Windows Server 2008 and IIS 7 to support FrontPage web applications.  This is a free-to-use download from the IIS community site.  I personally have not used this package, so you are on your own. For those running Windows Server 2008 R2 and IIS 7.5, Ready-To-Run has also released a FrontPage Server Extensions package for that platform as well.  But this time, you have to buy a license to use it.  I guess they figure that anyone still using FrontPage web applications are desperate enough to pony up some cash to put off the inevitable for one more version of Windows Server.

The Sun Sets on FrontPage

Despite the work by Ready-To-Run Software, this has only bought those you still using FrontPage Server Extensions a short lease on life (for your web applications, that is).  Windows Server 2012 is unlikely to have any support for FrontPage Server Extensions, so a revamp of your web applications is unavoidable.

Posted in Windows.

IIS SMTP Server SmtpOutboundCommandSupportOptions Bug

The Simple Mail Transport Protocol (or SMTP for short) Server has been a part of Internet Information Services (IIS) since the days of Windows NT Server.  Over the years, little about it has changed, nor is any big revamp expected by Microsoft.  This is underscored by the fact that the SMTP Server was not made a part of IIS 7, instead remaining available in Windows Server 2008 has an IIS 6 legacy feature.  And for the most part, this is fine.  Although totally useless as an inbound mail server (except for Windows Server 2003, when Microsoft added a matching POP3 server), it excels in one area that keeps it firmly a part of the Windows Server feature set, even though IIS has moved on to be fully integrated with the Microsoft .NET framework.  It's a capable relay for outbound email create by web applications.

The SMTP Server is a smart mail server, meaning that it can deliver messages directly to the recipient's mail server.  Which allows web applications to send out email directly from the server without having to relay through an intermediate mail system.  (Of course, there is a trick to this, and I'll cover the proper configuration the IIS SMTP Server in a future post.)  But while the SMTP Server hasn't changed much over the years, how mail servers communicate with other did when Enhanced SMTP (ESMTP) was introduced to extend the SMTP command set.  It has eclipsed the older SMTP command set and ESMTP is now used by the vast majority of all mail servers.  In fact, the large public email services like AOL, Gmail, Hotmail (or, and Yahoo won't even talk to a mail server that doesn't speak ESMTP.

And here lies a big problem for most IIS SMTP Servers.  Although the SMTP Server supports ESMTP, it is disabled by default!  Thus when the SMTP Server tries to deliver email to recipients at one of the large public email services, the connection is simply closed without any failure notice.  The SMTP Server keeps trying to deliver the email, each time being rebuffed by the large public email services, until it finally gives up and moves the message to the Badmail folder.

When I encountered this a couple of years back, I was trying to figure out why email relayed through the SMTP Server wasn't getting to Gmail and Yahoo recipients.  Mail was being delivered to other mail systems just fine, just not the big players, which represented about half of the intended recipients.  I finally stumbled upon an obscure TechNet article that partially (partially!) documented the IIS metabase setting called SmtpOutboundCommandSupportOptions.  This a bit-mapped value, and by setting or clearing various bits, you can turn off or on support for various ESMTP commands when making outbound connections.  (There is a similar setting called SmtpInboundCommandSupportOptions which does the same thing for inbound connections.)

According to the documentation, the value of the SmtpOutboundCommandSupportOptions setting defaults to 7697601, which turns on support for all the ESMTP commands for outbound connections.  But the default value as listed in the official Microsoft documentation is totally incorrect.  The real default value is 7!  Which means that ESMTP support for outgoing connections is disabled, so trying to communicate with the large public email services is a completely futile exercise.  By when I changed the setting to officially document default value of 7697601, the SMTP Server was now able to deliver email to the large public email services without problems, and the outbound mail queue was suddenly empty.

So how do you change the value of the SmtpOutboundCommandSupportOptions setting?  It's actually very simple.  It can done using the ADSUTIL.VBS script or by editing the metabase directly.

Using The ADSUTIL.VBS Script

ADSUTIL.VBS is a VBscript utility created by Microsoft for working with IIS metabase.  It must be run from a Command-Line Prompt windows using CSCRIPT.  It is normally find in the folder C:\inetpub\AdminScripts\, and on Windows Server 2008, requires that the "IIS 6 Scripting Tools" feature be installed.  Some hosting companies move this to a different folder in their Windows Server builds, which was the case for Windows Server 2003 at Peer 1, where it's locating the D:\AdminScripts folder (we stopped moving it in Windows Server 2008).  Once you've located the ADSUTIL.VBS script, open a Command Prompt windows, change to the directory where the script is located, and run the following command:

cscript adsutil.vbs SET SmtpSvc/SmtpOutboundCommandSupportOptions 7697601

which will return the following message:

SmtpOutboundCommandSupportOptions: (Long) 7697601

In order for the change to take effect, you need to restart the "Simple Mail Transport Protocol" service.

Editing The IIS Metabase

Should the ADSUTIL.VBS script not be available, you can edit the IIS metabase directly using Notepad.  NOTE: You will need to stop IIS in order to save the changes made in Notepad.

  1. Stop the IIS services using the command:

  2. Open the file C:\WINDOWS\system32\inetsrv\MetaBase.xml in Notepad.
  3. Under the section IIsSmtpService Location = "/LM/SmtpSvc", find the property SmtpOutboundCommandSupportOptions. Since this property appears only once in the metabase, the quickest way to locate it is Notepad's Find utility (Edit –> Find).
  4. Change the property so it reads:

  5. Save the file.
  6. Restart the IIS services using the command:


And that's all there is to it.

Posted in Windows.
All information in this blog is provided "AS IS" with no warranties and confers no rights.
The opinions expressed in this blog are mine alone and do not represent those of my employer.
Powered By PEER 1 Managed Hosting