Sizing The Windows Page File

A common question is how big should the Windows page file be?  In my opinion, as small as possible.

The Windows page file is analogous to the Linux swap file.  It's a file on the hard disk that serves as virtual memory.  If all running the applications won't fit inside physical RAM, Windows swaps out lesser-used segments of memory to the page file so it can make room for other applications.  But it's a relic from the days when computers had no where near the amount of RAM they have to today.

In practice, you never want to use the page file.  Why?  Because paging operations (swapping memory segments between physical RAM and the page file) is expensive.  Very expensive!  Disk I/O is orders of magnitude slower than physical RAM, and when Windows is doing a lot of paging operations, system performance takes a big hit.  It can slow the system to the point that the screen cursor needs minutes to respond when the mouse is moved.  And then the disk activity light and the power light are indistinguishable from each other — both shining with a steady glow.  Usually the only way to recover when this happens is a hard reboot of the system.

The problem is that the default for automatically allocating the page file hasn't changed since Windows NT Server 3.5.  It creates a page file that is 1.5 times the size of physical RAM.  A good value when servers had 512 MB of RAM, but extremely wasteful on a server with 16 GB of RAM.  That's a 24 GB page file!  And did I mention that the default location for the page file is the C drive?  With everything else that is vying for space on the C drive, the last thing you need is a gigantic file you never want to use.

You can have no page file, but this isn't a good idea.  There are a few rarely used things that Windows wants to keep in the page file, but these are never that big.  A good value for the page file is between 2 – 4 GB.  Resist the temptation to make it any bigger than 4 GB, regardless of what "best practices" say.  Remember, you never want to use the page file.  And you only ever need one page file.  Having multiple page files is even more wasteful than having a extremely large page file.  Don't do it!

Use a custom size for the page file and set the initial and maximum size to the same value.  If these values are not the same, then the page file will become fragmented as Windows shrinks and expands the page file.  You want to keep disk I/O operations for the page file to an absolute minimum, so let Windows create the page file once and keep it from constantly resizing it.

Posted in Windows.

DNS: Understanding The SOA Record

In the hosting industry, the Domain Name System (DNS) is one of the most critical pieces, right behind websites themselves.  Without DNS, that website you've worked so hard on would be completely invisible.  (Although it's possible to access some sites using only the IP address of their web server, this is not the case for virtual websites, which require that their hostname be included in the HTTP request header.  Without a working DNS record, virtual websites are completely inaccessible.)  But I've found that DNS is something that is not well understood by many website operators.  The basics of creating A records (which translate a hostname to an IP address) are simple enough, but when it comes to understanding how changes are propagated in DNS, this is often something of a mystery.

There is a widely held belief that any change made the DNS zone file of a domain is instantly seen throughout the Internet.  Yet nothing could be further from the truth.  When advising that changes be made to a zone file to fix a problem, I routinely add the following caveat:

Please allow up to 24 hours for any change to completely propagate throughout the world-wide DNS system.

Changes to a zone file are almost never instantaneous regardless of how despreate you are that they be instantaneous.  Any change requires time before it will be seen everywhere on the Internet.  But what many don't understand is that how fast or slow these updates are propagated is actually under their direct control through the SOA record.

Let me be completely clear on this one point.  Although you have control over the speed that updates are propagated throughtout the Internet, they will never, ever, be instantenous!  There will always be a delay.  Your only control is over how short or long this delay will be.

SOA: Start Of Authority

The SOA record is perhaps the least understood record in the entire zone file.  But it controls the speed that any update is propagated thourghout the Internet.  The purpose of the SOA record is:

  • Identify the DNS server that is authoritative for all information within the domain.
  • List the email address of the person in charge of the domain.
  • Control how often secondary servers check for changes to the zone file.
  • Control how long secondary servers keep the zone file active when the primary server cannot be contacted.
  • Control how long a negative response is cached by a DNS resolver (but for some DNS servers, this is also how long a DNS resolver should cache any response).

Now if you control all of the authorative DNS servers for a domain (that is, the DNS servers that actually host the zone files and can answers queries for the domain as opposed to having to ask another DNS server), then with the exception of how long negative responses should be cached, these settings may not seem as important since you can force the secondary servers to update whenever needed.  By if you are using third-party name servers which you do not control as your secondary name servers (such as Peer 1's SuperDNS servers), then these settings are vitally important to how fast any changes are propagated.  So let's go over each of these settings.

I will be using the official names for each of these fields as listed in RFC 1035: Domain Names — Implementation and Specification.

MNAME:  Primary Name Server

Fully-qualified domain name of the primary or master name server for the zone file.  Within the structure of DNS, there can only be one server that holds the master, editable zone file.  (Yes, there are exceptions, but I won't cover them here.)  All secondary name servers create their zone files by transferring the contents from the primary name server.  Changes to the domain's resource records are made to the primary name server's zone file and are then propagated to the secondary name servers when they check for updates.

The domain name of the primary name server must end with a period.

RNAME:  Responsible Person

Email address of the person responsible for the domain's zone file.  Often it will be an alias or group address rather than a particular idividual.  It uses a special format where the "@" character is replaced with a "." (period) character and the email address ends with a period.  So the email address would become (note that the endding period is part of the email address).

Never use an email address which uses a period before the "@" character (such as since DNS will automatically interpret the first period as the "@" character (where would become


Serial number of the zone file that is incremented each time a change is made.  The secondary name servers compare the serial number returned by the primary name server with the serial number in their copy of the zone file to determine if they should update their zone file.  If the serial number from the primary name server is greater than their serial number, they will do a zone update transfer.  Otherwise, no action is taken.

If you make a change to the zone file on the primary name server and forget to increment the serial number, the change will not be propagated to the secondary name servers even if you attempt to force a zone update transfer.  The primary and secondary name servers will remain out of sync until the serial number is incremented on the primary name server.  Unless you are manually editing the zone files (something that is not uncommon when using BIND), most DNS servers or frontend DNS applications will increment the serial number for you.  But if you find that updates are not being propagated to the secondary name servers, the serial number is the first thing you should check.

In the early days of DNS, the serial number was just that — a number that was incremented by 1 each time the zone file was changed.  So that one could have a better idea of when the zone file was actually changed, it's recommended (but not required) that you use the format YYYYMMDDnn, where YYYY is the year, MM is the month, DD is the day, and nn is the revision number (in case the zone file is changed more than once in a single day).

Never use a decimal in the serial number, such as 20130511.01, even if it is allowed by your DNS server.  The serial number is an unsigned 32-bit number, so using a decimal in the serial number will cause it be converted to something unexpected.

REFRESH:  Refresh Interval

Time in seconds that a secondary name server should wait between zone file update checks.  The value should not be so short that the primary name server is overwhelmed by update checks and not so long that propagation of changes to the secondary name servers are unduely delayed.  If you control the secondary name servers and the zone file doesn't change that often, then you might want to set this to as long as day (86400 seconds), especially if you can force an update on the secondary name servers if needed.  But if your secondary name servers are not under your control, then you'll probably want to set this to somewhere between 30 minutes (1800 seconds) and 2 hours (7200 seconds) to ensure any changes you make are propagated in a timely fashion.

Even if you configure your primary name server to send NOTIFY messages (which I will cover in a future article) to the secondary name servers whenever a change is made, you should never completely depend on this to ensure timely propagation of the changes, especially when using third-party secondary name servers. The decision to honor a NOTIFY message is entirely up to the secondary name server and some DNS servers do not support NOTIFY.

RETRY:  Retry Interval

Time in seconds that a secondary name server should wait before trying to contact the primary name server again after a failed attempt to check for a zone file update.  There are all kinds of reasons why a zone file update check could fail, and not all of them mean that there is something wrong with the primary name server.  Perhaps it was too busy handling other requests just then.  The Retry Interval simply tells the secondary name server to wait for a period of time before trying again.  A good retry value would be between 10 minutes (600 seconds) and 1 hour (3600 seconds), depending on the length of the Refresh Interval.

The retry interval should always be shorter than the refresh interval.  But don't make this value too short.  When in doubt, use a 15 minute (900 second) retry interval.

EXPIRE:  Expiry Interval

Time in seconds that a secondary name server will treat its zone file as valid when the primary name server cannot be contacted.  If your primary name server goes offline for some reason, you want the secondary name names to keep answering DNS queries for your domain until you can get the primary back online.  Make this value too short and your domain will disapear from the Internet before you can bring the primary back online.  A good value would be something between 2 weeks (1209600 seconds) and 4 weeks (2419200 seconds).

If you stop using a domain and delete it from the configuration of the primary name server, remember to remove it from the secondary name servers as well.  This is especially important if you use third-party secondary name servers since they will continue to answer queries for the deleted domain — answers which could now be completely incorrect — until the expiry interval is reached.

MINIMUM:  Negative Caching Time To Live

This field requires special attention since how it's interpreted depends on the DNS server you are using.  There have been three possible meanings for the MINIMUM field:

  • Defines the minimum time in seconds that a resource record should be cached by any name server.  Though this was the original meaning of this field (and it still retains the name from this meaning), it was never actually used this way by most name servers.  This meaning is now officially deprecated.
  • Defines the default Time To Live (TTL) for all resource records that do not have an explicit TTL.  This only applies to the zone file on the primary name server since a zone transfer to the secondary server adds the explicit TTL to the resource record if it is missing.  Versions of BIND prior to 8.2 use the MINIMUM field as the default TTL for all resource records, as do all versions of Windows DNS Server.
  • Defines the time in seconds that any name server or resolver should cache a negative response.  This is now the official meaning of this field as set by RFC 2308.

Unlike all the other SOA fields, MINIMUM effects every name server or resolver that queries your domain.  If your DNS server is compliant with RFC 2308, then this field only applies to how long a negative response (that is, for a query where no resource record is found) is cached.  But if your DNS server uses this as the default TTL for resource records without an explicit TTL, then it controls how long any response could be cached by a name server.

If you make this too long, then name servers and resolvers will keep using their cached result even after all the secondary name servers have updated their zone files.  And there is no method available for you to force these name servers and resolvers to flush their cache.  Again, if your DNS server is compliant with RFC 2308, it only applies to negative responses.  But if not, then all resource records without an explicit TTL will use this value as the default TTL.  If you were to set this to 1 week (604800 seconds), then it could take up to a week for any change to finally be seen everywhere on the Internet.

$TTL:  Default Time To Live

This was added in RFC 2308 to define the default TTL to should be used for any resource record that does not have an explicit TTL.  But as pointed out earlier, not all DNS servers support it.  BIND 8.2 and higher use $TTL to define the default TTL in their zone files, but Windows DNS Server does not, relying on the SOA MINIMUM field instead,  So check you DNS server manual to find out how it sets the default TTL.

Final Thoughts

There is no hard and fast rule for setting the refresh, retry, and TTL values.  For domains where changes are rarely done, longer values are usually preferred.  But if are planning to make changes, then reducing these values before hand, especially the default TTL, can go a long way to ensuring your changes get propagated in a timely fashion.  But you must change these values at least as far in advance as the default TTL.  If, for example, the current default TTL is set to one week, you'll need to change the default TTL at least a week before the zone file is changed to ensure that every DNS server and resolver is using the new TTL.  Otherwise you could find that scattered sections of the Internet don't see the change until the older, cached record finally expires.

Posted in DNS.

Disappearing VSS System Writer and ASP.NET

When a server backup program utilizes Volume Shadow Copy Service (VSS), sometimes you have to fix problems when one or more of the writers fails or disappears.  This can usually be repaired by running a script that re-registers the various DLLs used by VSS.  But we ran into a problem on one server that defied this traditional fix.  Even though the script was executed and the server rebooted several times, the System Writer would disappear every time the backup was attempted.  And it always failed in the same place — backing up the System State.

Since the tried and true fix of re-registering the DLLs wasn't working, we needed to look elsewhere.  And the problem turned out to be something we hadn't seen before.  Too many files in the Temporary ASP.NET Files directories.

The System State

According Microsoft TechNet, the System State is comprised of the following:

  • Boot files, including the system files, and all files protected by Windows File Protection (WFP).
  • Active Directory (on a domain controller only).
  • Sysvol (on a domain controller only).
  • Certificate Services (on certification authority only).
  • Cluster database (on a cluster node only).
  • The registry.
  • Performance counter configuration information.
  • Component Services Class registration database.

System files includes everything under the C:\Windows folder whether or not it's actually needed to restore the System State.  (Since the majority of the files found in the C:\Windows folder are required for the operation of the system, it's much safer to simply back up everything rather than miss a critical file.)  To create a System State backup, the VSS System Writer must first enumerate all the files and folders that make up the system files, and this where it ran afoul of the ASP.NET temporary files.

ASP.NET Temporary Files

One of the principal features of Microsoft.NET is that applications can be run on multiple operating systems without having to rebuild the program.  (The open source Mono Project uses this very feature to bring the .NET Framework to Linux, Apple's OS X, and many other non-Microsoft platforms.)  They are initially compiled into a machine-independent Intermediate Language (IL), and it's in this form that they are installed on the target system.  Then when the program is executed, the program's IL code is compiled into machine code by an operating system-specific compiler.

Having to compile the IL code is an expensive operation, so to speed up subsequent executions of the program, .NET saves the compiled machine code in a temporary directory.  When the program is run again, .NET checks to see if there is a cached copy of the compiled machine code.  If there is, it skips compiling the IL code and runs the cached machine code instead.  For ASP.NET programs, the compiled machine code is cached in a folder under C:\Windows\Microsoft.NET.

ASP.NET is different from other types of .NET programs in that each ASP.NET page is considered to be a separate program.  This means that lots of compiled machine code gets cached.  On a server that hosts hundreds of ASP.NET websites, there can be thousands, or even tens of thousands, of cached machine code files.

The System Writer Disappears

Because the default location of the ASP.NET temporary directories is under C:\Windows\Microsoft.NET, these files are considered to be system files, and therefore part of the System State.  So when VSS is used to create a backup of the System State, the System Writer enumerates all the system files, including the ASP.NET temporary files.  But there is a limit to the number of files that the System Writer can deal with.  When this limit is exceeded, the System Writer aborts with an error, causing the System State backup to fail.

Once the System Writer has aborted, it disappears from the list of VSS writers until the service that controls it — which is the Cryptographic Services — is restarted.  But even when the Cryptographic Services is restarted, the System Writer will simply abort again the next time it tries to enumerate all the ASP.NET temporary files.

Relocating the ASP.NET Temporary Directories

None of the ASP.NET temporary files are required to restore the System State from a backup.  If they are deleted, they are recreated by compiling the IL code the next time the ASP.NET application is executed.  But by virtue of their location under the C:\Windows\Microsoft.NET folder, they are considered to be part of the System State.  So to get the ASP.NET temporary files out of the System State, we need to move these temporary directories to a different location.

Doing this requires that you perform a number of steps, which I describe below.  The names and locations for the new folders are just my personal preference.  Change them as desired to meet the needs of your system so long as they are not located under the C:\Windows folder.

Create New ASP.NET Temporary Directories

Despite the various versions of .NET Framework, only three versions of .NET actually have ASP.NET temporary directories.  These are .NET Framework 1.1 (which is not included with Windows Server 2008 and higher and can largely be ignored), .NET Framework 2.0 (which includes .NET 3.0 and 3.5, as these are just extensions for .NET 2.0), and .NET Framework 4.0.  (.NET Framework 4.5 has just been released.  When installed, it replaces .NET Framework 4.0 if it's already installed.  I didn't include it in the example here, but relocating its ASP.NET temporary directories will be similar to .NET 4.0.)  And except for .NET Framework 1.1 (which is 32-bit only), each of these versions has a 32-bit and 64-bit temporary directory.

The first thing we need to do is to create new directories for the ASP.NET temporary files.  These commands will create new directories for the ASP.NET 2.0 and 4.0 temporary files on the D drive.

md "D:\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files"
md "D:\Microsoft.NET\Framework\v4.0.30319\Temporary ASP.NET Files"
md "D:\Microsoft.NET\Framework64\v2.0.50727\Temporary ASP.NET Files"
md "D:\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files"

Set Folder Permissions

Next we need to set the folder permissions to match the existing default directories.  The easiest way to do this is with the ICACLS command.  The first command removes the removes the inherited permissions and replaces them with a copy of the inherited permissions.  Thus changes to the permissions of the drive (root directory) will not effect the new temporary directories.  The remaining commands grant the required permissions.

icacls "D:\Microsoft.NET" /inheritance:d
icacls "D:\Microsoft.NET" /grant:r "BUILTIN\Administrators:(OI)(CI)(F)"
icacls "D:\Microsoft.NET" /grant:r "NT AUTHORITY\SYSTEM:(OI)(CI)(F)"
icacls "D:\Microsoft.NET" /grant:r "CREATOR OWNER:(OI)(CI)(IO)(F)"
icacls "D:\Microsoft.NET" /grant:r "BUILTIN\IIS_IUSRS:(OI)(CI)(M,WDAC,DC)"
icacls "D:\Microsoft.NET" /grant:r "BUILTIN\Users:(OI)(CI)(RX)"
icacls "D:\Microsoft.NET" /grant:r "NT SERVICE\TrustedInstaller:(CI)(F)"
icacls "D:\Microsoft.NET" /grant:r "NT SERVICE\WMSvc:(OI)(CI)(M,DC)"

Add Attribute tempDirectory To The compilation Tag In web.config

For ASP.NET to use temporary directories anywhere other than the default location, the directory must be specified using the tempDirectory attribute of the <compilation> tag in the system web.config file.  There is one file for each version of the .NET Framework.  (Again, these are the same versions that have ASP.NET temporary directories, so there is no web.config file for .NET 3.0 and 3.5.)  The tempDirectory attribute specifies the directory where the compiled machine code will be cached.  The web.config file is a XML file that can be edited with Notepad.

For ASP.NET 2.0 32-bit, we would edit the web.config file and locate this tag:


and change it as follows to use the new temporary directory:

<compilation tempDirectory="D:\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files">

The web.config file is located in the CONFIG folder for the .NET Framework version.  In our example, we will need to edit the following web.config files.

.NET Framework 2.0 – 32-Bit

.NET Framework 2.0 – 64-Bit

.NET Framework 4.0 – 32-Bit

.NET Framework 4.0 – 64-Bit

Restart IIS

For the changes made to the web.config files to take effect, IIS has to be restarted.  This is easily done from the command line.


Delete Files In The Old Temporary Directories

Now we need to delete the files in the old ASP.NET temporary directories so they are no longer part of the system state.  These files are actually in a subfolder named root, so we'll actually delete this folder along with all it's files and subfolders.  Again, this is easily done from the command line.

rmdir /s /q "C:\Windows\Microsoft.Net\Framework\v2.0.50727\Temporary ASP.NET Files\root"
rmdir /s /q "C:\Windows\Microsoft.Net\Framework64\v2.0.50727\Temporary ASP.NET Files\root"
rmdir /s /q "C:\Windows\Microsoft.Net\Framework\v4.0.30319\Temporary ASP.NET Files\root"
rmdir /s /q "C:\Windows\Microsoft.Net\Framework64\v4.0.30319\Temporary ASP.NET Files\root"

Restart The Cryptographic Service

To get the VSS System Writer back, we must restart the service that controls it, which as previously mentioned, was the Cryptographic Service.

net stop cryptsvc
net start cryptsvc

Verifying Everything Is Working

If you did everything correctly, you should see files created in the new ASP.NET temporary directories the next time the website is accessed.  And to verify the System Writer has returned, run the vssadmin command to list the writers.

vssadmin list writers

Now that the ASP.NET temporary files are no longer considered to be system files, the System State backup should complete without errors.

Posted in Windows.

Disabling Large Send Offload – Windows

In an earlier post, I described the Large Send Offload (LSO) feature of modern Ethernet adapters and why it can cause havoc with network performance.  And since this is enabled by default, you have to manually disable it.  For Windows, this can be done in the Ethernet adapter properties (which I prefer) or in the TCP/IP network stack.  I'll start with disabling LSO in the TCP/IP network stack since Microsoft uses some confusing terms that you'll want to be familiar with.

Micosoft refers to LSO as "TCP Chimney Offload".  For Windows Server 2008 and later, it's described in MS Support Article 951037.  LSO was first supported in Windows Server 2003 with the release of the Scalable Networking Pack, which integrated into Service Pack 2, and is described in MS Support Article 912222.

Disabling LSO on Windows Server 2003

This has to be done by editing the registry.  Open regedit and locate this key:


Then right-click on EnableTCPChimney and select Modify from the pop-up menu.  Change the value to 0 and click on OK.  You can also use this REG command (this command is one line but shown here here on two lines for clarity):

reg add HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters /v EnableTCPChimney
    /t REG_DWORD /d 0


Windows must be restarted before the change will take effect.

Disabling LSO on Windows Server 2008 and higher

This is easily done using a NETSH command:

netsh interface tcp set global chimney=disabled


Disabling LSO on the Ethernet adapter

This works in all versions of versions of Windows Server since it's done at the driver level.  Go to where the network adapters are located in the Control Panel.  For Windows Server 2003, this will be under Network Connections.  For Windows Server 2008, this will be under Network and Sharing Center  –>  Change Adapter Settings.

Now right-click on the network adapter and choose Properties from the pop-up menu.  At the top of this windows will be a "Connect using" text field with the vendor and model of the network adapter.  For my example, I'm using an Intel 52575 Gigabit adapter.  Just below this text field, click on the Configure button.

Now click on the Advanced tab, which shows the configurable properties for the adapter.  Find the entry for Large Send Offload.  This is how it's labeled on Intel adapters, but will vary (sometimes wildly) for adapters from other other vendors.  If it's modern adapter like this one, there will be a setting for both IPv4 and IPv6.  For older adapters, there will only be a setting for IPv4.  Change the value for Large Send Offload from "Enabled" (or "On") to "Disabled" (or "Off") and click on OK.

Intel NIC Properties

W A R N I N G:  Changing any of the adapter policies causes the driver to be restarted.  There will be a brief (1 – 2 min) loss of network connectivity.

Posted in Networking, Windows.

What’s Block Size Got To Do With It?

(With apologies to Tina Turner.)

On WMware ESXi/Vsphere versions through 4.X, how big a file can be on a datastore is completely dependent on the block size specified when the datastore was formatted.  And this in turn drives the maximum size of any virtual hard disk.  For many virtual machines, this might not be an issue.  But if you want to create a virtual hard disk greater than 256 GB, then you need to pay attention!

VMFS-2 and VMFS-3

By default, VMware File System (VMFS) uses a 1 MB block size, which supports a maximum file size of 256 GB on VMFS-3 and 456 GB on VMFS-2.  (No, you read that right.  The maximum file size actually went down between VMFS-2 and VMFS-3.)  To support larger file sizes, you need to use a larger block size when the datastore is formatted.

Block Size VMFS-2 Disk Size VMFS-3 Disk Size
1 MB 456 GB 256 GB*
2 MB 912 GB 512 GB*
4 MB 1.78 TB 1 TB*
8 MB 2 TB 2 TB – 512 bytes
16 MB 2 TB ** Not Valid **
32 MB 2 TB ** Not Valid **
64 MB 2 TB ** Not Valid **

* On ESXi 4.0, 512 bytes is subtracted from the maximum file size for any block size on a VMFS-3 datastore.  On ESXi 4.1, this only occurs when the block size is 8 MB.

Once the block size is set, the only way to change it is to reformat the datastore.  Which means moving the existing data elsewhere as a formatting a datastore, like formatting any disk, destorys the existing data.


In ESXi/VSphere 5.0, VMware introduced VMFS-5 (there is no VMFS-4) which uses a unified 1 MB block size that cannot be configured.  But the maximum file size is now 2 TB – 512 bytes, so block size no longer matters.  As is the case with any new feature, it's only available on ESXi/VSphere 5.0 and higher.

If you upgrade an older ESXi/VSphere to 5.0 or higher and the existing datastores use VMFS-3, you can upgrade the datastores to VMFS-5.  This is a non-destructive process, meaning the upgrade can be done with live on the datastore.  You should always have backups in case anything goes wrong!  Upgrading from VMFS-3 to VMFS-5 will not give all the VMFS-5 features, but you will get most of them.

Further Reading

See VMware Knowledge Base Article 1003565 for details on block size and how it effects the maximum file size.

See VMFS-5 Upgrade Considerations for information about upgrading an existing VMFS-3 datastore to VMFS-5.

Posted in VMware.

Large Send Offload and Network Performance

One issue that I continually see reported by customers is slow network performance.  Although there are literally a ton of issues that can effect how fast data moves to and from a server, there is one fix I've found that will resolve this 99% of time — disable Large Send Offload on the Ethernet adapter.

So what is Large Send Offload (also known as Large Segmetation Offload, and LSO for short)?  It's a feature on modern Ethernet adapters that allows the TCP\IP network stack to build a large TCP message of up to 64KB in length before sending to the Ethernet adapter.  Then the hardware on the Ethernet adapter — what I'll call the LSO engine — segments it into smaller data packets (known as "frames" in Ethernet terminology) that can be sent over the wire. This is up to 1500 bytes for standard Ethernet frames and up to 9000 bytes for jumbo Ethernet frames.  In return, this frees up the server CPU from having to handle segmenting large TCP messages into smaller packets that will fit inside the supported frame size.  Which means better overall server performance.  Sounds like a good deal.  What could possibly go wrong?

Quite a lot, as it turns out.  In order for this to work, the other network devices — the Ethernet switches through which all traffic flows — all have to agree on the frame size.  The server cannot send frames that are larger than the Maximum Transmission Unit (MTU) supported by the switches.  And this is where everything can, and often does, fall apart.

The server can discover the MTU by asking the switch for the frame size, but there is no way for the server to pass this along to the Ethernet adapter.  The LSO engine doesn't have ability to use a dynamic frame size.  It simply uses the default standard value of 1500 bytes,or if jumbo frames are enabled, the size of the jumbo frame configured for the adapter.  (Because the maximum size of a jumbo frame can vary between different switches, most adapters allow you to set or select a value.)  So what happens if the LSO engine sends a frame larger than the switch supports?  The switch silently drops the frame.  And this is where a performance enhancement feature becomes a performance degredation nightmare.

To understand why this hits network performance so hard, let's follow a typical large TCP message as it traverses the network between two hosts.

  1. With LSO enabled, the TCP/IP network stack on the server builds a large TCP message.
  2. The server sends the large TCP message to the Ethernet adapter to be segmented by its LSO engine for the network.  Because the LSO engine cannot discover the MTU supported by the switch, it uses a standard default value.
  3. The LSO engine sends each of the frame segments that make up the large TCP message to the switch.
  4. The switch receives the frame segments, but because LSO sent frames larger than the MTU, they are silently discarded.
  5. On the server that is waiting to receive the TCP message, the timeout clock reaches zero when no data is received and it sends back a request to retransmit the data.  Although the timeout is very short in human terms, it rather long in computer terms.
  6. The sending server receives the retransmission request and rebuilds the TCP message.  But because this is a retransmission request, the server does not send the TCP message to the Ethernet adapter to be segmented.  Instead, it handles the segmentation process itself.  This appears to be designed to overcome failures caused by the offloading hardware on the adapter.
  7. The switch receives the retransmission frames from the server, which are the proper size because the server is able to discover the MTU, and forwards them on to the router.
  8. The other server finally receives the TCP message intact.

This can basicly be summed up as offload data, segment data, discard data, wait for timeout, request retransmission, segment retransmission data, resend data.  The big delay is waiting for the timeout clock on the receiving server to reach zero.  And the whole process is repeated the very next time a large TCP message is sent.  So is it any wonder that this can cause severe network performance issues.

This is by no means an issue that effects only Peer 1.  Google is littered with artices by major vendors of both hardware and software telling their customers to turn off Large Send Offload.  Nor is it specific to one operating system.  It effects both Linux and Windows.

I've found that Intel adapters are by far the worst offenders with Large Send Offload, but Broadcom also has problems with this as well.  And, naturally, this is a feature that is enabled by default on the adapters, meaning that you have to explicitly turn it off in the Ethernet driver (preferred) or server's TCP/IP network stack.

In the next article, I'll describe how to turn off Large Send Offload on both Linux and Windows systems.

Posted in Linux, Networking, Windows.

Time and the Single Windows Server

How Windows servers update, or not update, their clocks is something that might not be readily understood by the average hosting customer, but it can be very important if your web applications are time-sensitive.  In this article, I’m going to cover time on standalone (or single – couldn’t resist not using this title) Windows servers.  A standalone server is one that is not a member of an Active Directory domain.  I’ll cover time on domain servers in the another article.

Installed out of the box, every Windows server is set to update its time from, which is an NTP service maintained by Microsoft.  But as the number of Windows servers on the Internet increases, this service has rapidly become overloaded.  I’ve encountered countless errors in the System Event Log where the server was unable to contact  The key to good timekeeping on Windows, therefore, is to find another reliable time source (short of purchasing your own Stratum 2 time server).

Naturally, the Internet long ago solved this problem with the NTP Pool Project.  It describes itself as “a big virtual cluster of timeservers providing reliable easy to use NTP service for millions of clients.”  These are systems located around the world acting as highly accurate time servers that allow anonymous connections via the Network Time Protocol (NTP).  The load is distributed across all the servers, so every time you connect, it will be to a different pool server.  All of which is handled through the magic of DNS.

The host name used for the NTP Pool Project server allows you to use any of the servers located around the world, or to restrict youself to servers located in a particular country or in a particular region.  Naturally, there are more servers located in some countries than others (United States, Germany, France, and United Kingdom are the top four, in that order), so in the US or UK, you can safely restrict yourself to your country.  But in most other places, you’ll want to use the bigger regional pool or the entire world-wide pool.  This is especially true for Africa (15 servers), South America (43 servers), and Oceania (89 servers).  The country pools use the two-letter country code in the host name, and the regional pools use the region’s name in the host name.  If you exclude the country or region in the host name, all the servers in the world-wide pool are available to you.

Setting a Windows server to use the NTP Pool Project servers (or any other NTP server, for that matter) is best done using the W32TM command.  You can set the NTP server from the “Internet Time” tab of the Date and Time applet in the Control Panel, but the W32TM allows to set multiple time servers and configure other options.  This command, which works with Windows Server 2003 and higher, will set the server to update time from the NTP Pool Project US servers (this command is one line and broken into two lines here for clarity):

w32tm /config "/" /syncfromflags:manual /update

Here is what the option mean:

/config  –  queries or updates the Windows Time Service configuration

/manualpeerlist  –  specifies the NTP servers to use.  This is a space-delimited list of DNS and/or IP addresses and must be enclosed in quotes when more than one server is specified.

/syncfromflags  –  sets what sources should used.  Choices are MANUAL (use the manual peer list), DOMHIER (use an Active Directory domain controller), ALL (use both sources), or NO (don’t sync from an NTP server).  The DOMHIER option only makes sense on a server joined to a domain.

/update  –  commits the changes.

After you change the NTP server list, you have to restart the Windows Time service, either from the Services MMC or using the NET commands as shown below:

net stop w32time
net start w32time

NOTE:  On Windows Server 2008 and higher, the Windows Time service is not set to automatically run on standalone servers.  You need to change the Startup Type in the service properties.

You can verify that your server is updating its time from the NTP Pool Project servers by looking for Time-Service events (Event ID 37) in the System Event Log.

Time Update Event

Time on standalone Windows servers is not as critical as it is on domain member servers and workstations, as we will see in the next article.  But having the correct time makes everything that uses a timestamp (and there are a lot of them) much easier to understand, especially when troubleshooting problems that only happen during a particular time.

Posted in Windows.

Getting Version Information of a Microsoft SQL Server Instance

When working on a customer's Microsoft SQL Server, it often helpful to know what edition it is and what service packs are install.  This is a Transact-SQL query that will return (almost) everything one needs to know about the SQL Server instance:

SELECT SERVERPROPERTY('machinename') AS 'Server Name',
     SERVERPROPERTY('instancename') AS 'Instance Name',
     SERVERPROPERTY('productversion') AS 'SQL Server Version',
     SERVERPROPERTY('productlevel') AS 'Product Level',
     SERVERPROPERTY('edition') AS 'SQL Server Edition',
     SERVERPROPERTY('collation') AS 'Default Collation',
     SERVERPROPERTY('licensetype') AS 'License Type';

The results returned by this query are:

Server Name  –  The name of the computer on which SQL Server is installed.

Instance Name  –  If SQL Server was installed as the default instance, this value will be NULL.  Otherwise, it will name of the instance.

SQL Server Version  –  The version of the SQL Server instance.  This can be used to determine what updates have been applied to the instance.

Product Level  –  The current installed Service Pack.  If this value is RTM (Release To Manufacturing), then no service pack is installed.

SQL Server Edition  –  This is the edition of the SQL Server instance.  Possible values are “Express Edition”, “Workgroup Edition”, “Web Edition”, “Standard Edition”, or “Enterprise Edition”.  The edition determines what features are available in the SQL Server instance.

Default Collation  –  The default server collation value that was selected when the SQL Server instance was installed.

License Type  –  The type of client licenses used by the SQL Server instance.  If this value is DISABLED, then the license type is per-processor (which is anonymous access).

This query works with all editions of SQL Server 2005 and higher.

Posted in Windows.

FrontPage Server Extensions – The End Is Nigh

How many of you out there are still using FrontPage Server Extensions?  A few hands.  Most have moved off FrontPage, but there are still a few whose websites are totally dependent on FrontPage.  For whatever reason, they haven't revamped their code to get away from FrontPage. Microsoft officially killed off FrontPage Server Extensions with the release of Windows Server 2008 and IIS 7.  But despite everything that Microsoft has done, FrontPage continues to cling to life by its fingernails.  It's been a while, but just recently I ran into someone still using FrontPage.  A customer whose old server died and had to be rebuilt was facing the sudden and unplanned upgrade from Windows Server 2003 to Windows Server 2008.  And all of his web applications depended heavily on FrontPage Server Extensions.  As one who has helped out customers migrating to new servers, I've had to advise them about the current state of FrontPage and their options.

Official Status of FrontPage Server Extensions – The View from Redmond

Windows Server 2003 (and R2) was the last version to include support for FrontPage Server Extensions.  Microsoft FrontPage 2002 (the web designer application) was last version to be included with Microsoft Office, in the higher-end editions of Office XP.  FrontPage 2003 was released as a standalone product and was not included in any edition of Office 2003.  And with that, Microsoft hoped to move people along to ASP.NET, Expression Web (the web designer replacement for FrontPage), and SharePoint.  Realizing that people might still need to work with their FrontPage web applications for a while, Expression Web was able to open and edit, but not create, FrontPage web pages. The FrontPage Server Extensions that were released for Apache running on Unix/Linux were withdrawn by Microsoft in 2006, and though they can still be used on the servers where they were installed, it's technically illegal to install them on a new server.  But I seriously doubt many in the Linux community shed any tears over this. Windows Server 2008 was released without any support for FrontPage Server Extensions from Microsoft.  Legacy ASP code was fully supported, but FrontPage was officially dead.  (But what Microsoft wasn't telling anybody was that FrontPage Server Extensions were still being used to publish ClickOnce applications as late as Visual Studio 2008.)

Ready-to-Run Software to the Rescue

While Microsoft declared FrontPage dead, there were enough people out there still dependent on FrontPage Server Extensions that Ready-to-Run Software created a package that could be installed on Windows Server 2008 and IIS 7 to support FrontPage web applications.  This is a free-to-use download from the IIS community site.  I personally have not used this package, so you are on your own. For those running Windows Server 2008 R2 and IIS 7.5, Ready-To-Run has also released a FrontPage Server Extensions package for that platform as well.  But this time, you have to buy a license to use it.  I guess they figure that anyone still using FrontPage web applications are desperate enough to pony up some cash to put off the inevitable for one more version of Windows Server.

The Sun Sets on FrontPage

Despite the work by Ready-To-Run Software, this has only bought those you still using FrontPage Server Extensions a short lease on life (for your web applications, that is).  Windows Server 2012 is unlikely to have any support for FrontPage Server Extensions, so a revamp of your web applications is unavoidable.

Posted in Windows.

IIS SMTP Server SmtpOutboundCommandSupportOptions Bug

The Simple Mail Transport Protocol (or SMTP for short) Server has been a part of Internet Information Services (IIS) since the days of Windows NT Server.  Over the years, little about it has changed, nor is any big revamp expected by Microsoft.  This is underscored by the fact that the SMTP Server was not made a part of IIS 7, instead remaining available in Windows Server 2008 has an IIS 6 legacy feature.  And for the most part, this is fine.  Although totally useless as an inbound mail server (except for Windows Server 2003, when Microsoft added a matching POP3 server), it excels in one area that keeps it firmly a part of the Windows Server feature set, even though IIS has moved on to be fully integrated with the Microsoft .NET framework.  It's a capable relay for outbound email create by web applications.

The SMTP Server is a smart mail server, meaning that it can deliver messages directly to the recipient's mail server.  Which allows web applications to send out email directly from the server without having to relay through an intermediate mail system.  (Of course, there is a trick to this, and I'll cover the proper configuration the IIS SMTP Server in a future post.)  But while the SMTP Server hasn't changed much over the years, how mail servers communicate with other did when Enhanced SMTP (ESMTP) was introduced to extend the SMTP command set.  It has eclipsed the older SMTP command set and ESMTP is now used by the vast majority of all mail servers.  In fact, the large public email services like AOL, Gmail, Hotmail (or, and Yahoo won't even talk to a mail server that doesn't speak ESMTP.

And here lies a big problem for most IIS SMTP Servers.  Although the SMTP Server supports ESMTP, it is disabled by default!  Thus when the SMTP Server tries to deliver email to recipients at one of the large public email services, the connection is simply closed without any failure notice.  The SMTP Server keeps trying to deliver the email, each time being rebuffed by the large public email services, until it finally gives up and moves the message to the Badmail folder.

When I encountered this a couple of years back, I was trying to figure out why email relayed through the SMTP Server wasn't getting to Gmail and Yahoo recipients.  Mail was being delivered to other mail systems just fine, just not the big players, which represented about half of the intended recipients.  I finally stumbled upon an obscure TechNet article that partially (partially!) documented the IIS metabase setting called SmtpOutboundCommandSupportOptions.  This a bit-mapped value, and by setting or clearing various bits, you can turn off or on support for various ESMTP commands when making outbound connections.  (There is a similar setting called SmtpInboundCommandSupportOptions which does the same thing for inbound connections.)

According to the documentation, the value of the SmtpOutboundCommandSupportOptions setting defaults to 7697601, which turns on support for all the ESMTP commands for outbound connections.  But the default value as listed in the official Microsoft documentation is totally incorrect.  The real default value is 7!  Which means that ESMTP support for outgoing connections is disabled, so trying to communicate with the large public email services is a completely futile exercise.  By when I changed the setting to officially document default value of 7697601, the SMTP Server was now able to deliver email to the large public email services without problems, and the outbound mail queue was suddenly empty.

So how do you change the value of the SmtpOutboundCommandSupportOptions setting?  It's actually very simple.  It can done using the ADSUTIL.VBS script or by editing the metabase directly.

Using The ADSUTIL.VBS Script

ADSUTIL.VBS is a VBscript utility created by Microsoft for working with IIS metabase.  It must be run from a Command-Line Prompt windows using CSCRIPT.  It is normally find in the folder C:\inetpub\AdminScripts\, and on Windows Server 2008, requires that the "IIS 6 Scripting Tools" feature be installed.  Some hosting companies move this to a different folder in their Windows Server builds, which was the case for Windows Server 2003 at Peer 1, where it's locating the D:\AdminScripts folder (we stopped moving it in Windows Server 2008).  Once you've located the ADSUTIL.VBS script, open a Command Prompt windows, change to the directory where the script is located, and run the following command:

cscript adsutil.vbs SET SmtpSvc/SmtpOutboundCommandSupportOptions 7697601

which will return the following message:

SmtpOutboundCommandSupportOptions: (Long) 7697601

In order for the change to take effect, you need to restart the "Simple Mail Transport Protocol" service.

Editing The IIS Metabase

Should the ADSUTIL.VBS script not be available, you can edit the IIS metabase directly using Notepad.  NOTE: You will need to stop IIS in order to save the changes made in Notepad.

  1. Stop the IIS services using the command:

  2. Open the file C:\WINDOWS\system32\inetsrv\MetaBase.xml in Notepad.
  3. Under the section IIsSmtpService Location = "/LM/SmtpSvc", find the property SmtpOutboundCommandSupportOptions. Since this property appears only once in the metabase, the quickest way to locate it is Notepad's Find utility (Edit –> Find).
  4. Change the property so it reads:

  5. Save the file.
  6. Restart the IIS services using the command:


And that's all there is to it.

Posted in Windows.
All information in this blog is provided "AS IS" with no warranties and confers no rights.
The opinions expressed in this blog are mine alone and do not represent those of my employer.
Powered By PEER 1 Managed Hosting