Is the IOCP listener actually listening?

Microsoft

Feb 11, 2019

First published on MSDN on Dec 12, 2012
What is the IOCP Listener?

The IOCP listener is a dedicated system thread that accepts incoming connections to SQL Server. It is a SQL Network Interface (SNI)-layer thread that listens for TCP/IP sockets or Named Pipes traffic (VIA is going away but still in the code). When a connection comes in, the listener accepts it and quickly hands it off to a SQL worker thread to process further. Think of the IOCP listener as the gate keeper – any new incoming connections go through it. Strictly-speaking, the IOCP listener is the one that accepts incoming TDS packets and determines what type of task needs to be created in SQL Server (see this post for types of tasks).

A frequent confusion introduced by its name is that the IOCP listener handles disk I/O. However, its purpose in life is to handle network I/O in SQL Server, and not disk I/O. The name IOCP stands for – I/O Completion Port and indeed it does use I/O Completion ports . But I/O completion ports can be used not only for asynchronous disk I/O but also for all type of other asynchronous I/O. As the note in the I/O Completion ports article states:

“The term file handle as used here refers to a system abstraction representing an overlapped I/O endpoint, not only a file on disk. For example, it can be a network endpoint, TCP socket, named pipe, or mail slot. Any system object that supports overlapped I/O can be used . For a list of related I/O functions, see the end of this topic.”

So in summary, in SQL engine, we use the IOCP thread to handle incoming TCP socket and Named pipe traffic.

Non-Yielding (Stalled) IOCP Listener

If the IOCP listener thread gets stuck - you will frequently associate this with a non-yielding IOCP – then no incoming connections can be accepted. SQL Server appears effectively unresponsive to the outside world, even though existing queries can be running just fine. That’s why the Scheduler Monitor thread (another system thread that checks SOS scheduler health) keeps a close watch on the IOCP listener and if it gets stuck for 15 seconds, it generates an error in the Errorlog and creates a memory dump. As a reference, the Scheduler Monitor creates a memory dump after 60-70 seconds for other scheduler issues. As you can see SQL is more aggressively monitoring the IOCP listener thread because it is a critical one in the system.

IOCP Listener and NUMA

There is one IOCP thread per server, if no NUMA is present. If NUMA is present, SQL Server creates one per NUMA node. Here is what mine looks like currently on an idle SQL Server.

Child-SP RetAddr Call Site

00000000`0959f238 000007fe`fd4c169d ntdll!ZwRemoveIoCompletion+0xa

00000000`0959f240 00000000`76b9a4e1 KERNELBASE!GetQueuedCompletionStatus+0x39

00000000`0959f2a0 00000000`005a1700 kernel32!GetQueuedCompletionStatusStub+0x11

00000000`0959f2e0 00000000`0055b450 sqlservr!SOS_Node::ListenOnIOCompletionPort +0x11d

00000000`0959f470 00000000`0055b116 sqlservr!SOS_Task::Param::Execute+0x12a

00000000`0959f580 00000000`0055af5b sqlservr!SOS_Scheduler::RunTask+0x96

00000000`0959f5e0 00000000`006944fa sqlservr!SOS_Scheduler::ProcessTasks+0x128

00000000`0959f650 00000000`006947dd sqlservr!SchedulerManager::WorkerEntryPoint+0x2d2

00000000`0959f730 00000000`00adc0cd sqlservr!SystemThread::RunWorker+0xcc

00000000`0959f770 00000000`006953d2 sqlservr!SystemThreadDispatcher::ProcessWorker+0x2db

00000000`0959f820 00000000`719737d7 sqlservr!SchedulerManager::ThreadEntryPoint+0x173

00000000`0959f8c0 00000000`71973894 MSVCR80!_callthreadstartex+0x17

00000000`0959f8f0 00000000`76ba652d MSVCR80!_threadstartex+0x84

00000000`0959f920 00000000`76f3c521 kernel32!BaseThreadInitThunk+0xd

00000000`0959f950 00000000`00000000 ntdll!RtlUserThreadStart+0x1d

Debug Scenarios

Here is a IOCP scenario I found on MSDN blogs. BTW, for that particular blog, NumaNode::RampUp is a very specific stage in the BPool ramp up described here and I think was the root cause that scenario.