FILE: Creating a Thread Pool for MFC-Based ISAPI Extensions

ID: Q197728

The information in this article applies to:

Microsoft Internet Server Application Programming Interface (API)
Microsoft Visual Studio versions 6.0, 97

SUMMARY

Afxpool.exe is a self-extracting executable that contains the necessary files for a Microsoft Visual C++ 6.0 project that demonstrates the technique of creating a thread pool for an MFC-based ISAPI extension.

This article explains the steps to create a worker thread pool within an ISAPI extension. Often when an ISAPI extension requires a significant amount of processing time to complete a request, it is necessary to create a separate thread of control to handle the processing. For additional information, please see the following article in the Microsoft Knowledge Base:

   ARTICLE-ID: Q185518
   TITLE     : HOWTO: Implement Worker Threads Using MFC ISAPI Extension

However, creating a new thread of control for each new request can cause a significant performance impact on the server. For this reason, it's advantageous to create a pool of worker threads from which a thread can be "recycled." The affect of this is that the cost of creating a new thread is removed from the cost of handling a new request.

The ISAPI extension will incur the cost of creating the threads when it first initializes but for each request it will merely request an available worker thread from the pool.

MORE INFORMATION

The following file is available for download from the Microsoft Software Library:

 ~ Afxpool.exe (size: 57844 bytes)

Release Date: DEC-07-1998

For more information about downloading files from the Microsoft Software Library, please see the following article in the Microsoft Knowledge Base:

   ARTICLE-ID: Q119591
   TITLE : How to Obtain Microsoft Support Files from Online Services

   FileName               Size
   ---------------------------------------------------------

   ExtThreadPool.aps       18,436
   ExtThreadPool.clw       420
   ExtThreadPool.cpp       6,184
   ExtThreadPool.def       173
   ExtThreadPool.dsp       4,332
   ExtThreadPool.dsw       549
   ExtThreadPool.h         1,560
   ExtThreadPool.ncb       66,560
   ExtThreadPool.opt       54,784
   ExtThreadPool.plg       723
   ExtThreadPool.rc        3,242
   ExtThreadPool.rc2       405
   Resource.h              322
   StdAfx.cpp              215
   StdAfx.h                669
   ThreadPool.cpp          3,130
   ThreadPool.h            863

The thread pool uses completion ports available under the Windows NT and Windows 2000 family of operating systems. The basic idea behind this thread pool is to create a single completion port, have all of the workers wait on this completion port, and when a new request comes in, post the request to this completion port. The completion port mechanism ensures that only one thread of control will be notified of this new request.

It possible to create and manage the thread pool without using completion ports with other synchronization objects, such as semaphores. However with completion ports the operating system takes a burden of scheduling and releasing threads in the most efficient manner (that is, Last In First Out).

In a typical ISAPI extension, there are two virtual functions created by the ISAPI MFC Wizard: GetExtensionVersion and TerminateExtension. The initialization of the completion port is accomplished in the GetExtensionVersion with the call to CreateIoCompletionPort. Note that the last parameter to this function allows you to specify how many threads from the pool can run concurrently. The most efficient way is to have as many running threads as CPUs. Last parameter 0 will allow the operating system to pick the number of threads matching the number of CPUs.

   BOOL CExtThreadPoolExtension::GetExtensionVersion(
      HSE_VERSION_INFO* pVer)
   {
      // Call default implementation for initialization
      CHttpServer::GetExtensionVersion(pVer);

      // Load description string
      TCHAR sz[HSE_MAX_EXT_DLL_NAME_LEN+1];
      ISAPIVERIFY(::LoadString(AfxGetResourceHandle(),
            IDS_SERVER, sz, HSE_MAX_EXT_DLL_NAME_LEN));
      _tcscpy(pVer->lpszExtensionDesc, sz);

      m_phThreads = new HANDLE[THREAD_COUNT];

      m_nThreadCount = THREAD_COUNT;
      m_hIoPort = CreateIoCompletionPort((HANDLE)INVALID_HANDLE_VALUE,
         NULL, 0, 0);

      // NOTE:  If a CWinApp object is not declare, this call should be
      // changed to CreateThread() instead.
      for(long n = 0; n < m_nThreadCount; n++)
      {
         CWinThread* pWinThread = AfxBeginThread(
            (AFX_THREADPROC)ThreadProc, m_hIoPort);
         m_phThreads[n] = pWinThread->m_hThread;
      }

      return TRUE;
   }

Here, you have three variables defined in the CExtThreadPoolExtension class: m_nThreadCount, m_hIoPort, and m_phThreads. m_nThreadCount is a type long which defines the number of threads to create within our thread pool (this value is hard coded in the ThreadPool.h file to the THREAD_COUNT variable). The m_hIoPort is the single completion port handle you will send to all of the worker threads to block on. And m_phThreads is the array of thread handles from the worker threads.

It should be noted that this sample is built using MFC in a shared library. If you create an MFC ISAPI Extension with the wizard as statically linked library, the instance of CWinApp won't be available. Because of this, the AfxBeginThread() call should be changed to CreateThread() instead.

Although the MFC ISAPI Wizard creates a GetExtensionVersion and TerminateExtension method, it is necessary to manually override the HttpExtensionProc() method of the CHttpServer as well. This can be accomplished by adding the method in the CExtThreadPoolExtension declaration:

   class CExtThreadPoolExtension : public CHttpServer
   {
   public:
      CExtThreadPoolExtension();
      ~CExtThreadPoolExtension();

   // Overrides
      // ClassWizard generated virtual function overrides
         // NOTE - the ClassWizard will add and remove member
         // functions here.
         //    DO NOT EDIT what you see in these blocks of generated code !
      //{{AFX_VIRTUAL(CExtThreadPoolExtension)
      public:
      virtual BOOL GetExtensionVersion(HSE_VERSION_INFO* pVer);
      virtual DWORD HttpExtensionProc(EXTENSION_CONTROL_BLOCK *pECB);

      //}}AFX_VIRTUAL
      virtual BOOL TerminateExtension(DWORD dwFlags);

      // TODO: Add handlers for your commands here.
      // For example:

      void Default(CHttpServerContext* pCtxt);

      DECLARE_PARSE_MAP()

      //{{AFX_MSG(CExtThreadPoolExtension)
      //}}AFX_MSG

   protected:
      HANDLE*           m_phThreads;
      HANDLE            m_hIoPort;
      long              m_nThreadCount;
   };

This is necessary because you must override the return value from the CHttpServer::HttpExtensionProc() to let IIS know that you're still processing this request, do not recycle the EXTENSION_CONTROL_BLOCK.

The overridden HttpExtensionProc is fairly simple:

   DWORD CExtThreadPoolExtension::HttpExtensionProc(
      EXTENSION_CONTROL_BLOCK *pECB)
   {
      // The result of the base class's (CHttpServer) HttpExtensionProc()
      // must be checked.  If it's successful, you will need to return a
      // HSE_STATUS_PENDING.  This is so that IIS will not shutdown the
      // connection and not destroy the ECB.
      // If the result of the base class is not successful, you should
      // simply propagate the error message up to IIS.

      DWORD  dwResult = CHttpServer::HttpExtensionProc(pECB);
      if(dwResult == HSE_STATUS_SUCCESS)
        return HSE_STATUS_PENDING;
      else
        return dwResult;
   }

You merely check the return call from the CHttpServer::HttpExtensionProc() and return pending if successful. Otherwise, just propagate the error up.

In the Default() implementation (or whatever routine your MFC ISAPI Extension calls in your message map), take the EXTENSION_CONTROL_BLOCK (note this is not the same as the CHttpServerContext object; you can't pass the CHttpServerContext object to the worker thread since that object's existence is only within the scope of the CHttpServer::HttpExtensionProc()) and post it to the completion port you created in GetExtensionVersion():

   void CExtThreadPoolExtension::Default(CHttpServerContext* pCtxt)
   {
      if(!PostQueuedCompletionStatus(m_hIoPort,
         (DWORD) pCtxt->m_pECB, 0, NULL))
      {
         char  szBuffer[4096] = { 0 };

         sprintf(szBuffer, "<H1>Error posting to completion port!
            Win32 Error = %i</H1>", GetLastError());
         DWORD dwBufLen = strlen(szBuffer);
         pCtxt->m_pECB->WriteClient(pECB->ConnID, szBuffer,
            &dwBufLen, 0);
      }
   }

If the completion port post fails, you need to generate an error message and report it back to the client. Please note that to output the error WriteClient callback directly from the ECB instead of the MFC wrapper is used.

This sample also demonstrates a way to pass various parameters to the worker thread. In addition to Default(), you also have ProcessName() and ProcessSample(). ProcessName()takes as its parameters three strings: First name, Last name, and Middle name. ProcessSample takes two parameters: Data and String. Data will be a type of integer and String is any string data.

The idea behind this is to take advantage of the parse map support from MFC ISAPI extension and allow for the worker threads to handle the appropriate requests. To accomplish this, you need to create two structures:

   struct ProcessNameData
   {
      EXTENSION_CONTROL_BLOCK*   pECB;
      CString                    sFirstName;
      CString                    sLastName;
      CString                    sMiddleName;
   };

   struct ProcessSampleData
   {
      EXTENSION_CONTROL_BLOCK*   pECB;
      DWORD                      nValue;
      CString                    sString;
   };

ProcessNameData structure will be used to temporarily hold the parameters passed to ProcessName() function by the MFC parse map. Within the ProcessName() function, you dynamically allocate a new ProcessNameData structure, set the EXTENSION_CONTROL_BLOCK and the parameters passed to us and pass it off to the worker thread. But when you pass the data off to the worker thread, you have to let the worker thread know that the data you are passing is of type ProcessNameData. This is accomplished by third parameter in the PostQueuedCompletionStatus() API. In the Default() function, you passed 0 as the third parameter. For ProcessName(), you're passing 1:

   void CExtThreadPoolExtension::ProcessName(CHttpServerContext* pCtxt,
                                             LPCTSTR szFirstName,
                                             LPCTSTR szLastName,
                                             LPCTSTR szMiddleName)
   {
      ProcessNameData*  pData = new ProcessNameData;

      pData->pECB = pCtxt->m_pECB;
      pData->sFirstName = szFirstName;
      pData->sLastName = szLastName;
      pData->sMiddleName = szMiddleName;

      if(!PostQueuedCompletionStatus(m_hIoPort, (DWORD) pData,
         (DWORD) 1, NULL))
      {
         char  szBuffer[4096] = { 0 };

         sprintf(szBuffer,
            "<H1>Error posting to completion port! Win32 Error = %i</H1>",
            GetLastError());
         DWORD dwBufLen = strlen(szBuffer);
         pCtxt->m_pECB->WriteClient(pCtxt->m_pECB->ConnID, szBuffer,
            &dwBufLen, 0);
      }
   }

By passing a value of 1 to the worker thread, the worker thread will know that it needs to process the data differently than from the Default() routine.

Similarly, you pass a value of 2 for the ProcessSample() function. And in this function, you would create a new instance of ProcessSampleData structure.

Within the ThreadProc (the actual routine that the worker thread executes), you need to have all of the threads block on a single completion port (which is passed to the thread routine from GetExtensionVersion). This is done when the thread calls GetQueuedCompletionStatus(). If no status is posted, the call will block until one is posted. When the completion port is posted, one of the worker threads will "awaken" and begin processing:

   unsigned int ThreadProc(void* p)
   {
      HANDLE                        IoPort   = (HANDLE*)p;
      unsigned long                 pN1, pN2;
      OVERLAPPED*                   pOverLapped;

      while(GetQueuedCompletionStatus(IoPort, &pN1, &pN2,
         &pOverLapped, INFINITE))
      {
         if(pOverLapped == (OVERLAPPED*)0xFFFFFFFF)
            break;
         else
         {
            switch(pN2)
            {
            case 0:
               DoWork((EXTENSION_CONTROL_BLOCK*)pN1);
               break;
            case 1:
               DoWork((ProcessNameData*)pN1);
               break;
            case 2:
               DoWork((ProcessSampleData*)pN1);
               break;
            default:
               // Huh?!?!?
               break;
            }
         }
      }

      return 0;
   }

As you can see, in the switch() statement, you test for the correct pN2 value (this is the third parameter you passed in the PostQueuedCompletionStatus() call). It should also be pointed out that there are three versions of DoWork(). One version will take a EXTENSION_CONTROL_BLOCK pointer, another takes a ProcessNameData pointer, and a third takes ProcessSampleData pointer. By explicitly casting pN1, you can make sure you are calling the correct version for the data provided.

It is necessary for the version of DoWork(), which takes the ProcessNameData and ProcessSampleData, to destroy those structure before exiting. This is because those structures were created for temporarily holding the parse map parameters:

   void DoWork(ProcessNameData* pData)
   {
      EXTENSION_CONTROL_BLOCK*   pECB = pData->pECB;
      CString  sBody;

      // Build the HTML body...
      sBody.Format(
         "<H1>This is thread %i in thread pool.  Hi there!</H1>",
         GetCurrentThreadId());
      sBody += "<HR><H2>I'm processing this for ";
      sBody += pData->sFirstName;
      sBody += " ";
      if(pData->sMiddleName.CompareNoCase("~") != 0)
      {
         sBody += pData->sMiddleName;
         sBody += " ";
      }
      sBody += pData->sLastName;
      sBody += "</H2>";

      // Pretend we're spending some time processing this request
      // this sample assumes 3 seconds.
      Sleep( 3000 );

      SendData(pECB, sBody);

      delete pData;
   }

   void DoWork(ProcessSampleData* pData)
   {
      EXTENSION_CONTROL_BLOCK*   pECB = pData->pECB;
      CString  sBody;

      // Build the HTML body...
      sBody.Format(
         "<H1>This is thread %i in thread pool.  Hi there!</H1>",
         GetCurrentThreadId());
      sBody += "<HR><H2>This is the sample data: ";
      sBody += pData->sString;

      CString  sTmp;
      sTmp.Format(" (%i)</H2>", pData->nValue);

      sBody += sTmp;

      // Pretend we're spending some time processing this request
      // this sample assumes 3 seconds.
      Sleep( 3000 );

      SendData(pECB, sBody);

      delete pData;
   }

SendData() is simple function that sends back a CString to the client.

However, before you start processing, you need to make sure the thread isn't being signaled to terminate. This is done by testing the value of the pOverLapped pointer. If the overlapped pointer's value is 0xFFFFFFFF (an invalid handle value), you know that the thread is suppose to terminate. So the routine will break out of the while loop.

To signal the worker threads to terminate, use the third virtual function created by the MFC ISAPI Wizard: TerminateExtension(). In this function, you need to signal all of the threads to close and close out our completion port as well:

   BOOL CExtThreadPoolExtension::TerminateExtension(DWORD dwFlags)
   {
      // extension is being terminated
      for(long n = 0; n < m_nThreadCount; n++)
         PostQueuedCompletionStatus(m_hIoPort, 0,
            0, (OVERLAPPED*)0xFFFFFFFF);

      WaitForMultipleObjects(m_nThreadCount, m_phThreads, TRUE, 120000);
      CloseHandle(m_hIoPort);
      delete [] m_phThreads;
      return TRUE;
   }

To signal all of the threads to close, post with an invalid handle value an overlapped pointer. Do this for the number of threads available. This ensures that each thread receives the notification. Once that's done, wait for all of the threads to exit by calling WaitForMultipleObjects() call on the handles returned from the AfxBeginThread() call (the handles were copied from the CWinThread object in GetExtensionVersion). In this sample, you will wait two minutes (120,000 milliseconds) for all of the threads to exit.

This will ensure that when you terminate the ISAPI extension, you are cleaning up before you map the DLL out of memory.

NOTE: All threads created do not run in the same security context as that of the ISAPI extension thread. This is because the security context of the ISAPI extension thread is valid only for that thread. To pass the security context over, you will need to pass the security token along with the parse map parameters. For example, if you need to pass the thread security token to the worker for ProcessName(), you would need to modify the ProcessNameData structure to:

   struct ProcessNameData
   {
      EXTENSION_CONTROL_BLOCK*   pECB;
      CString                    sFirstName;
      CString                    sLastName;
      CString                    sMiddleName;
      HANDLE                     hThreadToken;
   };

And the ProcessName() function needs to be changed to the following:

   void CExtThreadPoolExtension::ProcessName(CHttpServerContext* pCtxt,
                                             LPCTSTR szFirstName,
                                             LPCTSTR szLastName,
                                             LPCTSTR szMiddleName)
   {
      ProcessNameData*  pData = new ProcessNameData;

      OpenThreadToken(GetCurrentThread(), TOKEN_ALL_ACCESS, FALSE,
         &(pData->hThreadToken));
      pData->pECB = pCtxt->m_pECB;
      pData->sFirstName = szFirstName;
      pData->sLastName = szLastName;
      pData->sMiddleName = szMiddleName;

      if(!PostQueuedCompletionStatus(m_hIoPort, (DWORD) pData,
         (DWORD) 1, NULL))
      {
         char  szBuffer[4096] = { 0 };

         sprintf(szBuffer,
            "<H1>Error posting to completion port! Win32 Error = %i</H1>",
            GetLastError());
         DWORD dwBufLen = strlen(szBuffer);
         pCtxt->m_pECB->WriteClient(pCtxt->m_pECB->ConnID, szBuffer,
            &dwBufLen, 0);
      }
   }

Simiarly, in the DoWork(), we need to impersonate the thread context:

   void DoWork(ProcessNameData* pData)
   {
      ImpersonateLoggedOnUser(pData->hThreadToken);

      EXTENSION_CONTROL_BLOCK*   pECB = pData->pECB;
      CString  sBody;

      // Build the HTML body...
      sBody.Format(
         "<H1>This is thread %i in thread pool.  Hi there!</H1>",
         GetCurrentThreadId());
      sBody += "<HR><H2>I'm processing this for ";
      sBody += pData->sFirstName;
      sBody += " ";
      if(pData->sMiddleName.CompareNoCase("~") != 0)
      {
         sBody += pData->sMiddleName;
         sBody += " ";
      }
      sBody += pData->sLastName;
      sBody += "</H2>";

      // Pretend we're spending some time processing this request
      // this sample assumes 3 seconds.
      Sleep( 3000 );

      SendData(pECB, sBody);

      delete pData;
      CloseHandle (pData->hThreadToken);

   }

The included sample does not implement this. The sample's worker threads will all run in the context of the process. If the ISAPI extension is executed in process, it will run as Local System. If the ISAPI extension is executed out of process, it will run as the IWAM_<MachineName> user.

REFERENCES

For additional information, please see the following article in the Microsoft Knowledge Base:

   ARTICLE-ID: Q185518
   TITLE     : HOWTO: Implement Worker Threads Using MFC ISAPI Extension

Microsoft Visual C++ Win32 API Online Documentation

Additional query words:

Keywords          : kbnokeyword
Version           : WINDOWS:6.0,97; WINNT:
Platform          : WINDOWS winnt

Last Reviewed: December 21, 1998