Inside the Native API

Introduction

Virtually everybody familiar with NT has at the minimum heard that there is a hidden API that NT uses internally. This API, which is called the Native API, is almost entirely hidden from view, with only a handful of its function documented in generally accessible publications. This obfuscation has lead to a general belief that the Native API can provide applications magical powers, perhaps even allowing them to bypass security measures implemented by standard APIs like Win32. Thoughts along these lines usually lead to the Native API conspiracy theory: Microsoft is keeping the API for themselves and their own application to unfair advantage. The native API does expose some nuances not available through documented APIs (for example, you can specify whether or not file opens should be case sensitive, something not possible with Win32's CreateFile() or OpenFile()), however the majority of the APIs capabilities are accessible through documented channels.

This article will introduce you to the Native API and provide you a roadmap for what is in the API. I'll first describe what the Native API is, how it's invoked in normal operation, and how its used as a support infrastructure for the APIs of NT's operating environment subsystems. Then I'll take you on a tour of the API where I break it down into sets of related functions (memory management, synchronization, etc.). I'll talk about the capabilities available through the API's functions and note Win32 APIs that map to particular Native APIs where applicable. This comprehensive look at the Native API should help clarify misconceptions about how it's used, why it's used, and what the undocumented APIs are hiding from us (e.g. whether the conspiracy theory has validity).

Native API Architecture

The Windows NT Native API serves one purpose: as a means for calling operating system services located in kernel mode in a controlled manner. Kernel mode is where the core of NT executes, and it's in kernel mode that components have direct access to hardware and services that perform management of the computer's resources including memory, devices and processes. Thus, whenever a program executing in user mode wants to perform I/O, allocate or deallocate virtual memory, start a thread or process, or interact with global resources, it must call upon one or more services that live in kernel mode.

The Native API is equivalent to the system call interface on traditional monolithic operating systems such as most UNIXes. On most UNIXes, however, the system call interface is well documented and is generally available for use by standard applications. For example, the read() call for reading data from a file, socket, or input device in most flavors of UNIX is a system call that is handled by code in kernel mode. In Windows NT the Native API, its system call interface, is hidden from programmers behind higher level APIs such as Win32, OS/2, POSIX or DOS/Win16. The reason behind this is NT's architecture.

NT is a "modified microkernel" architecture. Instead of supporting one basic operating system API, NT implements several. It does this efficiently by implementing operating environment subsystems in user mode that export particular APIs to client programs. The "national language" API of NT is Win32, and the Win32 architecture demonstrates this concept. The Win32 operating environment subsystem is divided among a server process, CSRSS.EXE (Client/Server Runtime SubSystem), and client-side DLLs that are linked with programs that use the Win32 API. The core of the Win32 API is divided into three categories: windowing and messaging, drawing, and base services. Windows and messaging APIs include CreateWindow() and SendMessage(), and are exported to Win32 programs via the USER32.DLL library. BitBlt() and LineTo() are Win32 drawing functions and are provided in GDI32.DLL. Finally, base services include all Win32 I/O, process and thread, memory management, and synchronization APIs, and KERNEL32.DLL is the library that exports them.

When a Win32 program calls a Win32 API control is transferred within its address space into one of Win32's client-side DLLs. The DLL can execute one or more of the following options:

• return immediately to the caller
• send a message to the Win32 server to request help
• invoke Native APIs to carry out the function

The first option of returning to the caller is rarely possible and is only possible when the DLL can service the function without the help of operating system services. An example of this is GetCurrentProcess(). This API simply returns a handle to the current process that is cached in KERNEL32 when the process is started.

The second option is also rarely required. A client-side DLL only needs to send messages to the Win32 server when the server must participate with, and be aware of, the function's execution. The Win32 server creates a Win32 execution environment for its clients that involves maintaining some state associated with its client processes. Thus, the CreateProcess() API, exported by KERNEL32, requires an interaction with the Win32 server. The server in this case prepares a new process for execution by mapping in an executable image, creating a command-line argument structure, and so on. The Win32 server calls Native API functions to create the actual process image and prepare its address map.

The final option is the most frequently exercised. Let's talk about USER32 and GDI32 APIs first, before talking about KERNEL32's use of Native APIs. In versions of NT prior to 4.0, windowing and drawing functions were located in the Win32 server (CSRSS.EXE). This meant that whenever an application used these function a message would be sent to the server. In NT 4.0 the windowing and drawing components of Win32 were moved into a kernel mode component named WIN32K.SYS. Instead of sending a message to the server, the client-side DLLs just call directly into the kernel, saving the overhead of messaging and context switching to another process. This has enhanced NT's graphics performance (as evidenced by the Pinball sample game). GDI and USER functions have become NT's second Native API, but they are less mysterious than the primary Native API since drawing, windowing, and messaging APIs are well-documented.

KERNEL32 functions that call the Native API directly include all of its I/O (e.g CreateFile(), ReadFile(), WriteFile()), synchronization (e.g. WaitForSingleObject(), SetEvent()), and memory management (e.g. VirtualAlloc(), VirtualProtect()) functions. In fact, the majority of KERNEL32's exported routines use the Native API directly. The figure below shows the flow of control from a Win32 application executing a Win32 call (CreateFile()), through KERNEL32, NTDLL, and into kernel mode where control is transferred to the NtCreateFile system service. I'll talk about this process in detail.

NTDLL.DLL

The Native API is provided to user-mode programs by the NTDLL.DLL library. NTDLL.DLL, besides containing Native API user-mode entry points, has process startup and module loading code in it. The bulk of it, though, are the Native API stubs that transfer control to kernel mode. This is accomplished by executing a software exception. If you look at a stub for a Native API in NTDLL inside of a debugger you'll see something like this (on x86):

NtCreateFile:
mov
eax, 0x0000001A
lea
edx, [esp+04]
int
0x2E
ret
0x2C

In this case the call is NtCreateFile but every other native call looks almost identical. The first instruction is loading a register with the Native API's index number. Every Native API has a unique index number, which is generated automatically by a script that runs as part of the NT build process. Thus, the index number for a specific function can vary from build to build as Native APIs are added and removed. The second instruction loads a register with a pointer to the call's parameters. Next is the software exception instruction. NT registers a kernel-mode exception handler specifically for handling Native API software exceptions. On x86's this exception is 0x2E. The final instruction pops the parameters off the caller's stack. Note that on Windows 2000 and higher Windows uses special processor instructions when available instead of INT 0x2E. On Intel systems it uses SYSENTER and on AMD systems it uses SYSCALL.

Note that all of the Native APIs begin with "Nt". The export table in NTDLL.DLL also makes the Native API accessible through an alternate naming convention, one where command names begin with "Zw" instead of "Nt". Thus, ZwCreateFile() is an alias for NtCreateFile().

The Native API exception handler in kernel mode is named KiSystemService, and it is invoked whenever a Native API is executed in user mode. Its task is to determine if the API's index number is valid, and if so, pass control to the appropriate system service in kernel mode to service the request. It does this by simply using the index number passed from user mode to index into an array called KiSystemServiceTable. Each entry in this array includes a pointer to the appropriate function and the number of parameters the function expects. KiSystemService takes the parameters passed on the user mode stack (pointed to in the edx register on x86) and pushes them on the kernel stack before calling the function specified in the array for the index.

Win32 Native APIs introduced in NT 4.0 are handled by the same exception handler, but the index numbers of Win32 functions specify that a second array of system service pointers should be used. The function pointers in the second array reference functions in WIN32K.SYS.

Each system service performs operations specific to the API they implement, of course, but most of them must deal with the validation of the parameters passed to them from user mode. Many parameters are pointers, and dereferencing an invalid pointer in kernel mode without taking precautionary measures can prove catastrophic. Validating parameters is straight-forward, but the number of Native APIs and the number of parameters they take have made getting it right tough for Microsoft. In 1997 I wrote a program called NTCrash that barraged the Native API interface with garbage parameters. The program discovered 13 WIN32K system services that failed to perform comprehensive parameter validation, the result of which were Blue Screens. Microsoft closed these holes in Service Pack 1.

About year later I revisited NTCrash and tweaked it to be more intelligent about generating garbage - the garbage this new version, NTCrash2, produces hits boundary conditions that can be easy to miss in validation. In fact, this revision found 40 more APIs (15 Native APIs and 25 WIN32K Native APIs) with Blue Screen holes. Microsoft has been made aware of the holes and they will be closed in Service Pack 4. Testers within Microsoft have told me that they use a greatly enhanced version of my NTCrash to test system call interfaces.

After parameter validation, system services usually call functions supplied by NT's Executive subsystems. These subsystems all live in kernel mode, and each is responsible for managing certain resources. Example subsystems include the Process Manager, Virtual Memory Manager, I/O Manager, and Local Procedure Call facility.

Native API Catalog

There are about 240 Native APIs in Win2K, up from around 200 in NT 3.5.1. In this section I break the Native API down into categories of related functions and I start each category by briefly describing the capabilities of the group. For each function I list Win32 functions that obtain functionality similar to that provided by the Native API (which they usually do by actually calling upon the Native API).

Currently, the only documentation on Native APIs is located in the Windows NT Device Driver Kit (DDK) and the Windows NT Installable File System Kit (IFS Kit). The DDK actually describes the parameters and usage of a around 25 Native APIs, and includes prototype and parameter information for a few others in NTDDK.H (e.g. NtQueryInformationProcess(). The IFS Kit documents about 25 more APIS only by providing prototypes in header files that come as part of the kit, and sometimes through their use in sample code. Most of APIs included in the IFS Kit are in the file I/O and security categories. The IFS Kit is available from Microsoft for a fee after signing an NDA (see Microsoft's Web site for information). You can find prototypes and some minimal documentation for many Native APIs in the third-party book Windows NT/2000 Native API Reference.

The format of the following tables is as follows: Column 1 shows the native API, column 2 shows Win32 functions that map to the API, column 3 provides a brief description of the APIs functionality, and column 4 shows if its documented in the DDK or not (either explicitly or by its prototype). Functions new to Win2K are shown in red, those new to XP in blue, and those new to Server 2003 in green.

File Maps

Back to Top