The ETW Sensor

What we covered before

In the previous post we saw how NewGen-AV starts.

The Launcher starts the CoreEngine, the CoreEngine initializes the components, starts the event processing thread and then starts the EtwSensor.

Now we can look at the sensor itself.

Building the First Sensor

This is the first real source of telemetry for the AV.

For this MVP, NewGen-AV does not have kernel callbacks, API hooks, file system minifilters, network sensors, or any fancy endpoint visibility layer. It has ETW.

ETW stands for Event Tracing for Windows. In simple terms, it is a Windows tracing system that lets us subscribe to events produced by the operating system and by providers running on the machine.

For our AV, ETW is useful because it gives us a way to observe things like process creation and image loading without injecting into every process and without writing a driver yet.

But there is an important limitation from the start.

ETW is telemetry.

It is not prevention.

It tells us that something happened. It does not stop that thing before it happens.

That means this design is good enough for a first MVP, but it can lose races. A malicious process may start running before we receive the event, normalize it, scan the file and decide to kill the process.

Why sensors should stay decoupled from detection logic

One important architectural decision is that sensors should not contain detection logic.

A sensor should observe.

The engine should decide.

The EtwSensor should not know whether a process is malicious or what should happen when something suspicious is detected.

Its job is only to collect telemetry from ETW, convert it into an internal event format, and send that event to the engine.

In this MVP, the sensor detects two types of events: ProcessCreate and ImageLoad.

This gives us a cleaner flow:

ETW event -> EtwSensor -> CoreEngine

This separation matters because today we only have one sensor: ETW.

Tomorrow, we may add kernel callbacks, userland hooks, network telemetry and maybe file system monitoring too.

If each sensor starts implementing its own detection logic, the project becomes hard to reason about very quickly.

We decouple the components through the callback function and the NormalizedEvent structure.

The EtwSensor does not know what happens after calling the callback.

The CoreEngine does not need to know how the sensor collected or normalized the event.

As long as the normalized event contract stays stable, we can change one side without rewriting the other.

What the EtwSensor does

The EtwSensor has one job: listen to ETW, convert raw Windows events into NewGen-AV events, and send them to the engine.

It should not decide whether a process is malicious, run any kind of analysis, or kill processes.

Its job is to observe and report.

The simplified flow is this:

  -> Windows ETW event
  -> EtwSensor
  -> NormalizedEvent
  -> CoreEngine callback
  -> CoreEngine queue

How the EtwSensor exposes its interface

From the outside, the EtwSensor exposes a small surface:

class EtwSensor {
public:
    EtwSensor();
    ~EtwSensor();

    bool Start();
    void Stop();

    static void SetCallback(
        std::function<void(const NormalizedEvent&)> callback
    );
};

That is basically it.

The CoreEngine creates the sensor, registers a callback, and starts it.

_etwSensor = std::make_unique<EtwSensor>();

_etwSensor->SetCallback(
    [this](const NormalizedEvent& normalizedEvent) {
        this->OnEtwEventDetected(normalizedEvent);
    }
);

if (!_etwSensor->Start()) {...}

The important part is the callback.

The EtwSensor does not need to know that the engine has a queue, a mutex, a condition variable, a static analyzer or a process termination function.

It only knows that when it has a normalized event, it can call a function owned by the engine.

How the ETW session starts

The ETW sensor starts with Start().

Conceptually, the function does three things:

Start()
  -> create an ETW session
  -> enable the provider we care about
  -> start a consumer thread

The implementation starts by naming the session, allocating EVENT_TRACE_PROPERTIES, configuring real-time mode, and starting the trace.

In this MVP, we want a real-time ETW session, not a trace written to a file.

wchar_t sessionName[] = L"NewGen-AV-ETWSession";

const size_t sessionNameChars = wcslen(sessionName) + 1;
const size_t sessionNameBytes = sessionNameChars * sizeof(wchar_t);
const size_t propertiesBytes = sizeof(EVENT_TRACE_PROPERTIES);
const size_t totalBufferSize = propertiesBytes + sessionNameBytes;
const ULONG bufferSize = static_cast<ULONG>(totalBufferSize);

_sessionProperties =
    reinterpret_cast<EVENT_TRACE_PROPERTIES*>(malloc(bufferSize));

RtlSecureZeroMemory((PVOID)_sessionProperties, bufferSize);

_sessionProperties->Wnode.BufferSize = bufferSize;
_sessionProperties->Wnode.Flags = WNODE_FLAG_TRACED_GUID;
_sessionProperties->Wnode.ClientContext = 1;
_sessionProperties->LogFileMode = EVENT_TRACE_REAL_TIME_MODE;
_sessionProperties->LoggerNameOffset = sizeof(EVENT_TRACE_PROPERTIES);

ULONG status = StartTrace(
    &_sessionHandle,
    sessionName,
    _sessionProperties
);

if (status == ERROR_ALREADY_EXISTS) {
    ControlTraceW(
        0,
        sessionName,
        _sessionProperties,
        EVENT_TRACE_CONTROL_STOP
    );

    status = StartTraceW(
        &_sessionHandle,
        sessionName,
        _sessionProperties
    );
}

There is one practical case to handle here: the session may already exist, for example after a dirty previous run.

For now, the MVP handles this by stopping the existing session and starting it again.

This is not glamorous code, but it matters. Without it, a dirty previous run could prevent the sensor from starting again.

How the provider gets enabled

Starting a session is not enough.

We also need to tell Windows which provider we want to receive events from.

For this MVP, the sensor enables the Microsoft Windows Kernel Process provider. The events we will receive are detected and logged directly by the Windows Kernel.

constexpr GUID MicrosoftWindowsKernelProcessProviderGuid =
    { 0x22FB2CD6, 0x0E7B, 0x422B,
      { 0xA0, 0xC7, 0x2F, 0xAD, 0x1F, 0xD0, 0xE7, 0x16 } };

Then it enables that provider on the session:

status = EnableTraceEx2(
    _sessionHandle,
    &MicrosoftWindowsKernelProcessProviderGuid,
    EVENT_CONTROL_CODE_ENABLE_PROVIDER,
    TRACE_LEVEL_INFORMATION,
    0,
    0,
    0,
    nullptr
);

At this point we have a session and a provider enabled on that session. The remaining problem is consuming the events.

How the consumer thread reads events

The CoreEngine already has its own event processing thread.

The EtwSensor also creates a thread.

That thread is responsible for opening the trace and letting Windows call our callback whenever an ETW record arrives.

_threadHandle = CreateThread(
    nullptr,
    0,
    EtwConsumeThread,
    this,
    0,
    nullptr
);

We run our custom EtwConsumeThread on a newly spawned thread. That thread is the consumer thread of the ETW sensor.

The sensor then configures an EVENT_TRACE_LOGFILE.

EVENT_TRACE_LOGFILE traceLogfile{};
traceLogfile.LoggerName = sessionName;
traceLogfile.ProcessTraceMode =
    PROCESS_TRACE_MODE_REAL_TIME |
    PROCESS_TRACE_MODE_EVENT_RECORD;
traceLogfile.EventRecordCallback = EtwRecordDispatch;

There are two important pieces here: we are consuming the trace in real time, and every received record goes to EtwRecordDispatch.

Then the sensor opens the trace and starts processing it:

sensor->_traceHandle = OpenTrace(&traceLogfile);

ULONG status = ProcessTrace(
    &sensor->_traceHandle,
    1,
    nullptr,
    nullptr
);

ProcessTrace() is a blocking call, so it runs in its own thread.

If we ran it directly in the main thread, the AV would block there and the rest of the engine lifecycle would block.

How raw ETW records get dispatched

Every ETW record received by the sensor goes through the EtwRecordDispatch callback function.

This function is the first filtering point: the sensor looks at the ETW event ID and decides whether the event is something we currently care about.

void WINAPI EtwSensor::EtwRecordDispatch(PEVENT_RECORD EventRecord) {
    NormalizedEvent normalizedEvent;
    bool sendToEngine = false;

    switch (EventRecord->EventHeader.EventDescriptor.Id) {
        case RawEtwProcessEvent_ProcessStart:
            sendToEngine =
                NormalizeProcessCreateEtwEvent(
                    EventRecord,
                    &normalizedEvent
                );
            break;

        case RawEtwProcessEvent_ImageLoad:
            sendToEngine =
                NormalizeImageLoadEtwEvent(
                    EventRecord,
                    &normalizedEvent
                );
            break;

        default:
            return;
    }

    if (_onEventDetected != nullptr && sendToEngine) {
        _onEventDetected(normalizedEvent);
    }
}

Right now, the MVP forwards two event types:

• ProcessCreate
• ImageLoad

Other events are ignored for now, including process stop, thread start, thread stop and image unload. They may become useful later, especially when we start thinking about lineage, correlation and race conditions.

Why We Normalize Events

Raw ETW records are not the format I want the rest of the AV to consume.

The engine should not need to understand ETW internals, so the sensor normalizes raw records into NormalizedEvent.

struct NormalizedEvent {
    EventSource eventSource;
    ULONG pid;
    FILETIME timestamp;
    std::variant<EtwEvent> specificEvent;
};

For ETW events, the specific event is an EtwEvent.

struct EtwEvent {
    EtwEventType etwEventType;
    std::variant<
        ProcessCreateEtwEvent,
        ProcessTerminateEtwEvent,
        ImageLoadEtwEvent,
        ImageUnloadEtwEvent,
        ThreadCreateEtwEvent,
        ThreadTerminateEtwEvent,
        NetworkConnectEtwEvent
    > details;
};

struct ProcessCreateEtwEvent {
    ULONG parentPid;
    ULONG sessionId;
    std::wstring imagePath;
    ProcessTokenElevationType elevationType;
};

struct ImageLoadEtwEvent {
    ULONGLONG imageBase;
    SIZE_T imageSize;
    std::wstring imagePath;
};

This structure is more verbose than throwing raw fields around, but it gives us a cleaner boundary.

The engine can switch on EtwEventType without caring about how Windows exposed the original payload.

How process creation gets normalized

For a process creation event, the sensor fills the common fields first.

It sets the source to ETW, copies the timestamp from the ETW header, and extracts the created process PID from the payload.

normalizedEvent->eventSource = EventSource::Etw;

normalizedEvent->timestamp.dwLowDateTime =
    EventRecord->EventHeader.TimeStamp.LowPart;
normalizedEvent->timestamp.dwHighDateTime =
    EventRecord->EventHeader.TimeStamp.HighPart;

if (!GetTdhIntPropertyAny(
        EventRecord,
        { L"ProcessID", L"ProcessId", L"NewProcessId" },
        &payloadPid
    )) {
    return false;
}

normalizedEvent->pid = payloadPid;

Then it extracts parent PID, session ID, image path and token elevation type.

The image path is pulled from ImageName or ImageFileName.

If both are missing, the sensor falls back to Unknown.

std::wstring imagePath =
    GetTdhStringProperty(EventRecord, L"ImageName");

if (imagePath.empty()) {
    imagePath = GetTdhStringProperty(EventRecord, L"ImageFileName");
}

if (imagePath.empty()) {
    imagePath = L"Unknown";
}

Then the final event is built:

normalizedEvent->specificEvent = EtwEvent{
   .etwEventType = EtwEventType::ProcessCreate,
   .details = ProcessCreateEtwEvent{
        .parentPid = parentPid,
        .sessionId = sessionId,
        .imagePath = imagePath,
        .elevationType = elevationType
    }
};

At this point, the sensor only knows that a process was created and it has enough metadata to send the event forward.

How image loads get normalized

The second event type currently handled by the MVP is image loading.

This matters because a process may load an executable image or DLL after creation.

For this event, the sensor extracts PID, timestamp, image base, image size and image path.

The structure built at the end is:

normalizedEvent->specificEvent = EtwEvent{
    .etwEventType = EtwEventType::ImageLoad,
    .details = ImageLoadEtwEvent{
        .imageBase = imageBase,
        .imageSize = imageSize,
        .imagePath = imagePath
    }
};

Again, no detection happens here. The sensor is still only translating Windows telemetry into NewGen-AV telemetry.

Why TDH Is Used

You will notice that the code uses helpers like this:

GetTdhIntPropertyAny(
    EventRecord,
    { L"ProcessID", L"ProcessId", L"NewProcessId" },
    &payloadPid
);

TDH is the Trace Data Helper API.

I use it here to read named properties from ETW records instead of manually parsing raw buffers everywhere.

I tried to manually parse the events because I thought it would be faster than calling Win API functions, but I was getting inconsistent data.

It looks like ETW event payloads are not always as clean as I currently understand it. It may change across providers, manifests and event versions.

So we can delegate the parsing of the events to the TDH helper. For each field, the MVP tries a small list of acceptable names and uses the first one that works.

How the event gets sent to the engine

Once normalization succeeds, the sensor sends the event to the engine through the callback.

The snippet below is taken from EtwSensor::EtwRecordDispatch, that we examined a few paragraphs above.

if (_onEventDetected != nullptr && sendToEngine) {
    _onEventDetected(normalizedEvent);
}

This is the handoff point.

After this line, the EtwSensor is done.

The CoreEngine receives the event, pushes it into its queue, wakes up the processing thread, and later runs the relevant analysis path.

How the sensor stops

The sensor also owns its shutdown path.

Conceptually, Stop() does this:

Stop()
  -> CloseTrace()
  -> stop the ETW session with ControlTrace()
  -> wait for the consumer thread
  -> close the thread handle
  -> free the session properties buffer

The destructor also calls Stop(), so the sensor should not leave an ETW session running behind if the object is destroyed.

For an MVP, this shutdown logic is good enough. Later, I may consider more detailed state handling and better error reporting around partial shutdowns.

Where the sensor still has limitations

This sensor is intentionally limited.

First, it only forwards ProcessCreate and ImageLoad. The enum already has room for more ETW event types, but the implementation does not process them yet.

Second, ETW is post-event. By the time the engine receives the event, the process may already have executed some code.

This creates a TOCTOU problem: the engine observes a process-create event, then later resolves and scans a file path. Between those two moments, the file on disk may have changed, been renamed, been replaced, or become unavailable. The path seen in telemetry is not a perfect guarantee that the bytes scanned later are the same bytes that were originally executed.

This is why the current MVP should not be described as real-time prevention or reliable protection. It is an event-driven detection and response prototype.

Third, path quality is not guaranteed. Sometimes the event may not give us a useful image path, and the code falls back to Unknown. Path conversion from kernel-style paths to DOS paths happens later in the static analyzer, not inside the sensor.

Fourth, the current event model still uses PID as the main process identifier. There is already a TODO in the code about this, because PIDs can be recycled. At some point, lineage and stronger process identity will matter.

For now, this is acceptable for the first MVP.

What this post covered

So to recap, the EtwSensor is the first telemetry source of NewGen-AV.

It creates a real-time ETW session, enables the Kernel Process provider, starts a consumer thread, receives raw ETW records, filters the event types we currently care about, normalizes them into NormalizedEvent, and sends them to the CoreEngine through a callback.

The important design decision is that the sensor does not detect and does not respond.

It only observes and reports.

When we add more sensors later, they should follow the same idea: collect telemetry, normalize it, and hand it to the engine.

Where we go from here

In the next post we will move one step further in the pipeline.

The ETW sensor can tell us that a process was created or that an image was loaded, but that is not enough to decide whether something is malicious.

Next we need to understand what happens when the engine receives an event, resolves the path, loads the file and asks the static analyzer to scan it.