The AV Starting Point

What we covered before

Before diving into sensors, ETW, detection logic, and process termination, we need to understand how to start the AV.

In the previous post I described the high-level architecture of the AV: sensors collecting events, the engine processing them and deciding whether and how to react.

Now we move from the architecture diagram to the code implementation.

How the Launcher starts the AV

Everything starts here.

The executable we run to start the AV should not know how the AV works internally.

If we wrote everything directly in main(), at the beginning it would probably look fine: a few function calls, a few initializations, some callbacks here and there.

But then we would add API hooks, a driver module, more detection rules, more response logic.

Suddenly the main() becomes the place where everything happens, and that is exactly what we want to avoid.

So we decide that:

• The Launcher starts the engine.
• The CoreEngine owns the AV logic.

That separation is the first architectural decision.

What we want the launcher to do is this:

main():
  -> Set a clean exit mechanism (Ctrl+C handler)
  -> Enable SeDebugPrivilege
  -> Initialize the CoreEngine
  -> Start the CoreEngine
  -> Shutdown the CoreEngine

And that is basically what is implemented, with a few checks around it.

int main() {
    CoreEngine coreEngine;
    g_EngineInstance = &coreEngine;

    if (!SetConsoleCtrlHandler(ConsoleCtrlHandler, TRUE)) {return 1;}
    if (!EnableDebugPrivilege()) {return 2;}
    if (!coreEngine.Initialize()) {return 3;}

    coreEngine.Start();
    coreEngine.Shutdown();

    return 0;
}

The launcher does not know about ETW internals, event queues, AMSI scanning, or process termination details.

It only prepares the environment and hands execution to the engine.

How the CoreEngine exposes its interface

To understand what happens when we call Initialize(), Start() and Shutdown(), we need to look at the CoreEngine.

The important thing here is the separation between the public surface and the internal machinery.

Publicly, the engine is simple. You initialize it, start it, and shut it down.

From the outside, the engine exposes three main actions:

That is the contract between the launcher and the AV logic.

Everything more complex stays behind that interface.

What the CoreEngine owns inside

Internally, the CoreEngine owns the components that make the AV work:

• the EtwSensor, responsible for receiving telemetry from ETW
• the StaticAnalyzer, responsible for analyzing executables
• the event processing thread
• the shared queue used to move events from the sensor thread to the engine thread
• the shutdown logic, which keeps the engine alive until we explicitly stop it
• the response functions, which are called when a malicious verdict requires action

A simplified version of the class looks like this:

class CoreEngine {
public:
    bool Initialize();
    void Start();
    void Shutdown();

private:

	std::atomic<bool> _isRunning{ false };
	
    std::unique_ptr<EtwSensor> _etwSensor;
    StaticAnalyzer _staticAnalyzer;

    HMODULE _hNtdll;
    pfnNtTerminateProcess _NtTerminateProcess = nullptr;

    // queue, mutex, condition variables, threads
    ...

    void OnEtwEventDetected(const NormalizedEvent& normalizedEvent);
    void EventProcessingLoop();
    bool TerminateMaliciousProcess(DWORD pid);

    void OnProcessCreateAnalysis(NormalizedEvent normalizedEvent);
    void OnImageLoadAnalysis(NormalizedEvent normalizedEvent);
};

So the CoreEngine is not a detector by itself and it is not a sensor either.

It is the place where those components are wired together.

The sensors observe, the analyzers evaluate, the queues decouple the threads, and the engine decides when each piece should run.

The engine lifecycle

The CoreEngine follows a simple lifecycle:

Initialize()

Initialize() prepares the engine before it starts processing events.

At this stage, that means resolving the native APIs we need from ntdll, initializing the StaticAnalyzer, creating the EtwSensor, and registering the callback that will receive normalized events from the sensor.

Conceptually, this is the setup phase.

The engine is not yet processing events.

bool CoreEngine::Initialize() {
    _hNtdll = GetModuleHandleW(L"ntdll.dll");
    if (...) {...}

    _NtTerminateProcess =
        reinterpret_cast<pfnNtTerminateProcess>(
            GetProcAddress(_hNtdll, "NtTerminateProcess")
        );
    if (...) {...}

    if (!_staticAnalyzer.Initialize()) {...}

    _etwSensor = std::make_unique<EtwSensor>();
    if (!_etwSensor) return false;

    _etwSensor->SetCallback([this](const NormalizedEvent& normalizedEvent) {
        this->OnEtwEventDetected(normalizedEvent);
    });

    return true;
}

Since ETW is our only sensor for now, if something fails here, the engine should not continue.

Starting a security tool in a half-initialized state is worse than not starting it at all.

In the future, if we have multiple sensors, we may decide that one failed sensor is not enough to stop the whole AV. We could also consider self-healing logic, where a failed sensor tries to restart.

For this MVP, we just exit.

The important piece in this function is the callback registration.

The EtwSensor will call OnEtwEventDetected() each time it has a meaningful event to send to the engine.

That callback - defined in CoreEngine - receives a NormalizedEvent, which is produced by the EtwSensor.

We will go deeper into the ETW sensor in the relative post, but for now this is enough:

struct NormalizedEvent {
    EventSource eventSource;
    ULONG pid;
    FILETIME timestamp;
    std::variant<EtwEvent> specificEvent;
};

enum class EventSource {
    Etw,
    ApiHooks,
    Kernel
};

The point is that the engine does not process raw ETW data directly.

It receives an internal event format with the fields we care about: source, PID, timestamp, and event-specific details.

Start()

Start() is where the engine begins doing work.

It starts the event processing thread, starts the ETW sensor, and then keeps the main thread alive while the AV is running.

The important detail is that Start() should not become the place where detection logic lives.

Its job is runtime coordination.

The actual event handling happens inside the event processing loop.

At this point, the simplified flow is:

Start()
  -> start event processing loop in a parallel thread
  -> start the ETW sensor, which also starts its own thread
  -> wait until shutdown is requested

We end up with three threads:

• the orchestrator thread, sleeping and waiting for shutdown
• the ETW sensor thread, reading and normalizing events
• the event processing thread, receiving normalized events and processing them

void CoreEngine::Start() {
    
    if (!_etwSensor) {...}
    
    // Setting _isRunning to true
    if (_isRunning.exchange(true) == true) {...}
    
    // Start the thread that will process the events
    _eventProcessingThread =
        std::thread(&CoreEngine::EventProcessingLoop, this);
    
    // Start the EtwSensor (it will spawn its own thread)
    if (!_etwSensor->Start()) {...}
    
    // the main thread goes to sleep, it will get woken up on program close
	// this will save cpu consumption
	std::unique_lock<std::mutex> lock(_shutdownMutex);
    _shutdownCondition.wait(lock, [this]() {
        return !_isRunning;
    });
}

The main thread does not need to burn CPU while waiting.

It can block on a condition variable and wake up when the engine shuts down.

Shutdown()

Shutdown() stops the engine safely.

For this MVP, I am keeping this simple.

When shutdown is requested, the engine flips the running flag, stops the ETW sensor, wakes up the event processing thread, waits for it to exit, and then wakes up the main engine thread.

Shutdown()
  -> stop the ETW sensor
  -> wake up and join the event processing thread
  -> wake up the main engine thread

void CoreEngine::Shutdown() {
    if (_isRunning.exchange(false) == false) {
        return;
    }

    if (_etwSensor) {
        _etwSensor->Stop();
    }

    _queueNotificationEvent.notify_one();

    if (_eventProcessingThread.joinable()) {
        _eventProcessingThread.join();
    }

    _shutdownCondition.notify_one();
}

The exchange(false) prevents double shutdown.

If the engine is already stopped, and _isRunning is already false, there is nothing else to do.

How events move through a shared queue

At this point, we have two different pieces of code running in parallel:

• the EtwSensor thread.
• the CoreEngine event processing thread.

To pass events between them, we use a shared queue.

EtwSensor thread -> Shared queue -> Engine processing thread

When the EtwSensor receives an event, it calls the callback registered by the engine.

That callback pushes the event into the queue and notifies the processing thread that new work is available.

void CoreEngine::OnEtwEventDetected(const NormalizedEvent& normalizedEvent) {
	// pushing event into the queue
    {
        std::lock_guard<std::mutex> lock(_queueMutex);
        _eventQueue.push(normalizedEvent);
    }
    
	// waking up the event processing thread
    _queueNotificationEvent.notify_one();
}

This function belongs to the CoreEngine, so it has access to the queue, mutex and condition variable.

But it is executed by the EtwSensor thread.

That is why the mutex matters: it protects the queue from concurrent access.

The condition variable matters too, because it lets the processing thread sleep until there is actual work to do.

Without it, the processing thread would probably end up constantly checking the queue in a loop and wasting resources.

The event processing loop looks like this:

void CoreEngine::EventProcessingLoop() {
    while (_isRunning) {
        NormalizedEvent normalizedEvent;

        {
            std::unique_lock<std::mutex> lock(_queueMutex);

            // Wait until there is an event or shutdown signal
            _queueNotificationEvent.wait(lock, [this]() {
                return !_eventQueue.empty() || !_isRunning;
            });

            if (!_isRunning && _eventQueue.empty()) {
                break;
            }

            normalizedEvent = _eventQueue.front();
            _eventQueue.pop();
        }

        // ETW Event 
        if (const auto* etwEvent =
                std::get_if<EtwEvent>(&normalizedEvent.specificEvent)) {
            switch (etwEvent->etwEventType) {
                case EtwEventType::ProcessCreate:
                    OnProcessCreateAnalysis(normalizedEvent);
                    break;

                case EtwEventType::ImageLoad:
                    OnImageLoadAnalysis(normalizedEvent);
                    break;

                default:
                    break;
            }
        }
    }
}

First, the processing thread sleeps if there is nothing to do.

When the callback notifies it, the thread wakes up, pops the first event from the queue, and handles it according to the event type.

Right now the handled ETW cases are ProcessCreate and ImageLoad.

An important detail is that the actual analysis happens outside the lock.

The lock should only protect the queue.

It should not be held while running static analysis, terminating processes, printing logs, or doing anything time-consuming.

Otherwise, the sensor may be blocked from pushing new events while the engine is busy analyzing the previous one.

For now, one event processing thread is enough.

If the engine becomes overloaded later, this design can evolve into a thread pool.

The sensor side does not need to change much: it will still push events into the queue.

The difference is that multiple worker threads may consume events from that queue instead of only one.

For this first version of the AV, I will not invest time into building the thread pool.

What we covered in this post

So to recap, we saw how NewGen-AV starts.

The launcher stays intentionally simple: it handles shutdown, enables the required privilege, initializes the CoreEngine and starts it.

Inside the engine, the important pieces are the EtwSensor, the StaticAnalyzer, the event queue, the processing thread, and the shutdown logic.

The key design decision is the separation between telemetry collection and detection/response: the sensor pushes normalized events into a queue, and the engine processes them from a separate thread.

Where we go from here

In the next post we will dive into the EtwSensor.

We will see:

• its lifecycle
• how it gets raw events from ETW
• how it normalizes them
• the callback function from the EtwSensor point of view