Static Analysis and AMSI

What we covered before

In the previous posts we saw that the Launcher starts the CoreEngine, the CoreEngine initializes the components, starts the event processing thread and then starts the EtwSensor. We saw the EtwSensor and how it turns Windows telemetry into NormalizedEvent objects.

At that point, the AV knows that something happened. A process was created or an image was loaded, but that does not mean the AV knows whether the file is malicious or not.

This post is about the next step in the MVP pipeline: static analysis.

More specifically, this post covers how NewGen-AV takes the path received from ETW, resolves it into something Windows can open, loads the file into memory, scans it with AMSI (more on it down the line), turns that result into an internal verdict, and kills the process if the verdict is malicious.

This is still a very simple design and relies completely on AMSI's capabilities, but for now it's sufficient.

Where Static Analysis Sits

The current pipeline looks like this:

  -> ETW event
  -> EtwSensor (catch and normalize)
  -> NormalizedEvent
  -> CoreEngine queue
  -> StaticAnalyzer
  -> AMSI
  -> verdict
  -> process kill if malicious

The EtwSensor alerts the CoreEngine which receives the event and decides which analysis path should run.

The StaticAnalyzer is responsible for taking a file path and producing a verdict.

The response code is responsible for acting on that verdict.

This separation matters because scanning and responding are not the same thing.

Scanning answers: what does this file look like?

Response answers: what do we do now?

For this MVP, if a file is detected as malicious, the response is simple: kill the process.

How the StaticAnalyzer exposes its interface

From the outside, the static analyzer is small.

struct StaticAnalysisResult {
    std::wstring dosPath;

    StaticDetectionVerdict verdict = StaticDetectionVerdict::NotAnalyzed;
    StaticDetectionSource source = StaticDetectionSource::None;
};

class StaticAnalyzer {
public:
    bool Initialize();
    bool AnalyzeFile(const std::wstring& kernelPath, StaticAnalysisResult& outResult
    );
};

The engine gives it a path. The analyzer fills a result.

That result contains:

• the DOS path used for the scan.
• the verdict.
• the detection source.

Right now the only real detection source is AMSI.

There are placeholders in the code for custom byte-array analysis and YARA rules, but they are not implemented yet. You get the idea of how the main object of static analysis could then juggle more sub-components to do specific types of static analysis.

How StaticAnalyzer initializes

The StaticAnalyzer is initialized when the CoreEngine initializes.

if (!_staticAnalyzer.Initialize()) {
    printf("[CORE] Error in StaticAnalyzer init.\n");
    return false;
}

Inside StaticAnalyzer::Initialize(), two things happen.

First, AMSI is initialized.

Second, the analyzer builds the device-to-drive-letter mappings needed for path conversion.

bool StaticAnalyzer::Initialize() {
    if (_isInitialized) return true;

    if (!_amsiAnalyzer.Initialize()) {
        return false;
    }

    if (!RefreshMappings()) {
        return false;
    }

    _isInitialized = true;
    return true;
}

This is an important step because ETW paths are not always in the format we want.

Sometimes we may receive a normal DOS path like:

C:\Windows\System32\cmd.exe

But sometimes Windows internals may expose paths that look closer to this:

\Device\HarddiskVolume3\Windows\System32\cmd.exe

The file loading code wants a path it can open.

So before scanning, the analyzer needs path normalization.

How path resolution works

Path resolution is handled through ConvertKernelPathToDosPath().

The analyzer wraps the shared helper:

std::wstring StaticAnalyzer::ConvertKernelPathToDosPath(
    const std::wstring& kernelPath
) const {
    return ::ConvertKernelPathToDosPath(
        kernelPath,
        _deviceDosMappings
    );
}

The mappings are built by asking Windows which DOS drive letters map to which device paths.

QueryDosDeviceW(
    dosDeviceName,
    targetPath,
    static_cast<DWORD>(std::size(targetPath))
);

Conceptually, this gives us mappings like:

\Device\HarddiskVolume3 -> C:

Then, when a kernel-style path arrives, the helper can convert it into a DOS-style path.

It also handles paths that are already DOS paths and paths prefixed with \??\ or \\?\.

This part is not detection.

It is plumbing.

But without it, the scanner may fail before we even get to AMSI.

How the file gets loaded

Once the analyzer has a DOS path, it loads the file into memory.

std::vector<BYTE> fileData;
if (!LoadFileToBuffer(outResult.dosPath, fileData)) {
    outResult.verdict = StaticDetectionVerdict::Error;
    return false;
}

The helper opens the file, checks the file size, allocates a buffer, reads the content and returns it to the analyzer.

HANDLE hFile = CreateFileW(
    filePath.c_str(),
    GENERIC_READ,
    FILE_SHARE_READ | FILE_SHARE_WRITE,
    NULL,
    OPEN_EXISTING,
    FILE_ATTRIBUTE_NORMAL,
    NULL
);

The sharing flags matter.

The process may already be running and the file may be in use.

For this MVP, the code tries to read the file while allowing read and write sharing.

If the file cannot be opened or read, the analyzer returns an error verdict.

No file data means no AMSI scan.

How AMSI acts as the first scanner

AMSI stands for Antimalware Scan Interface. It is a Windows interface that submits content to the antimalware provider registered on the machine and returns that provider's verdict.

For this MVP, I use it as the first static scanning layer.

The AmsiAnalyzer owns the AMSI context.

HRESULT hr = AmsiInitialize(
    L"NewGen-AV-AmsiContext",
    &_amsiContext
);

The context is initialized once and reused during the lifetime of the analyzer.

When a file needs to be scanned, the analyzer opens an AMSI session, scans the buffer, closes the session and converts the result into a small internal enum.

hr = AmsiScanBuffer(
    _amsiContext,
    (PVOID)fileBuffer.data(),
    static_cast<ULONG>(fileBuffer.size()),
    contentName.c_str(),
    amsiSession,
    &amsiResult
);

Then:

if (AmsiResultIsMalware(amsiResult)) {
    return AmsiScanVerdict::Detected;
}

return AmsiScanVerdict::Clean;

This is intentionally simple.

The AMSI wrapper returns detected, clean, or error.

NewGen-AV maps that into its own static verdict.

How AMSI results map to NewGen-AV verdicts

The StaticAnalyzer does not return raw AMSI values to the engine.

It converts them into StaticDetectionVerdict.

enum class StaticDetectionVerdict {
    Clean,
    Malicious,
    NotAnalyzed,
    Error
};

enum class StaticDetectionSource {
    None,
    Amsi,
    Custom_ByteArrayAnalysis,
    Yara_Rules
};

AmsiScanVerdict amsiResult =
    _amsiAnalyzer.ScanBuffer(fileData, outResult.dosPath);

switch (amsiResult) {
    case AmsiScanVerdict::Detected:
        outResult.verdict = StaticDetectionVerdict::Malicious;
        outResult.source = StaticDetectionSource::Amsi;
        break;

    case AmsiScanVerdict::Error:
        outResult.verdict = StaticDetectionVerdict::Error;
        outResult.source = StaticDetectionSource::Amsi;
        break;

    case AmsiScanVerdict::Clean:
        outResult.verdict = StaticDetectionVerdict::Clean;
        outResult.source = StaticDetectionSource::Amsi;
        break;
}

This gives the rest of the engine one language for detection results.

Today the source is AMSI.

Tomorrow it could be YARA, custom byte signatures, metadata checks, or something else.

The engine should not need to care which scanner produced the verdict.

It should care about the final verdict and the source.

Where the Engine Calls the Analyzer

There are currently two engine paths that call static analysis.

The first one is process creation:

void CoreEngine::OnProcessCreateAnalysis(
    NormalizedEvent normalizedEvent
) {
    auto etwEvent =
        std::get<EtwEvent>(normalizedEvent.specificEvent);
    auto processCreateEtwEvent =
        std::get<ProcessCreateEtwEvent>(etwEvent.details);

    StaticAnalysisResult staticAnalysisResult;

    if (_staticAnalyzer.AnalyzeFile(
            processCreateEtwEvent.imagePath,
            staticAnalysisResult
        )) {
        ...
    }
}

The second one is image loading:

void CoreEngine::OnImageLoadAnalysis(
    NormalizedEvent normalizedEvent
) {
    auto etwEvent =
        std::get<EtwEvent>(normalizedEvent.specificEvent);
    auto imageLoadEtwEvent =
        std::get<ImageLoadEtwEvent>(etwEvent.details);

    if (imageLoadEtwEvent.imagePath.empty() ||
        imageLoadEtwEvent.imagePath == L"Unknown") {
        return;
    }

    StaticAnalysisResult staticAnalysisResult;

    if (_staticAnalyzer.AnalyzeFile(
            imageLoadEtwEvent.imagePath,
            staticAnalysisResult
        )) {
        ...
    }
}

So the MVP scans both the image path of a newly created process and the image path from image-load events.

This does not mean the AV has complete coverage.

It only means those two ETW event types currently feed the static analyzer.

How a malicious verdict becomes a process kill

Once the static analyzer returns successfully, the engine checks the verdict.

If the verdict is malicious, it calls TerminateMaliciousProcess().

case StaticDetectionVerdict::Malicious:
    TerminateMaliciousProcess(normalizedEvent.pid);
    printf(
        "[CORE] ProcessCreate DETECTION PID %d is malicious - %ls\n",
        normalizedEvent.pid,
        processCreateEtwEvent.imagePath.c_str()
    );
    break;

The same idea exists for ImageLoad.

case StaticDetectionVerdict::Malicious:
    TerminateMaliciousProcess(normalizedEvent.pid);
    printf(
        "[CORE] ImageLoad DETECTION PID %d is malicious - %ls\n",
        normalizedEvent.pid,
        imageLoadEtwEvent.imagePath.c_str()
    );
    break;

The current response is intentionally direct.

If the AMSI-backed verdict is malicious, the MVP tries to kill the process associated with the event.

How NtTerminateProcess kills the target process

The kill logic lives in the CoreEngine.

During initialization, the engine resolves NtTerminateProcess from ntdll.dll.

_NtTerminateProcess =
    reinterpret_cast<pfnNtTerminateProcess>(
        GetProcAddress(_hNtdll, "NtTerminateProcess")
    );

When the response code runs, it opens a handle to the target process and calls the native termination API.

bool CoreEngine::TerminateMaliciousProcess(DWORD targetPid) {
    if (targetPid <= 4) return false;

    HANDLE hTarget = OpenProcess(
        PROCESS_SUSPEND_RESUME | PROCESS_TERMINATE,
        FALSE,
        targetPid
    );
    if (!hTarget) return false;

    NTSTATUS status = _NtTerminateProcess(
        hTarget,
        static_cast<NTSTATUS>(0xDEADC0DE)
    );

    CloseHandle(hTarget);
    return NT_SUCCESS(status);
}

There are a few important details here.

The MVP avoids targeting very low PIDs.

It opens the target process with termination rights.

For testing purposes, NtSuspendProcess was also resolved and tested. The related code is still present but commented out. This is also why the process handle is currently opened with the PROCESS_SUSPEND_RESUME access right.

Then it calls NtTerminateProcess.

It closes the handle.

It returns whether the native call succeeded.

That is the whole response for now.

What This Does Not Guarantee

The current AV kills after detection.

Detection happens after ETW delivered an event, after the event was normalized, after the engine popped it from the queue, after path resolution, after file loading and after AMSI scanning.

That chain can be fast.

But it is still post-event.

So this is not reliable prevention.

A short-lived malicious process may finish its work before the MVP kills it.

A process may also fail to be scanned because the path is missing, the file cannot be opened, the path cannot be resolved properly, AMSI returns an error, or the detection source simply does not flag it.

That does not make the MVP useless.

It makes the boundary clear.

This is a first static analysis and response pipeline.

Not a complete protection product.

What this post covered

So to recap, this post covered the first static analysis path in NewGen-AV.

The engine receives a normalized ETW event and calls the StaticAnalyzer.

The analyzer converts the path, loads the file, sends the buffer to AMSI and maps the AMSI result into a NewGen-AV verdict.

If that verdict is malicious, the CoreEngine tries to terminate the process with NtTerminateProcess.

The design is simple, but it gives the MVP a complete first loop:

observe -> normalize -> scan -> verdict -> kill

This is the first point where NewGen-AV stops being only a telemetry viewer and starts reacting to what it sees.

Where we go from here

The next useful step is not pretending this is enough.

It is testing where it breaks.

We could look at timing, short-lived processes, path failures, image-load behavior, AMSI errors and cases where killing only one PID is not enough.

So in the next post we will limit test this version, hack around it and decide what's worth implementing as next feature.

That is where the next engineering decisions will come from.