Many believe that Microsoft will eventually provide a version of Copilot that will run inside Windows on your PC. Microsoft just tipped its hand with a new AI model, or LLM, designed specifically to run on native devices.

On Monday, Microsoft introduced “Phi-3-mini”, a 3.8-billion parameter language model that the company claims rivals the slightly older ChatGPT 3.5 and Mixtral 8x7B in performance. The paper is titled “Phi-3 Technical Report: A Highly Compliant Language Model Natively on Your Phone”, which is clear evidence that Microsoft now has an LLM that can run directly on your PC. Is.

Microsoft hasn't said that the Phi-3-mini will be the next Copilot, running natively on your computer. But there's a case to be made that it is, and we have an idea of ​​how well it will work.

1.) Native AI matters

It's a familiar argument: If you issue a search request to Bing, Google Gemini, Claude, or Copilot, it lives in the cloud. It could be embarrassing (“Is this wart bad”?) or sensitive (“Could I get in trouble for stealing mail?”) or something you don't want to leak at all, like a bank statement. List of

Corporations may want to ask questions of their data through AIs like Copilot, but to date there are no “on-premises” versions of Copilot. The native version of Copilot is almost a must-have option at this point.

Mark Hechman / IDG

Mark Hechman / IDG

Mark Hechman / IDG

2.) The Phi-3-mini is small, and that's cool.

Many local LL.M The necessary files on a PC alone require at least 8 GB of RAM and several gigabytes of storage space – and that's assuming the LLM has been “quantized” or compressed. There's a reason Copilot runs in the cloud. That's because Microsoft's Azure cloud has both the compute and storage necessary to host it.

This is one of the most important lines of the paper: “Thanks to its small size, phi3-mini can be quantized to 4 bits so that it occupies only ≈ 1.8GB of memory,” it says. “We experimented with quantized.
Model on the iPhone 14 with the A16 Bionic chip running phi-3-mini on the device and receiving more than 12 tokens per second completely offline.

This means that PCs with an additional 1.8GB of memory can run the native version of Copilot. That's a lot more PCs than the status quo, including perhaps legacy PCs with 8GB of RAM. And if an iPhone can run it, chances are most PCs can too.

LLMs spit out data like a dot-matrix printer, so twelve tokens per second is about 48 characters per second—not great, but not too shabby either.

3.) Its performance is also good – but not great.

Microsoft still wants to get people to use Copilot in the cloud and pay for subscriptions. Copilot Pro. And Microsoft takes the time to show how Phi-3-mini performs like other LLMs. But a small number of parameters means an AI model doesn't “know” as much as more sophisticated LLMs.

Arxiv.org

Arxiv.org

Arxiv.org

But Microsoft has a solution. “The model simply doesn't have the ability to store a lot of 'factual knowledge,' which can be seen for example with low performance on TriviaQA (a benchmark),” the paper notes. “However, we believe that such a vulnerability can be addressed through search engine enhancements.”

Bing to the rescue!

4.) It was a month before the Microsoft build.

Microsoft's developer conference kicks off in a month's time. That gives Microsoft time to get a version of Copilot running on top of the Phi-3-mini to show developers and users before the May 21 launch. This should be enough time to develop a prototype and test some queries.

The copilot, of course, will be the star of the show. Can we find one that runs natively on your computer? I think that's more likely than ever.