What is Mixture of Experts (MoE)?

Mixture of Experts (MoE) is a model architecture technique that speeds up AI inference by routing tasks to the most capable part of the model. 

MoE models are specially trained to answer specific subcategories quickly and accurately. 

Why you should care about inference 

Think of it this way: If you were a student with a question about human anatomy, would you knock on every professor’s door until you got your answer? Or would you go to your biology professor first? You’d likely go straight to your biology professor, the best match in your “mixture of experts.” 

Why? Because you want the correct answer as quickly as possible.

Even though all your professors are knowledgeable in their own subjects, you knew your biology professor would have the answer when it came to human anatomy. That’s why you asked them, instead of taking a detour through the English department. 

Mixture of Experts uses the same logic. 

Read a blog post about scaling intelligence with MoE 

For inference to be successful, AI models need to do a lot of calculations in a short period of time. As models get bigger, they become more complex and inference slows down. Factors like model size, high user volume, and latency can all limit performance. 

To overcome these challenges, Mixture of Experts creates a neural network that supports faster inference at scale. 

How does MoE use deep learning? 

Deep learning is an AI technique that teaches computers to process data and learn through observation, imitating the way humans gain knowledge.

There are 2 defining characteristics that support model function: 

• Transfer learning is when a model applies information about 1 situation to another and builds upon its internal knowledge. Many foundation models have hundreds of neural layers that are pretrained with deep learning techniques. This is how models discover relationships and patterns within a dataset.
• Scale refers to hardware—specifically graphics processing units (GPUs)—that allows the model to perform multiple computations simultaneously. 

MoE integrates deep learning training and transfer learning to identify patterns and subcategories within prompts. MoE models can then quickly identify the best “expert” to answer the input. MoE uses GPUs to scale and accelerate the prompt-to-answer pipeline. 

Learn more about foundation models 

How does MoE use neural networks? 

Neural networks form the underlying architecture of deep learning. They’re made up of many layers of neurons that interpret data. 

Traditionally, each layer interprets incoming data and sends it to the next layer, and so forth, until it reaches a neuron that can answer the prompt. These typically dense neural networks are called feed-forward networks (FFNs). 

FFNs send data in 1 direction, through all of its parts: input layers, hidden layers, and output layers. As data flows from the input layers to the output layers, hidden layers learn the patterns and trends of each input to deliver a final result. 

Unlike FFNs, MoEs can take multiple paths to deliver an output. When MoEs identify experts, it shortens the path to a final result and expands model capacity. This is how models learn new information and identify patterns without using more memory, compute, or time. 

To block out the noise of other computations happening at the same time, MoE introduces sparsity.

How does MoE use sparsity?

Sparsity is a technique that helps neural networks save memory by using fewer weights. 

Weights are calculations that tell a model what action to take. Each weight is scored based on its ability to answer each prompt. This allows the input to be mapped to the correct expert. But, not every weight is necessary for every prompt. Sparsity identifies the necessary weights and ignores the weights that aren’t critical. 

In technical terms, this means unnecessary weights are set to 0. When the model sees 0, it knows to skip those calculations (because anything multiplied by 0 equals 0). This means experts can focus only on the weights that matter.

When unnecessary weights are hidden, the model has more memory and can work faster. The tricky part is finding the sweet spot between increasing speed and decreasing accuracy or performance.

Find more ways to optimize inference