ibm-research/PowerMoE-3b

PowerMoE-3B is a 3B sparse Mixture-of-Experts (sMoE) language model trained with the Power learning rate scheduler. It sparsely activates 800M parameters for each token. It is trained on a mix of open-source and proprietary datasets. PowerMoE-3B has shown promising results compared to other dense models with 2x activate parameters across various benchmarks, including natural language multi-choices, code generation, and math reasoning. Paper: https://arxiv.org/abs/2408.13359 Note: Requires installing HF transformers from source. This is a simple example of how to use **PowerMoE-3b** model. import torch from transformers import AutoModelForCausalLM, AutoTokenizer device = "cuda" # or "cpu" model_path = "ibm/PowerMoE-3b" tokenizer = AutoTokenizer.from_pretrained(model_path) model = AutoModelForCausalLM.from_pretrained(model_path, device_map=device) model.eval() prompt = "Write a code to find the maximum value in a list of numbers." input_tokens = tokenizer(prompt, return_tensors="pt") for i in input_tokens: input_tokens[i] = input_tokens[i].to(device) output = model.generate(**input_tokens, max_new_tokens=100) output = tokenizer.batch_decode(output) for i in output: print(i)