FMEngine#

FMEngine is a utility library for training very large foundation models. The goal of fmengine is to provide the following:

  • Ergonomic interface for training foundation models. It is sufficient easy for a beginner to use, but also provides enough flexibility for advanced users to customize their training.

  • Efficient optimizations built in. FMEngine is equipped with Flash Attention and various fused ops to accelerate training.

  • HPC-friendly installation with pre-built docker and singularity/apptainer containers. FMEngine is mainly designed and tested on Slurm clusters. We provide starter scripts for running FMEngine on Slurm clusters.

  • Compatible with existing frameworks and tools, particularly with HuggingFace. Since FMEngine is built with DeepSpeed, it is also compatible with all DeepSpeed features.

Acknowledgement#

FMEngine is primarily implemented and maintained by the Efficient Architecture and Systems Labs @ ETH Zurich.

We thank our friends for their generous support:

Contributors#