Zero Redundancy Optimizer (ZeRO)
AIZero Redundancy Optimizer (ZeRO): A method that removes duplicated data across devices by sharding model states, greatly reducing the memory needed to train large models.
Zero Redundancy Optimizer (ZeRO): A method that removes duplicated data across devices by sharding model states, greatly reducing the memory needed to train large models.