Gated linear unit (GLU)
AIGated linear unit (GLU): A layer that multiplies one linear projection by a sigmoid-gated second projection, letting the network control which information passes through.
Gated linear unit (GLU): A layer that multiplies one linear projection by a sigmoid-gated second projection, letting the network control which information passes through.