Nx1.3B Moe support？

[sdli1995]

Does will train from draft for a little moe llama ？

[ymcui]

We are indeed interested in creating smaller MoE models, but it is unlikely that we will train from scratch, as this is extremely computationally expensive.