This is a limitation of the GIF format.
GIFs don't support a (smooth) alpha channel. Instead, it only supports a single transparent color. This means if you use transparency, you can't have smooth edges.
https://en.wikipedia.org/wiki/GIF

If you want it to be smooth, disable transparency.
If you want transparency, you can't have smooth edges.