The name Shemini Atzeret is derived from the verse "On the eighth [Shemini] day you shall have a solemn assembly [Atzeret]" (Numbers 2.9:35).
It is defined by the sages as "a festival in itself" (Sukkah 47a), so even though it comes immediately after Sukkot, it is a separate festival in its own right, and its laws and prayers are not related to those of Sukkot.
Our sages see this festival as a private celebration between God and Israel.
On the festival of Sukkot, we pray on behalf of the entire world, whereas Shemini Atzeret is like a "small banquet made by the King for His beloved" (Rashi" on Numbers 2.8:35).
This festival also marks the beginning of the winter season, for on it we begin to pray for the rains of the coming year.
Perhaps because of this special significance, it was decided to connect this festival with the ceremony for completing the yearly cycle of Torah readings and with the festivities for Simhat Torah–"the Rejoicing in the Law."
(This was probably introduced during the Geonic period, because it is not mentioned in the Talmud.)
Both in Israel, where Shemini Atzeret and Simhat Torah are celebrated on one and the same day, and in the Diaspora, where Shemini Atzeret is celebrated on one day and Simhat Torah on the next, there is a growing tendency (in customs as well as inward feeling) to have the aspect of Simhat Torah override the aspect of Shemini Atzeret.
Simhat Torah is an expression of Israel's joy in having the merit to complete the Torah readings, and of gratitude for having received the gift of the Torah.
It is true that we celebrate the Giving of the Torah on the Shavu'ot festival; however, on that occasion the emphasis is on the great event of Divine revelation and its awesome and sublime significance.
On Simhat Torah, there is greater stress on joyful, grateful acknowledgment of the gift of the Torah, and for having merited to receive it and live by it.
–Rabbi Adin Steinsaltz
From A Guide to Jewish Prayer by Rabbi Adin Steinsaltz