Does my sentence sound natural?

"Take him down from office" doesn't sound natural. I would use "to remove him from office" or "have him removed from office".

“To have him removed from office” would sound more natural. “Take him down” would work too, but would be much more informal and would be more appropriate if your audience was young people

The \[opposition party\] spread a rumor about the president to take him down. No need to say "from office" at the end. That's understood in context. And as for "opposition party"... they would say the name of the party. Opposition party is an accurate and real term that makes sense... but is actually used quite rarely in common speech.

It would sound more natural if you just ended the sentence with “to take him down.”

A native speaker would probably say “opposing party” instead of “opposition party”, but other than that your sentence sounds natural