The change of Trump's top diplomat comes as the US is engaging in delicate, high-stakes negotiations with North Korea over its nuclear program and the prospect of Trump meeting Kim Jong Un this spring.
Speaking to reporters at the White House, Trump said he respected Tillerson's "intellect" and said he "got along well with Rex."
"I think Rex will be much happier now," Trump said.
In tweeting the announcement earlier Tuesday morning, Trump announced that Gina Haspel would be the next CIA director.
White House press secretary Sarah Sanders told reporters that Trump asked Tillerson to step aside.
Trump "thought it was the right time for the transition with the upcoming North Korea talks and various trade negotiations," a senior administration official said, adding that Trump asked Tillerson to step aside on Friday.
Tillerson did not speak to Trump and is unaware of the reason behind his firing, Undersecretary for Public Diplomacy Steve Goldstein said.
Trump has wanted Pompeo as his secretary of state for months now, and the White House began planning for him to take the job last fall, sources told CNN.
Trump's anger at Tillerson after it leaked that his secretary of state called him "a moron"
never subsided, and many in the White House saw their differences as irreconcilable.
Tillerson had few, if any, allies in the West Wing. Though chief of staff John Kelly was initially on his side when he took over, he eventually grew weary of defending him -- especially after the "moron" remark, which Kelly saw as insubordination on Tillerson's part.
Sources close to the President say it was clear Tillerson didn't support Trump. They say Tillerson wanted to handle foreign policy his own way, without the President. Trump didn't feel that Tillerson backed him, a source told CNN.