Last month, Gingrich had tweeted: "Robert Mueller is superb choice to be special counsel".
But Kenneth Starr, who investigated the Whitewater and Monica Lewinsky scandals as an independent counsel during the administration of former President Bill Clinton, rallied to Mueller's defense. According to current regulations under which Mueller was hired, only the attorney general can fire a special counsel.
Until now, Mueller had drawn widespread praise from Republicans and Democrats alike.
However, Trump could follow the example of Richard Nixon, who eventually got rid of Archibald Cox, the special prosecutor during the Watergate scandal, in October 1973.
But expressions of discontent with Mueller are bubbling up nonetheless.
Three members of the legal team known to have been hired so far by Mueller have given political donations nearly exclusively to Democrats, according to a CNN analysis of Federal Election Commission records.
CORNISH: Is there a concerted White House strategy to go after or discredit special counsel Bob Mueller?More news: Key takeaways from Attorney General Sessions' testimony
Peter Carr, a spokesman for Mueller, declined to comment on Ruddy's remarks. In general, it is impermissible to ask prospective government hires about their political leanings, including whether they've made donations.
Mueller has already served 12 years as Federal Bureau of Investigation director - two years longer than the 10-year term.
The White House pushed back, but only hours after the claim had been parsed and dissed with warnings about how it would only make things worse for Trump, who chose to ignore it all together in a series of tweets on Tuesday morning, and seemed more focussed on a fresh setback to his travel ban.
The comments came a day after a close friend of President Donald Trump was quoted in a television interview as saying the president was considering dismissing Mueller. The person demanded anonymity to discuss strategy on the sensitive matter. The interview was first reported by PBS.
Ruddy said the ethics behind Mueller's presence are concerning considering how he was appointed to be special counsel. The White House has used a memo he wrote to justify Trump's decision to fire FBI Director James Comey, but that Trump action may now be part of the probe.
Trump can not directly dismiss Mueller. "I think that Mueller should not have taken the position if he was under consideration and had a private meeting with the president and was privy maybe to some of his thoughts about that investigation or other matters before the bureau".
Mueller's investigation will oversee an FBI probe into whether members of Trump's campaign team and his associates colluded to help Russian Federation with its interference in the election.