However, Pakistan Cricket Board (PCB) recently retained Sarfaraz as the captain in the 50-over format and appointed Babar Azam as skipper in T20Is.
"It is a correct decision to appoint Sarfaraz as skipper in One-Day Internationals (ODIs)," cricketpakistan.com.pk quoted Afridi as saying while speaking to reporters on Thursday.
"Sarfaraz should have been removed as Test captain after the South Africa tour as it is a difficult challenge to lead the side in three formats.
"I had advised Sarfaraz to leave Test cricket after the South Africa tour irrespective of the result, but he didn't listen," he added.
Earlier, former Pakistan greats Zaheer Abbas and Mohsin Khan had also expressed the same opinion about Sarfaraz and how he should relinquish Test captaincy.
Pakistan are slated to play three ODIs in Karachi on September 27, 29 and October 3, and as many T20Is in Lahore on October 5, 7 and 9 against Sri Lanka at home.