What actions made Ngo Dinh Diem an unpopular leader in South Vietnam?
Probably the fact that he (being fervently catholic) favored Catholics and shunned Buddhists.
Diem was not even a bad political leader considering that before becoming President he was a (efficient and workaholic apparently) functionary in the old French colonial administration.
The problem was that he was a fanatical Catholic (he even spent some time in a seminary in the US where allegedly he was more fervent than the priests themselves!) that had difficulty in accepting other religious groups. The Buddhists in particular resented the fact that he was trying to limit their freedom of cult and not allowing them to celebrate their festivity or having their ceremonies.
The worst was that he used his brother's "secret police" (basically, thugs) to harass and persecute the Buddhists.
As icing on the cake, his brother's wife, the famous Madame Nhu, was a kind of vitriolic figure (caustic and vehemently offensive) ferociously opposing the Buddhists. She used to give statements to the press that always were more damaging than useful to Diem such the one after the public suicide by fire of the Buddhist monk Thích Quảng Đức where she retorted: “If the Buddhists want to have another barbecue, I will be glad to supply the gasoline.”!!!
In addition, Diem surrounded himself of “loyal” relatives (to maintain his power) quite unpopular with his people (Buddhists and officers of the Army worried of the increasing power of these figures) and even among the Americans.