How does Hamlet react to Ophelia when she returns his gifts, and why?
This depends on how one decides to produce the scene, but normally, Hamlet reacts in a very venomous and angry manner.
When Ophelia attempts to return Hamlet's "tenders of affection", firstly, Hamlet denies ever having given her any gifts -- " No, not I. I never gave you aught ." (III.I.105)-- and then, as Ophelia insists that he did, immediately counters her offering with a fierce question: " Ha, ha, are you honest?" (III.I.113).
Hamlet's motivations and thoughts here are both questionable and subjective here, which is why I previously said that how, exactly, Hamlet reacts depends on how the particular Hamlet is choosing to motivate Hamlet in this scene. But we do at least assume that Hamlet is feigning insanity here, due to the fact that Hamlet claims never to have given her anything, when its been established that he definitely has.
However, I think that the answer to the "why" question you ask is because Hamlet is primarily angered at the thought that Ophelia has taken the side of Claudius and Polonius and not his own, and is taking out his pent-up aggression towards both womanhood and fate itself on her.
Just as well, Hamlet's question of Ophelia's honest can go two ways: he can mean "honest" as in to question if Ophelia is telling the truth right now (as opposed to being puppeted by Polonius and Claudius, which, in reality, she is), or if she is chaste, or sexually pure. Either interpretation is valid, given that Hamlet's madness is, for the most part, feigned, and that he later makes several sexual references toward Ophelia.
For the most part, though, I would say that Hamlet's reaction to Ophelia attempting to give him back the letters is venomous because its what spurs Hamlet's infamous and wrathful "get thee to a nunnery" speeches. Hamlet gets unbelievably angry at Ophelia's betrayal here that, to me at least, his behavior can only be described as venomous.
Here are some different productions of this scene, so that you might see for yourself! :)