Why doesn't Hamlet kill Claudius when he hears him confessing?
Because he wants Claudius to go to hell, and assumes that that cannot be accomplished if Claudius is praying and repenting for his sins.
In Hamlet, it is established in the canon that if one prays and repents their sins, that they will, undoubtedly, go to heaven. However, when Claudius murdered Hamlet Sr., this was an act done in surprise/sneakily, and Hamlet Sr. did not have a chance to repent for his sins, thus ending up in hell.
Hamlet does not kill Claudius when he assumes that he is praying because he doesn't want Claudius to have the luxury of going to heaven while his father, unjustly murdered, suffers in hell. So Hamlet resolves to catch Claudius when he is doing something sinful, like drinking or swearing, so that he is forced to go to hell. He doesn't want to do Claudius the "favor" of sending him to heaven.
However, the ironic thing about this scene is that Claudius is not really confessing at all, but soliloquizing to himself, and it would've been the perfect moment for Hamlet to do it. I hope this helps!