Since fluorine is the most electronegative element, the difference in electronegativity between itself and hydrogen will be the biggest of the group.
In order for a hydrogen bond to be formed between two molecules, you need to have an attraction between one molecule's electron-poor hydrogen atom and another molecule's lone pair of electrons.
So, the bigger the electronegativity difference between hydrogen and the halide it's bonded to, the greater the partial positive charge on the hydrogen atom.
Here's where atomic size becomes important. The smaller the atomic size of the halide, the more negative its lone pairs of electrons will be.
Think of it like this. As you go down group 17, the lone pairs will occupy increasingly bigger orbitals due to the increasing energy levels on which they are added.
This implies that their negative charge will be spread out on greater areas and thus less concentrated.
So, two factors go hand in hand here. Less electronegative halides imply a smaller difference in electronegativity with hydrogen.
This in turn will result in a smaller partial positive charge on the hydrogen atom. A smaller partial positive charge on the hydrogen and less negative lone pairs lead to a much weaker interaction between one molecule's hydrogen and another molecule's lone pairs