Not merely the genetics. The physiological implications of human/giant crossbreeding are quite scary.

Who knows - maybe Hagrid is infertile? ;-)

I don't think Rowling will ever elaborate whether Muggles can breed with magical creatures or not. I could accept that one needs to have magical blood to interbreed with a different species. On a similar note: can humans turn into werewolves, as well? Or does this require inherent magical abilities? Questions which, I'm sure, won't be answered in the books. That's what fandom is for.