You have given Augustine's explanation, which is perhaps the best explanation from the point of view that hopes to "rationalize" the Bible, but unfortunately, I do not believe the Bible is rational on this point. The reason is simple: From early times Jews interpreted Genesis 6 as fallen angels having sex and children with mortal women. Now this wouldn't matter so much if the same idea did not get confirmed in the NT book of Jude, which quoted the part of Enoch that talked about the fallen angels having sex with women!
So I see no way out of the obvious implications. The book of Jude explicitly and repeatedly quotes the book of Enoch which says that Genesis 6 was talking about fallen angels having sex and children with mortal woman, so this must be the meaning intended by Jude.
You apply the same fautly reasoning to the Letter of Jude as you do in Genesis, so you argument is equally bad. Adding two negatives does not make positive, you still have negative. Your reasoning and conclusion are all negative.