Some food for thought:
St Joseph the Betrothed is always referred to liturgically as the Betrothed, not the Husband or Spouse of the Theotokos.
The Virgin is never referred to liturgically as a wife, only as a mother. St Joseph is shown in icons of the Nativity as some distance away from the Virgin and newborn Child, pensive, pondering this awesome and incomprehensible mystery. This distance also illustrates that he played no part in the conception of this Child. Consider Joseph's situation: Joseph would have been familiar with what we call OT scripture. Exodus in particular is stuffed full of terms and imagery which we know are prefigurations of the Mother of God. Mary bears the Root of Jesse, the Bread of Heaven (John 6), the Word of God (John 1). The Ark contains the rod of Aaron, Manna and the Law. Mary is the human Ark of the New Covenant, a constant motif in both liturgical language, and in the iconography of all the feasts of the Virgin (the four-posted structure with a domed roof).
Now, Joseph was a good Jew, he would have been brought up with a strong sense of the sacred. He would have been raised knowing the stories in scripture of people touching the Ark of the Covenant and suffering instant death. He would have also known that only the high priest dared enter the Holy of Holies of the Temple to offer the yearly sacrifice to the presence of God who "dwelt there". Undoubtedly at some stage Joseph would have been inspired by the Holy Spirit to realise the true meaning behind these images and stories from scripture, as well as the temple rituals. Once the meaning of these became clear to him, how, then, could Joseph possibly consider marital relations with this woman, the living Tabernacle, the new Ark, the Holy of Holies, knowing that she has given birth to the Son of God? Not that sex is bad, evil or wrong between married couples, just as eating and cooking meat are not bad, evil, or wrong in themselves, but when put into service to God in the Temple, be it sacrificial animals, or, in the case of this woman who was dedicated to the Temple as a child, they became holy, and only the high priests could participate in the sacrifice. Christ Himself is the great and eternal High Priest, the "prince who eats bread before the Lord" (Ezekiel 44). Good man that he was, Joseph would most likely have regarded himself as utterly unworthy to even be in the presence of such a treasure blessed by God, let alone consider sleeping with her.