Of course the "whole thing" had romantic and sexual implications. Two single adults of opposite sexes, meeting alone to talk, drink some wine. Most people of whatever gender would see this situation as at least a preliminary to moving towards intimacy.
Possibly if you shared a strong interest in a particular hobby or cause there would be reason to rule out a more personal connection as a potential from "getting to know each other", but even then, that kind of interest sharing is often the basis for couples forming pairs.
Jumping from these obvious facts to a debunking of the provocative clothing rape excuse is a stretch indeed. As you said, your social time with this guy was entirely platonic and he made neither sexual advances nor hints. Obviously no issue of "uncontrollable urges" or of clothing choices leading to rape.
Wondering, if the roles had been reversed, if you had been working out in the garden perhaps in worn work jeans with dirt on the knees, and you had accepted an invitation from him to come over to talk, drink wine, get to know each other better, how much would you have cleaned up?
Are you saying that the idea of developing a closer physical relationship would have NEVER crossed your mind, even just to rule out that possibility?