For now, you want to fuck someone

You realized that while Inoue wasn't lying about her being unable to fuck you, she didn't exactly say that you can't fuck her. To be more accurate, while she has to tell you that her business relationship with your family prevents her from taking advantage of the situation, that rule has a grey area. If you force yourself on her, she could claim that it couldn't be avoided. And judging by the small wet patch on the crotch are of her robe, Inoue wasn't lying about her being tempted.

From what you know, she was edging, bringing herself to the brink of orgasm but stopped short of crossing it because she's seducing you.

But you were thinking about fucking another girl in the brothel, not her.