Why we aren't really OKAY?

I am angry.
It feels like I was his friend, but he was never mine. I talk to him, he listens without a reaction. He would hardly ever share anything about himself. I am not allowed into his comfort zone.
I would (willingly) go places for him, and I naturally expect him to be giving. I would never intentionally annoy him, hence, I never thought my tolerance would be used. There are times, I wanted and tried to make him feel special. But I was never on the receiving side of that equation.

"Are we okay?" He asked.
"No we are not" I answered.
"I am sorry."
"It is not about apology. Apology doesn't mean that I would automatically feel better."
"But you cried over my shoulder."
"It doesn't mean that the bad you have done has been undone."
"What do you mean?"
"I mean you keep on building walls between us with your attitude. And I go places to bring them down. Guess what, this wall is for you to practice breaking the ice."
He softly touches my shoulder saying: "It is okay."
I want to scream from the top of my lungs: "It is not fucking okay." But I don't voice out this feeling. I just say: "If I said what is on my mind, you would be hurt."
The ideas keep running back and forth in my head faster than the speed of light. I think maybe I should tell him what I really think and enjoy the relief and lose him forever. Or perhaps, it is just wise to let go of my anger at the gym and come back feeling better.

What a million hugs cannot compensate for is the fact that I felt like the worthless secondary option he has after he is done with the more interesting things in life. That's what I meant when I cried and asked: "Why are you doing this to me?"
I could be very good at coaxing him and lifting his spirits when he allows me. I can't say that my tricks would work on me, if he ever bothered test them. However, I would say that I can't bring this last wall down on my own and I don't know how to have it broken down.

"We ARE okay, right?" He asked.
"No we are not" I answered.