My boyfriend doesn't believe he was my first!?

We've been together for about 4 years but broke up in 2005..we were back together in November 07. Let me just say it like this we had ual , i did not bleed. Now hes blaming me that i had with some one else. However, i do remember that after 2 days i saw some brown reddish discharge. I feel so upset he has hold this in himself, i mean he told me the same day it happen "he knew that hes my first" and now Thiss!!!?? To women, has this happened to you?