Unless there is a risk of a collision between the OSP attacker and the GK, I would say the observer was absolutely wrong to tell you that you should have raised the flag as there was no OS offense. (I wonder if the observer had a different sense than you did about the possibility of a collision?)I once did a similar thing. An attacker was chasing the ball from the halfway but the ball had so much pace on it that it was going straight to the GK. I decided there's no point putting the flag up, just let it roll to the GK.
Post match I was told by an observer I still should have put the flag up because everyone in the ground is presuming he is onside when your flag stays down (even the observer said he wasn't sure whether I'd spotted he was actually offside), so put it up and then the referee can wave you down when it lands at the GK
I had the flip side of this when starting to do high school games and being assigned to a very experienced partner for my first game. (Yes, yes, I know, two whistles on the field is satanic, blah, blah, blah, but it's what happens in US high school a lot.) After the game my partner had one question, asking me why I called a particular OS as the attacker never interfered. I told him there was a possible collision, and his answer was basically, good, just wanted to be sure you understood that. In the same way, IMO, the observer should have been asking you what you had on that play, not wondering if you spotted it, as you appear to have properly kept the flag down. (Again, unless the observer thought there was a good chance the attacker would get it or have the possibility of a collision with the GK.)