Question on a passage in Golden Fool

[Elderlingbosgy]

I was wondering what you think Fitz is trying to remember in this passage from Golden Fool on p. 71-72 of the Bantam mass market paperback:

Dutiful gave me a very direct look and he seemed to speak with Verity's bluntness as he asked me 'Could you put your duty to your monarch ahead of protecting a member of your own family?That is what I asked myself? If my mother were threatened, what could I be forced to do? Would I betray the Six Duchies for the sake of her life?

Lord Golden shot me a Fool's glance, one that was well pleased with this boy. I nodded to it, but felt distracted. Dutiful's words itched at me. I suddenly felt there was something important I needed to remember but could not trace the thought any further. I could not think of an answer to Dutiful's question either, so the silence lengthened.

I don't think he would have forgotten when he was pressured to choose between serving King Shrewd and his love for Molly. What could it be then? I was thinking it's a suppressed memory of his mother. Thoughts?