How to determine which Form XObjects can be cached?

I have a little problem with Form XObjects.

To speed up rendering, I going to use caching.

However it does not working as expected, because some of them rendered each time with different settings (e.g. another fill color).

Imagine we have colorless Rect XObject and RedRect, GreenRect, BlueRect XObjects which just fills Rect with appropriate color (see below):

If I cache Rect XObject, RedRect, GreenRect and BlueRect will have same color.

Any ideas?

21 0 obj
<</Type /XObject
  /Subtype /Form
  /FormType 1
  /Length 22
>>
stream
0 0.25 16 0.75 re
B
endstream
endobj

23 0 obj
<</Type /XObject
  /Subtype /Form
  /FormType 1
  /Length 35
>>
stream
0 0 1 rg

0 0 0 RG
/Rect Do
endstream
endobj

24 0 obj
<</Type /XObject
  /Subtype /Form
  /FormType 1
  /Length 35
>>
stream
0 1 0 rg

0 0 0 RG
/Rect Do
endstream
endobj

25 0 obj
<</Type /XObject
  /Subtype /Form
  /FormType 1
  /Length 35
>>
stream
1 0 0 rg

0 0 0 RG
/Rect Do
endstream
endobj

4 0 obj
  <</Type /Page
/Contents 5 0 R
/Resources

            /XObject
         <<
           /Rect 21 0 R
           /BlueRect 22 0 R 
           /GreenRect 23 0 R 
           /RedRect 24 0 R 
         >>
      >>
  >>
endobj