The problem is that the more files you have in the cache, the more work the browser has to do to find them. I just checked my cache and I have about 1,000 files in 20MB, most of which are small images. I think 20MB is about right for a high-speed connection (DSL or cable). It's possible that if you're using dial-up (e.g., 56Kbps) then more would be helpful, but I doubt that more than 50MB makes sense.The link below is to a chapter from one of the "Dummies" books, which has some more discussion and recommends a size from 35MB to 75MB, again depending on size.
www.dummies.com