Like nearly everyone using a computer for visual design, I like to keep a folder of images I have grabbed off the internet. Here is the directory structure I use for the things I find most visually interesting. The underscores mean that even if end up dragging a load of rubbish into the main folder, it is easy to sort at a later date.
This helps me start the day with a nice tidy desktop. Plain grey, no dashboard, a couple of aliases to my most used shared folders and whatever I am working on at that time. (below)
CONCLUSION: Keep all of the chaos. It sustains the colour and the energy in your work, just, for sanity’s sake, find someway of containing it.