There is an interesting analogue to this in how the brain processes visual information. Visual information processing can be broadly differentiated into a what system (concerned with object recognition and color recognition) and a where system (concerned with location, movement, depth perception and figure-ground organization etc). The where system is actually color blind and this gives rise to some of the most interesting phenomena in art. For example, the impressionist technique of closely placing complementary hues of similar luminance gives rise to a sense of movement is closely related to the fact that the where system being color blind ignores the hue but the what system does proces it and this creates a spatial imprecision that makes the brush strokes seem to 'jitter' around, creating a sense of movement or fluttering etc. There are other differences between the two systems - the where system is more sensitive to contrast and is faster, relative to the what system. There is a very large amount of visual information processing that is completely achromatic or colorblind in that sense. There are a couple of reasons offered for this kind of segregation. One is evolutionary - the where system is similar in most lower mammals, which seems logical given that the functions it serves have primitive biological significance. Movement is important for survival in a predator-prey context. Spatial and depth information and figure-ground segregation is important to navigate a three-dimensional world. The evolutionarilly more recent what system was thus probably overlaid onto this earlier system and in that sense, the what system is a primate add-on. The second explanation is that it is more efficient (or parsimonious) to carry information about an object's appearance (shape and color) separately from its motion and trajectory. I find it interesting that engineers in more recent technologies (e.g., HDTV) have arrived at similar strategies to transmit images. Rather than redefine every pixel repeatedly, more efficient strategies are to redefine only those pixels that change or to define an objects shape and color separately from its motion and trajectory, echoing the subdivisions of our visual systems.
So, I suspect this is a very personal decision to do with what visual concerns you are trying to deal with in your art/photography. I paint and that always seemed a more natural way to deal with color concerns for me. What interested me with photography as a medium had to do with the density of information that it conveyed (see Ivins' "Prints and Visual Communication"). Once I started following along those trains of thought, I realized just how much visual information was achromatic in nature and B/W photography became a natural medium to explore these issues. I was also becoming uninterested in 'things' but was more interested in the 'relationships between things' and much (though not all) of that seemed to be achromatic in nature. Finally, as much as we live in a modern world of acrylic gloss, the truth is we inhabit a fairly 'grey' world, for the most part. Saturated color is the exception rather than the norm and I was a little uninspired with color photography which seemed mostly about searching for saturated color. I think Edward Weston's advice to Cole about 'seeing color as form' is worth pondering. I find much (not all) of color photography degenerates into a sort of 'making portraits of things' which I'm not terribly interested in, but that may just be betraying my visual concerns.