I was being very slightly oblique, I guess. But, pointing out that 'black' is understood to be in the general category of colors, and 'zero' in the general category of numbers, by practically everyone whether or not they accept the premise that they actually are members of those categories.
The counterexample, 'potato', is something that is clearly not a color (nor a number) in that it would not even be a sensible answer to a question about color or quantity.
The whole point of this is that, when we feel a need to actually *say* that black is not a color, we are saying that, whatever we think it is, it's pretty close to the idea of a color. It's something that would be a sensible answer if we ask what color some object is. The only sense in which it's not a color is that it's not the color of any wavelength of light.
For much of human history (up to some time in the middle ages, in Europe) people felt the same way about 'zero' being a number. If someone asked how many of something there were, 'none' was a non-numeric out-of-band response, indicating that the question was in error. But at some point the Arabic numerals were adopted, and 'zero' became a way to say 'none' that was numeric, in-band, and didn't require anyone to reformulate a question or deal with an error. And then we had the same fight about negative numbers, and then about imaginary and complex numbers, etc.
So... It's up to everybody whether to accept the idea that 'black' is a color or not. I'm happy to go around thinking that it is. For the same reason that most people think that having an opinion about whether it is or not even makes sense. It's just the 'zero' of the color spectrum.