I've experienced this, even though my mask did not contain additional colours beyond those defined in layers.cfg
Most noticable between detail textures that are noticeably different in colour\tone and on terrains using high resolution sat\mask maps where the blending between different layers is sharp as opposed to smooth
Generally playing around with the colour of detail textures will mimimize it..
edit: the above only applies if you've ensured that your mask contains no colours beyond those defined in layers.cfg, no aliasing or smooth transitions that add additional colours to the mask or more than 4 layer types per segment