These control the volume reduction of a stem when the player misses notes/fails out
Number value is in decibels

Left over from Beatles Rock Band (to hide artifacts of stem isolation). Still works!

If your song has 'fake' stems produced by frequency banding, you can use this to make the game mute the stems less harshly when players make mistakes or fail, in order to prevent the overall sound from becoming too deadened
The singer has a separate setting from the rest of the instruments

Setting this to 0 will cause no muting when notes are missed, even with multitrack audio. For the singer, this will keep the vocal track playing even if the singer fails

The default in-game value for instruments is -96.0; For vocals it's -12.0
These are the values the game uses if no value is specified

Don't use positive numbers. Don't go lower than -96.0

Manually reducing the vocalist's volume with the game's options will still use the default values. Custom mute_volume_vocals values only effect the muting volume when the singer fails out
But who still plays with fails enabled anyway?

Many thanks to OrangeHarrison for this information