I don't have an Xbox but I think you can go to Settings -> Kinect and disable that part of it.
Also for headsets I think in the Privacy Settings "text and speech" you can set it to friends only I believe.
----
Posted in wrong section? meant for PC?
http://www.crashwiki.com/pvz-garden-warfare-disable-chat-disable-mic-use-push-talk-use-voip/
Right click speaker icon in system tray.
Select Playback Devices.
Select a Playback Device other than what you use for sound output.
Right click it and select Set as Default Communications Device.
Hit OK.
Left click on the speaker icon in the system tray.
Click the icon below the slider on the device you chose to mute it.
Do the same for your Input. Change Skype, Teamspeak, Raidcall, Vent, whatever to use the correct devices.