I have this problem now and again too. With BF3 running (on rare occasions) I get 99% usage of all 16GB of memory I have on my system!
Here are my stats:
OS: Win 8 x64
RAM: 16GB
CPU: i5 3570K
GPU: ATI RADEON HD 5670
Originally played on W7 with no problems whatsoever for many months. Nothing in the Task Manager telling me how the memory is being used so assumed dead process. I have only viewed after closing BF3 but I monitored usage ingame for a week and found it did not happen so it is a rare occurance. Usually , when it happens , I have been playing for a few hoursbut it seemed to happen within a few minutes today. My Video drivers are fully upgraded and I use a desktop with all the usual power settings off. I was on a forum on tomshardware at some point and I came accross around 10 people with the same issue.
http://www.tomshardware.co.uk/forum/840-73-windows-extremely-high-memory-usage
http://hardforum.com/showthread.php?t=1675042
http://www.mmo-champion.com/threads/1079173-97-RAM-usage-in-Battlefield-3-is-8GB-the-sweet-spot-for-gaming-now
http://forums.overclockers.co.uk/showthread.php?t=18333705
http://battlelog.battlefield.com/bf3/forum/threadview/2832654625233545341/
http://battlelog.medalofhonor.com/mohw/forum/threadview/2832654348314439337/
http://answers.ea.com/t5/Battlefield-3/Battlefield-3-overloads-RAM-using-Windows-8/td-p/41716
Even after some idle time , the system still has 16GB usage so I will need to restart to stop the usage and , if last time is anything to go by.
This is a widespread problem and since Win 8 is similar to Win 7 it should be fine , but I also think that the problem is pointing to an issue with the ATI and nVidia drivers partially.