This solution is for ArcGIS 9.3
Happens because 32 bit apps are not allowed to use more than 2 gb of RAM.
Here the step by step solution:
1. Install Microsoft Visual Studio 2008 (we need its command prompt to Unlock the 2 gb Limit form ArcMap)
2. Go to start menu and run the command prompt from the Visual Studio Folder.
3. Type, editbin /largeaddressaware "c:\program files (x86)\arcgis\bin\arcmap.exe"
4. ArcMap should not be running.
5. This should remove the error.
for any problems email me at me_faizananwar at yahoo.com