Please refer to this post of mine on how to configure Forms Based Authentication (FBA) on a SharePoint Web Application.
Consider the following scenario:
You have configured FBA on a Web Application and created a Site Collection that uses FBA. Your Web Application uses mixed mode authentication (FBA and Windows Authentication). When you search for users in People Picker it shows you matches for both FBA and AD users. You would like to restrict People Picker to only show matches for FBA Users.
You can achieve this by running the following stsadm command:
stsadm -o setproperty -pn peoplepicker-onlysearchwithinsitecollection -url http://www.nameOfMySiteCollection.co.uk/ -pv yes
The ‘peoplepicker-onlysearchwithinsitecollection’ part in there is misleading and doesnt really do what it implies, however, it does get rid of the AD user mataches from People Picker.
Please note that if you type in the full username of an AD user and click on the Check Names icon it will still resolve the user.