There is no config option available yet. I agree it could be usefull even on a project basis.
For now you need to adjust the code.
Please check core/filter_api.php
Removing filter fields
Moderators: Developer, Contributor