First of all, directly editing sendmail.cf is not a recommended procedure. The sendmail.cf file should be treated like a binary file unless you really, really know what you're doing and even then it's not a good idea. Edit the macro configuration file /etc/sendmail.mc and use m4 to generate your sendmail.cf file. Your problem here is clearly expressed in the error message. The data type you are attempting to use for your mailertable does not exist on your machine. To fix this, edit /etc/sendmail.mc and search for something like this line:
FEATURE(mailertable, `dbm /etc/mail/mailertable')dnl
then change it to look like:
FEATURE(mailertable, `hash -o /etc/mail/mailertable')dnl
Regenerate your sendmail.cf file with m4 and restart sendmail.