VirtueMart uses the email settings of joomla. So the basic configuration is done in the joomla "Global Configuration" view (tab server).
The Vm Config provides more virtuemart specific settings.
"mail format"
The normal format is html. Of course most email clients are not capable to read complex html, here is a nice listing of best practice for designing emails http://templates.mailchimp.com/getting-started/html-email-basics/.
It is also possible to use the text email. Customisation of the text is then done by the language override system of joomla.
"Use the vendor email address"
allows to use the email of the vendor instead of the joomla account settings. This is interesting for multivendor enviroments and needs usually an own webserver, because otherwise these emails are often handled as spam.
"Which orderstatus should sent an email"
In the tab checkout provides settings to configure if an email is sent to the shopper respectivly vendor depending on the orderstatus. Paymentmethods can override the global settings and use their own configuration.
The order status "Pending" is for internal purposes. For example some payment systems need an order to display the fee. Almost any payment system want an order number and so on. When an order status keeps on "Pending", then it means that the customer left the cart without confirming the order.
Troubleshooting
No email is sent
VirtueMart relies on the joomla mailing system. Therefore it is very important to ensure that the normal joomla emails are sent. The feature "mass mailing" or the normal joomla registration are a good way to test if the normal joomla email works. When the normal joomla email works, then it can be one of the following reasons.
No email sent to shopper
This is usually combined with the email which should sent the invoice to the shopper. When the pdf creation breaks, it is very likely that further emails are not sent. In this case it is important to check the pdf creation Invoices