There are 2 different ways for a browser to show a webpage: the quirksmode (sometimes it's called the 'compatibilitymode'), or the standards-mode. When there's no doctype specified, (actually, that's not allowed, as the doctype is a required element of every html-page), the browser will switch to quirksmode automatically. In the past, it wasn't required to specify a doctope (it simply didn't exist). So if you don't add a doctype, the browser thinks your website is made for an older browser.
As the browser thinks your website is made for an older browser, it'll simulate bugs that occured in older browsers. So he deliberately displays your website not correctly.
A lot of websites without doctype do are very old. When they were made, they were tested on an older browser. At that time, that browser had some bugs, so a lot of people did't made there websites according the web standards, so the website was showed well by the browser. Later, that bugs were solved. Now, the browsers do are according the webstandards, but the websites aren't. Due to this, websites won't be rendered correct, and that's why the quirksmode is there: in the quirksmode, the browsers pretend to be old browsers (with the bugs), and the website will be rendered correctly.
For a new website, which is according the webstandards, the quirksmode is not useful at all. That's why you should always specify a doctype. To do this, you should put the following line at the top of your html-page:
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
<title>Title of your website</title>
</head>
<body>
Content of your website
</body>
</html>