[Home | Contact | What's New? | Products | Services | Tips | Mike |
Living with Schizoaffective Disorder

Please to Forgive

This site totally sucks when viewed on a smartphone.
I'll fix this Real Soon Now.

White Hat
Search Engine Optimization

A few simple, honest practices can dramatically increase your search engine referrals
by making your pages more appealing in search results.

Michael D. Crawford

Michael David Crawford, Consulting Software Engineer
Solving The Software Problem
mdcrawford@gmail.com

August 2, 2005 - Second Draft

Copyright © 2005, 2012 Michael David Crawford.

Note: I will be expanding on this article soon, first to discuss keywords, and then to discuss paid advertising.

Why Optimize?

I often see tales of woe posted on messageboards by webmasters who invested everything they had into their websites, then worked for months or even years with little to show for it. Why do some succeed when others fail?

The success of one's website depends on many factors but one of the most significant is how well it ranks in the search engines for the keywords relevant to the content of one's site. If one's site is in the top ten, so that it appears on the first page of search results, the search engines refer a great deal of free traffic. If one's site only appears after clicking through page after page of listings, there will be few search engine referrals and one must depend on expensive advertising to draw traffic to one's site.

Search engine optimization, or SEO is the practice of improving the design and content of websites so draw more search engine traffic. As we shall see, there are ethical and unethical ways to accomplish it. I advise you not to hire anyone who does not pratice ethical SEO. You should be able to make the most important optimizations to your website yourself after reading this article, and if you are a web designer, reading this will enable you to offewr ethical SEO to your clients. If you plan to hire a consultant to optimize your website, reading this article will help you understand what they will do for you, and to figure out if they are really Black Hat SEOs whose work might well pay off in the short term, but yield a disastrous collapse in your traffic once the search engines catch onto their game.

Contents

What's With the Hat?

[Top]

The business of search engine optimization is infested with sharks: scammers, swindlers, spammers and fraud artists. They are the Bad Guys of the Internet, and so like the Bad Guys in the old cowboy movies, we say they wear Black Hats. You could tell the good guys in the cowboy movies because they wore White Hats.

The practice of White Hat Search Engine Optimization, also known as Ethical SEO, aims to make it easier for the search engines to tell when your site's content is relevant to a query. In most cases we do this by making a site genuinely more useful to its human visitors, so that they will recommend it to others by linking it from their own websites. The most important White Hat technique is simply to have a website worth recommending. That is not easy by any means, and certainly can require a great deal of work, but it is the honest way, and the only way likely to yield lasting results.

But for every White Hat technique there is a Black Hat technique for simulating it. While I work hard to write articles that help one understand various topics that often turn out to rank well for the relevant keywords, the Black Hats write software to generate thousands of grammatically correct yet meaningless articles filled with keywords carefully chosen to draw search engine traffic and yield maximum ad revenue. While I depend on others to recommend my writing by linking my articles, the Black Hats operate link farms, thousands of pages all linking to the single site they wish to promote.

Black Hats of all sorts are known as "hackers", but that is a name they appropriated from those who originally wore the badge with honesty and pride, those who work hard and cleverly to develop computer software and hardware. I have been proud to call myself a hacker - the White Hat kind - for many years, so I prefer to call the dishonest ones Black Hat hackers. The Black Hat SEOs have the specialty of hacking the search engines to make money dishonestly. White Hat SEOs by contrast work to make it easier for the search engines to do a better job.

What If I Don't Care About Ethics?

[Top]

While you may not care to follow the straight and narrow path, I assert it is in your best interest to do so. There is nothing illegal, not yet anyway, about being a Black Hat SEO, but all of the search engines employ smart people who work hard to defeat them. They have to, because the success of a search engine company depends critically on delivering search results that are as relevant as possible to each query and useful and interesting to the searcher. Whenever a Black Hat succeeds by capturing a search engine's traffic, their visitors soon figure out they have been had and instead try their search on a different search engine.

It is in fact quite possible to make a great deal of quick, easy money by spamming the search engines. But I assume that if you are reading this article at all that you have a website whose long-term success you are committed to. Ethical SEO yields the stablest, most secure long term results:

From time to time each search engine improves their search algorithms in ways that tend to defeat search engine spammers and promote legitimate websites. Even if you are able to escape notice by flying under the radar, an algorithm change can, and almost certainly eventually will bring your site's traffic crashing down.

If you succeed with your unethical website optimization well enough to become a high-traffic site, you could fare even worse: you might get caught, as WordPress did when they sold hidden links of their high PageRank homepage to help advertising publishers capture search engine traffic, and as a result was removed from Google's index. As GoogleGuy said:

There definitely appear to be hidden links on the root page of wordpress.org using CSS, e.g. "text-indent: -9000px; overflow: hidden". That's clearly against our quality guidelines...

Google's guidelines are quite clear on things like hidden text and hidden links to duplicate content. People should have a skeptical reaction when someone comes trying to buy links to spammy/duplicate pages, esp. if they want control of a subdomain or a subdirectory on your own site--linking to content like that can trigger effects to a whole site's reputation...

There is no ticket to easy street. Success on the web, lasting success, takes long hard work. A search for "search engine optimization" at any of the search engines yields page after page of both result listings and advertisements claiming top ten placement and quick results. I don't even have to look to know they are frauds. Let me make this absolutely clear:

Don't quit your day job. If you work both smart and hard on your new site, you could be earning a living from it in a year. But it will likely take months before you get many search engine referrals. Once a site is well-established in the search engines, referrals to new content can show up in less than a week. But it takes time before the search engines learn your site is worth treating well, time for your visitors to learn about your site and for some to link it from their sites, and time for the search engines to learn that your site is so highly regarded by others.

PageRank Simplified

[Top]

How do search engines decide where to present each page in their results?

A search engine's objective is to present each result in the order of its relevancy to the search query. How to do so in an automated way, for every page on the Web and for every possible combination of keywords in all the languages people speak is a very deep and difficult problem. Search engines keep most of their methods - their algorithms - secret, in part to make life more difficult for the Black Hat SEOs and in part to maintain their competitive advantage over competing search engines.

One algorithm is known publicly, the PageRank algorithm invented by Google founders Larry Page and Sergey Brin when they were graduate students in Computer Science at Stanford University. Its details were published when they were granted U.S. Patent number 6,285,999 for PageRank's invention. The word "PageRank" is a trademark of Google Inc.

Unfortunately, public knowledge of the PageRank algorithm allows anyone to manipulate the PageRank of their site. Thus the PageRank Google calculates for real search results, while based on the patented algorithm, is much more complicated than I describe here, mostly to defeat the efforts of the Black Hat SEOs.

PageRank measures the probability that one would end up on a given page when clicking links randomly while surfing. If one ends up at a dead end, that is, a page with no links, a new page is selected randomly.

Google will tell you the PageRank of any page on the web if you install one of the browser plugins listed below, . The Toolbar PageRank is an integer from zero to ten, but the PageRank used calculated by Google is a floating point number that is scaled and rounded for the Toolbar.

If two pages have otherwise equal PageRank, Google will present the one with higher PageRank first in its search results. Pages with higher PageRank are thus considered more important or reliable by Google.

The success of the PageRank algorithm quickly propelled Google to the dominant position among the competing search engines. This combined with the fact that PageRank is easily measured quickly transformed the practice of search engine optimization into a never-ending quest for higher PageRank. While PageRank is important, and will help you to improve your PageRank, it is by no means the only factor in your success, as we shall see.

PageRank can be understood as a democratic process: each page on the web has one vote, and distributes that vote among the other pages it links to. The votes of pages that themselves have higher PageRank are given greater weight in the results.

If more pages link to one of your pages, its PageRank will increase. But it is important to understand that it is the PageRank of the page on which your link falls that counts, not the PageRank of the homepage of the site that gives you a link. I am often spammed with link exchange requests from sites that brag of their high PageRank, only to find that their link pages are buried, and have low PageRank. There is still value to swapping links, but not as much as the spammers claim.

A page that has no inbound links is given a small, but nonzero PageRank. The ToolBar PageRank will be reported as zero, but the value Google uses is a small fraction.

The PageRank contributed by a given page to those it links to is equal to its own PageRank divided by the total number of links it has. Thus if I have a PageRank five page that links to one of your pages, but my page has four other links on it, it will increase your page's PageRank by just one.

The PageRank reported by the Google Toolbar and the other plugins listed below is an integer scaled to fall between zero and ten. It is thought to be logarithmic, so that it is much harder to get to the next step than it was to advance from the previous one. No one outside of Google is sure what the base of the logarithm is, but let's suppose it is 10. A real PageRank of 1 would then have a Toolbar PageRank of 0, while PageRank 10 gives a Toolbar PageRank of 1. To achieve a Toolbar Pagerank of 10 requires a PageRank of 10,000,000,000.

From here on, when I refer to "PageRank", I mean the Toolbar PageRank. It is often abbreviated as "PR".

I worked for seven years to build my previous company's website with the result that its homepage rose to a PageRank of 6 and some of my other pages were PageRank 5. After focussing my efforts on GoingWare's Bag of Programming Tricks for several months, it rose from 5 to 6 and a few of my other pages rose 4 to 5. I am working now to get all my articles to PR 5. Some of them are still 4, and a few are 3.

However, I didn't have much of a clue about the webmastering business for most of that time. If I had it all to do over again with what I know now, and worked at it full-time, I expect I could achieve the same results in a year. That's why I said earlier that if you worked both smart and hard, you could earn your living from your website in a year.

Wikipedia, The Free Encyclopedia, has an article on PageRank with the full technical details.

What's My PageRank?

[Top]

You can find the PageRank of any page on the web by downloading and installing one of the following web browser plugins:

SearchStatus also reports the Alexa Traffic Rank for a site. Alexa's rank is reported for the whole domain a page falls in, rather than for a single page. Internet Explorer users can install the Alexa Toolbar. You can also search at Alexa's homepage to find the traffic rank for any site. As I write this, GoingWare's is about 90,000, up from 180,000 a few months ago when I kicked off my marketing campaign.

A word of warning: all the PageRank plugins as well as the Alexa Toolbar work by reporting the URL of each page you visit to a server. If you're concerned about your web surfing privacy, don't install them.

Alexa calculates its Traffic Rank from the tallying up the number of times its its toolbar reports visiting a site. If a lot of people who have the toolbar visit your site, your traffic rank will increase. Not many people use the toolbar yet, so the statistics reported for low traffic sites are not reliable: the traffic rank for a low traffic site can be greatly affected by just a few people. If you install the Alexa Toolbar or SearchStatus then spend a few days working vigorously on your own website, your traffic rank will skyrocket because of your own activity.

Don't depend on Alexa to report your traffic - analyze your web server log files instead. Web server log file analysis is the subject of an upcoming article at GoingWare's Bag of Programming Tricks.

Get Clicked

[Top]

I have a popular PageRank 5 page that's in the top ten for a couple keywords where all the other top ten pages are PageRank 6 and above. For one keyword it's number eight where number nine is PageRank 6 and number ten is PageRank 7.

The reason I think is that people like to click it in the search result listings, and (I'm pretty sure) they don't then just hit their Back button. I'm pretty sure all the search engines notice whether someone stays at a page they've clicked on rather than hitting Back and doing more searches.

How? The URL in the search results is often a redirect rather than the actual destination link. For some engines, this is the case for every search result, Google just does this for a small sample. Hover your mouse over a search result that turns up one of your pages - does your page's URL show up at the bottom of the browser window, or some funny-looking long URL at the search engine's domain?

This won't happen often at Google, but if you check each time you search you'll see it eventually.

What happens when you click the link is that it takes you to a server that records which search result you clicked on, and then sends you on to the destination page. If the search engine notices that a particular page gets clicked on more than others above it, the search engine ranks that page a little higher. Similarly, pages that don't get clicked on move lower.

This yields my first advice for White Hat SEOs:

Use <title> tag text that will make searchers want to visit your page.

Look over the most important pages on your site. The text in the title tag appears at the top of the browser window, not in its content area. Don't confuse it with the <h1> tag, which appears in the content area of a window. Your page's link text in a search result listing will be whatever is in your <title> tag.

There is more that you can do though. Try searching for one of your page's exact URL in Google. You'll see a page showing the title, a snippet of sample text, and some links where you can find the number of backlinks and so on. My question for you is, is that sample text what you want people to see?

When I realized that this sample text for many of my pages was simply "Copyright 2005 Michael D. Crawford. All Rights Reserved." I got a clue and did the following:

Carefully compose one or two compelling sentences describing the page and place it in the very first <p> element after the <h1> element. Place the exact same text as the content attribute of a <meta name="description"> tag in your page's <head> element.

When no other sample text seems more suitable, most search engines will use either the meta description or the first paragraph after the <h1>. Here is this page's meta description:

<meta name="description" content="A few simple, honest practices can
dramatically increase your search engine referrals by making your pages
more appealing in search results." />

Now look to see how this page is presented by Google by searching for its URL. Your mileage may vary:

Search Google for http://www.warplife.com/tips/webmaster/search-engine-optimization/

Try searching for the URLs of a few of your own pages. Are the results what you want people to see? Are they the best they could be?

Some people don't like to use <h1> tags. These people don't like to get search engine referrals. Instead, the prefer to use <p> text that has been made larger either with a <font> tag or CSS. You see, they prefer controlling the appearance of their pages to making money from their website. This leads me to:

Use an <h1> tag on every page on your site. Usually you want its text to be the same as your <title> tag's. If you don't like how <h1> text looks, use Cascading Stylesheets to control its appearance.

Your sample text can be as long as about 160 characters, with the exact amount being different depending on the search engine. You don't have to use all the available space, but use most of it. If it's really short, the search engines will fill in the available space with some randomly selected text from your page, which might not be appealing. Make your case as close to the beginning as possible, as sometimes the end is cut off and replaced with other text from your page.

Copy the following ruler into your page below your meta description tag to measure the length of its content element. It is an HTML comment so it won't show up in your page:

<!-- ....10........20........30........40........50........60........70........80 -->

If you're not able to compose appealing page intros, consider hiring a professional ad copywriter. Certainly post improved drafts as you think them up.

Once I realized the importance of all these, I pulled a couple all nighters implementing them all over my site. As a result, my traffic skyrocketed, with traffic to some of my pages multiplying five to ten times, with many of my pages moving up significantly in the search engine results.

Grab Their Attention

[Top]

My next two pieces of advice are much harder to achieve, but are the most important:

Have content that's worth your visitors' while to visit. Otherwise they will just hit their back button.

Now, I'm sure if we had the answer to that question, we could all retire. Yet it still seems to be news to many folks. It's worth your while to labor over your content over a long period of time, posting improved versions as you're able to revise them.

A corrollary to this is:

Post each page's most compelling content Above The Fold.

That is, capture your visitors' attention and convince them to stay without them having to scroll their browser windows. If you use a hi-resolution monitor, either shrink your browser windows or test with a lower resolution to judge the effect of your pages' "above the fold" content.

GoingWare's site was originally meant to advertise my software consulting business. Someone gave me this advice years ago and it made a huge difference to me. One result of it was that I was able to work in my profession throughout the economic downturn, while many of my colleagues either had to sell or even lost their homes and had to move back in with their parents.

I have continuously applied this advice with each redesign of all my pages.

And this leads to, finally:

Do not use "Intro" pages. Give your visitor what they want to see when they first visit your homepage. Never, ever, make them sit through a Flash intro.

One sure sign of a website designer who doesn't understand how to make your website earn money for you is that they recommend a Flash intro for your site. My problem is not with Flash, however: some sites have static graphic homepages, perhaps with a great big logo that you click to enter. You will find many of the visitors you might have otherwise retained will instead hit their Back buttons.

The search engines will see that they returned, when the visitor you just lost either performs a new search or clicks a different link in the original search result page. Your position in the search results will then get a little worse.

Homepage Usability cover

Homepage Usability: 50 Websites Deconstructed

Jakob Nielsen and Marie Tahir

[ Buy at Amazon]
[ Buy at Powell's]

Good advice for designing your homepage is Jakob Nielsen's book Homepage Usability. In it, he has screenshots of a hundred homepages from prominent websites, with meticulous critiques.

Now, if you try to apply my advice to every page on your site all at once, you will quickly go mad. Instead, work on your most important pages first: your homepage, then a few of your most popular content pages. Wait to apply it to your less significant pages until you get free time. After a little while it will become very easy for you to apply these techniques to your new pages.

Try Googling the URLs of your ten most significant pages. Does Google show what you want people to see? Now look at each of the pages. Will they capture your visitors even if they don't scroll their windows?

Once you can say "yes" for your top ten pages, you will fare much better in the search engine results the next time they update their indices and your website revenue will increase significantly.

I know this very well. I Bet the Farm back in May of 2005 by taking a lot of unpaid time off my consulting work to work on my site, and as a result of the above and a lot of other things I did, in July I quit my job for AdSense.

[Home | Contact | What's New? | Products | Services | Tips | Mike]