Block Patterns In robots.txt

It's simple enough to block a search engine from crawling a fixed URL or folder, but case your URLs are dynamic such as /node/345/edit, you will need to define a pattern instead.

Use the following to block patterns:

Disallow: /node/*/edit$

The trailing "$" ensures that the URL ends with "/edit", otherwise any URL containing "/edit" will be blocked.

Now you can kick any incoming crawlers that try to visit those URLs into the bottomless pit in the middle of your town!

[ Source ]

Register Your Site With Search Engines

Submitting your site to a search engine will help the search engine crawl your site and let you configure some options for each search engine.

Bing and Google will also send you notifications if they find any errors with your site.

Google Labs also has some performance profiling tools to help determine some slow points in your website.

Submit your sites at the following URLs:

• Google Webmaster Tools
• Bing Webmaster Center
• Yahoo Site Explorer

Each will require you to verify your site. The easiest way to do that is to simply paste a meta tag they provide into your output HTML.

Note: Yahoo generates a meta tag that is not self closing. Be sure to add the closing "/" at the end of the meta tag.

Once verified, you can start optimising the search engine for your website.

Remember to configure your "robots.txt" file to disallow any important folders or URLs.