Why to de-index content
There will be times when you will want to de-index old or useless pages from Google and other search engines. As a website naturally evolves you may want to remove / 404 pages to keep your site and file structure smaller, or you may want to optimise your site for page rank, and do this in part by de-indexing pages and making your site smaller so your links will provide your pages the maximum benefit. To de-index a page it simply a matter of returning a 404 status code when a search engines tries to crawl it, this is easily done through PHP using a 404 header. After a search engine requests the page and it continually returns a 404 it will eventually drop from its index.
De-indexing your entire website and putting up a new one
The concept and execution of de-indexing can be taken to an extreme in terms of de-indexing your entire site in order to start from scratch. This may seem like a crazy idea, but there are times when getting rid of your entire site may be the only option in order to really optimise your site. An example would be a website that is completely built on URL parameters. Not only do these URL's look poor in search results, but having a large number of parameters can also cause crawl problems.
ddmseo.com/home.php?page=seo&article=de-indexing // Poor looking URL that can cause no end of issues
ddmseo.com/articles/de-indexing // Much better looking URL for users and search engines
In this circumstance you might probably have to de index the entire website from Google, build a new site and set up as many redirects as possible for back linked pages. The only real issue you face if you want to de-index your entire site and create a brand new one with new URL's and pages is making sure you keep all your backlinks. If your site is years old you might have hundreds or thousands of links to redirect and while this may seem like a tedious problem as long as you redirect most of them you will find that pretty quickly your new website will be indexed and ranked up to level close to your old site.
Saving your backlinks
You can use various tools within Google and Bing Webmaster Tools to work out your backlinks. If you have any sort of e-commerce site and dynamically generate the URL's you could easily dynamically generate a 301 header from your old pages to your new ones. While it may seem a bit far to go to de-index your entire website from Google just to make sure you have perfect URL's, it’s something that is important. As long as you have a very well built site Google will start re-indexing your pages almost immediately, and within a few days you will see your new pages pushing up to search positions close to your original pages. A few things to watch out for are to make sure the old pages that are not back linked are returning 404 status codes. You will also want to immediately re-submit a new site map to all major search engines. Having to de-index a site and start afresh can easily happen if you set up a poor URL structure when you launch your website. But dumping your entire website to start again is not something you should be afraid of; it gives you the opportunity for a new design, a new URL structure and many other things.