Over the last few weeks I've been working on a project on/off that involves web crawling, scraping of data online/offline from various sources, and analysing/re-formatting it. However, only recently have I been reminded of the increasing importance of this practice.
When you work for a company that has financial and/or philosophical limitations you need to be resourceful. To this end, research becomes a critical aspect of your job from time/effciency perspectives and you'll soon discover the wonderful world of comparators/aggregators such as ones on shopping.
From a technical standpoint they're not much more different than a standard web scraper. Create a web crawler which then looks for certain patterns within the web sites that they have crawled and use this information in order to create dynamically generated websites containing pertinent information within regards to a particular subject domain. The other alternative would be to rely on a push type mechanism similar to the way RSS/Atom feeds work where relevant information is pushed out at regular intervals although this would obviously require co-operation at both ends of the spectrum. Hence, the former philosophy is more favoured. The basic code/structure for such a piece of software is as follows.
http://www.searchtools.com/robots/robot-code.html
- as usual thanks to all of the individuals and groups who purchase and use my goods and services
http://sites.google.com/site/dtbnguyen/
http://dtbnguyen.blogspot.com.au/
- as usual thanks to all of the individuals and groups who purchase and use my goods and services
http://sites.google.com/site/dtbnguyen/
http://dtbnguyen.blogspot.com.au/
