Wednesday, July 13, 2011
Online Shopping Comparators
Over the last few weeks I've been working on a project on/off that involves web crawling, scraping of data online/offline from various sources, and analysing/re-formatting it. However, only recently have I been reminded of the increasing importance of this practice.
When you work for a company that has financial and/or philosophical limitations you need to be resourceful. To this end, research becomes a critical aspect of your job from time/effciency perspectives and you'll soon discover the wonderful world of comparators/aggregators such as ones on shopping.
From a technical standpoint they're not much more different than a standard web scraper. Create a web crawler which then looks for certain patterns within the web sites that they have crawled and use this information in order to create dynamically generated websites containing pertinent information within regards to a particular subject domain. The other alternative would be to rely on a push type mechanism similar to the way RSS/Atom feeds work where relevant information is pushed out at regular intervals although this would obviously require co-operation at both ends of the spectrum. Hence, the former philosophy is more favoured. The basic code/structure for such a piece of software is as follows.
For those who don't already know this you can often bypass a lot of download managers out there if the 'Download Manager' doesn...
- we're going to attack this from a number of different perpective since it's a bit more complex then you'd normally expect. As...
If you're like many others on the planet you probably can't be bothered purchasing a new piece of software in order to test an appl...