I am a web developer working with a company that sells electric scooters online. We are currently extending the functionality of our affiliate marketplace - I am in the process of migrating the current marketplace to a solution based on Joomla + Virtuemart + VM Shopbuilder.
The objective of this project is to build a search bot that we can use to spider merchant's websites and find new products that become available. We have affiliate relationships with lots of merchants that have yet to provide us with a product feed - so we are looking for a solution that will save us the time of regularly researching the product listings on their site. This search bot will need a control panel and the ability to deliver search reports to our web administrator.
As a PHP programmer, I realize that the accuracy of such a script will be rather poor. We currently have a custom script in place that we call the "Pricescrubber". It contains most of the basic search functionality we describe below, so you will be able to hack through the existing code and appropriate anything you see fit. I will work with you to achieve the best result we can get while staying within our budget. We are also open to your own suggestions for the search methods employed here to improve efficiency and accuracy.
We are looking for the following features:
1. Search bot control panel (standalone or integrated into Joomla as a component)
a. Ability to add new merchant sites to be spidered
b. Search merchant websites by auto-filling the target sites search box with product keywords drawn from Merchant + Product matches in existing Virtuemart catalog database
c. Compare search results against merchant products that already exist in Virtuemart catalog, as well as existing search bot entries with "valid", "invalid", and "added" statuses. (explained below)
d. Flag potential new products that don't yet exist in our Catalog
e. Log the URL of these pages along with a guesstimate of new product information (product name, price)
f. All spider results will be stored in MySQL database and associated with merchant website
g. Ability to review bot's search results from control panel displaying the following information (date and time of potential match, match ID, guessed name, guessed price, url link to product page that opens in IFRAME for manual review)
h. Ability to mark entry status (pending, valid, invalid, added)
i. Entries with "invalid" status are used to filter out future results with the same returned results
j. Entries with "valid" and "added" status are used to compare against results with the same data and determine if there has been a potential price change
k. Daily digest email notification of potential new product matches and price changes
l. Ability to run search bot via schedule cron task
2. Control Panel Layout
a. Split screen IFRAME with search result data on top and result URL browser on bottom for the manual review of entries
b. Vertical listing of entries and matches with simple textual readout and silk icons for actions
3. Search bot to VM Shopbuilder integration (add product --> edit information --> insert entry into VM Shopbuilder database)
a. Ability to "Add new found product to catalog" that adds the entry to the VM Shopbuilder Import Manager
b. Automatically change match status to "Added" upon adding to catalog
c. Ability to manually edit the product description before entering item into VM Shopbuilder database
d. Save and associate custom description information with product so as to retain information even after future updates or price changes
Please post any questions you might have.