I have some difficulties with the application I'm creating. I'm trying to do some Web Crawler for gathering some information from the Web. I can use a few http proxies and I want to go for each request through another proxy. But the problem is not so simple. I want each request to go through the most efficient proxy. For example after a call is made I want to check the response time. After some requests have been made the application should start "learning" that the requests for some domains are faster if they are done through a specific proxy, and it should direct each call to the most efficient one. The performance of the proxies is variable so I want to have it very adaptive.
Currently I'm just take for each request the next server and I didn't write any logic to find the best matching proxy.
Can anyone help me with this issue? Please let me know if you know some pattern that I can use for what I need. Thanks.