Sorry, we don't support your browser.  Install a modern browser

Run more than one spider crawl at a time#69

I’m guessing that running a single web crawl at a time is necessary for optimization purposes, and possibly helps prevent abuse.

But when managing multiple websites, running just one at a time really slows things down. For 5-10 websites, it’s annoying and tedious, but not that big a deal. But as the number of managed websites grows, that limitation gets to be more problematic and starts to hinder productive work.

I don’t know what the most optimal number of crawls at a time is. Maybe it can be relative to the number of websites in the account, or the number being crawled, or perhaps it should be based on the plan. Whatever it is, one just isn’t enough.

a year ago
Merged into Queue spider web crawls#68
a year ago