Less than 60% of traffic on the internet is human. . .The rest is spiders
– Security firm Imperva
They should have expected this when they named it the World Wide WEB. But, you won’t see these spiders online. They are silent, but like real-life spiders, extremely useful. Well, most of them are.
Spiders, also called web-crawlers (take THAT Peter Parker) are used by search engines. Spiders catalogue the internet. When you search for “Most popular cat videos” on Google, the search giant doesn’t go out and query the internet right away. It’s already done that, via spiders. They crawl through every website they can access, collecting words, phrases, alt-text, even images and songs. Those queries are then stored in massive databases that Google references. The spiders fetch the data so it’s within easy reach.
The science is amazing. Google is silent about how many spiders it has at any one time, but the technology is advanced enough that it could be as few as 1000. Google’s not the only one combing through the Internet looking for cat videos, of course.
It occurred to me the other day that we are starting to see spider technology in the form of personal shoppers. Visit a Walmart at 2:00pm and you are likely to see the aisles crowded with people. Not customers, but spiders. Humans who are going out and fetching the things on your grocery list.
The search results are either stored on site in a set of lockers that look a little like a representation of a database, or brought directly to your door. Just like the Google spiders.
The humans are much slower, of course. And unlike Google, you pay for every spider crawl. But, just as online spiders dominate the web traffic, human spiders, especially during the middle of the day dominate the traffic at a Walmart.
And they even do delivery. Full service spiders.
Stay safe
Rodney M Bliss is an author, columnist and IT Consultant. His blog updates every weekday. He lives in Pleasant Grove, UT with his lovely wife, thirteen children and grandchildren.
Follow him on
Twitter (@rodneymbliss)
Facebook (www.facebook.com/rbliss)
LinkedIn (www.LinkedIn.com/in/rbliss)
or email him at rbliss at msn dot com(c) 2021 Rodney M Bliss, all rights reserved