We have added a feature to the Search tab under the stats box on each Hub that enables you to see when that Hub that has last been crawled by search engines.
When an Idle Hub is crawled, search engines will see that it has a noindex tag, which indicates to search engines that the page should not appear in search results. When a Hub falls into Idle status, it may take some time before the Hub is crawled, so it may remain in search engine results for some time. If you update an Idle Hub quickly, right when you see it is no longer Featured, you may bring it out of Idle status before it is crawled, meaning you can avoid having that Hub removed from search engines' indexes entirely.
Do keep in mind, though, that even after a Hub has been crawled, it might take a search engine some time to take it out of the index or add it back in. Essentially, once a Hub has been crawled and a search engine knows what we say it should do with a Hub, the ball is in the search engine's court.
On the flip side, it may take a while between the point in time at which an Idle Hub becomes Featured again and the point at which it is added back to search engine indexes, as the Hub must be crawled first.
We added this feature to make it easier to see why your Hub is being indexed (or not indexed). We hope you find it useful!
For more information on Featured, Pending, and Idle status, stop by our official Learning Center entry.
On more than one level the "why" is huge--thanks!
What a fantastic feature! Thank you Simone and the HubPages Team!
Very handy to have the last crawl info. here. Thanks.
Not only a great idea, but a great help. This is a benefit not only for idle hubs but hubs we are trying to promote! Knowing when (and if) Google or Bing have hits on our hubs is just another great weapon in our arsenal of hub promotions. Thank you, thank you!!!
Hey everyone! I am so pleased that you find this feature to be helpful.
Many thanks to Paul Deeds and Derek Gulbranson for getting this in order and making sure these things are as clear as possible!
This feature has allowed me to see that my latest hub - http://izzym.hubpages.com/hub/People-Wh … re-Pathway - which covers a very important and potentially life-saving topic, was crawled by the googlebot 8 hours ago, when the noindex tag was still on it.
Now it could be weeks before it comes around again, during which time this hub will not be read except by hubbers and social traffic.
It may actually take Google some time to remove your Hub from the index, so you might still be getting search traffic for quite some time.
This is a new hub that was put into pending mode.
It was visited by the googlebot during this period, presumably because other hubbers linked it to Facebook and Twitter.
Google saw the noindex tag. It'll likely be weeks before it gets indexed now.
If you are going to hide newly published hubs, then take away all linking from it to social networks to stop the Googlebot from seeing it.
The last two hubs I published were visited by G bot within 30 minutes, only to be slapped and told to go away! Hiding doesn't work when the hubs are sent to followers, etc. I used to love the way new hubs were indexed so quickly, now this good feature works against us and imposes an indexing delay and lost income. Why is this beneficial to established and trusted writers? I have never had a pending hub fail to get the Prince's Kiss and awaken. This is just a pain.
Is listed as being crawled 7 days ago, with an update to say 22 hours ago.
so what is going on with the tomatoes one!!!
All we can do is make sure that Hubs are properly tagged and let you know when they are crawled- once they are, it is up to a particular search engine to make a move, I'm afraid!
The tool says it was crawled 14 hours ago - with the NOINDEX tag in place.
Surely there must be a better way. This is extremely annoying!
Does HP have any stats on the proportion of pages that are crawled during the pending period and don't get indexed for days or weeks!
I suspect that EVERY new page gets crawled while pending - the bot sees the NOINDEX tag and goes away - not to return for days or weeks - surely this is bad and should be addressed. The hubber and HP loses income, which can never be regained!!!
To be fair, most Hubs do not start making significant income until they're quite mature (e.g. have been around for months- if not longer).
In my experience that notion is out of date.
Why doesn't HP introduce 'Trusted Authors' who wouldn't have all their new hubs sent to the 'sin bin'. Many other sites do it. It would free-up scarce resources. A Win Win!
In theory that's a good idea but I can see problems - it would lay HP open to accusations of favouritism.
Actually..... That would be a moot point... given the current favoritism practices!
We have list of hubbers NOW whose work is featured immediately.
Most of us have to wait.
And just how are they supposed to 'mature' if they get hidden in pending status from the get-go? Saying 'it's up to the search engine to re-crawl,' is a cop-out, because it was HP who installed the 'no index' PENALTY!! (For that is what it truly is!) If it's not indexed, the crawler may not come back for weeks, months or at all...lost income ALL AROUND. This was NOT a good move on HP's part!
Being able to "see" when it was last crawled does nothing to get rid of a 'no-index' tag. We need information on HOW to do THAT!
Re-write the article? Add/change tags or keywords? Or, once it's de-indexed, that's that--too bad, so sad?
"To be fair, most Hubs do not start making significant income until they're quite mature (e.g. have been around for months- if not longer)."
Its still lost income - If you impose delays so that an income of $100 that would have occurred in 4 weeks is delayed a week, after 5 weeks your net income is $100 not $200!
The tool should test whether the page is ACTUALLY indexed - that is what is important!!!!
Not to a Smoke and Mirrors Salesperson!
Maybe the tool can also now upgrade all that missing information on the measuring of hub Viewing Durations? It seems that half of mine, even though some of the hubs are over two years old haven't yet been measured yet for some rather obscure reason?
Thanks for the info. Great feature that likely as not I wouldn't have found without seeing this.
Awesome added feature! Thanks Hubpages! I never learned how to find out when the bots come a-crawlin' my hubs, so this added feature is very helpful to me. Cheers!
This is one of the best features yet. Thank you so much for giving us insight into it as well.
Great job HP! Thanks for adding this feature. I can use all the data I can get.
This is an excellent feature! I have been away for several months, and coming back, I see many useful tools that the HP team has implemented. It's a great, and wonderful surprise, to come back to a site that has so many useful tools for the writer.
Thank you and kudos to the HP team!
The points DzyMzLzzy and janderson99 made in posts (above) are correct--seeing when hubs are crawled is not helpful if hubbers cannot do anything about indexing. Those are the "tools" HP needs to provide--so we can see if our hubs are indexed and, if not, show us how to ensure they are indexed by Google and other search engines.
Its a wonderful feature Simone and HP team. i like that i can know when my site has been crawled.
this is such a very good feature. now i can manage my hubs even better. Thanks for this one!
Excellent idea for hub management. We can never have enough analytical ideas.
Many thanks for this feature, HP! I was pleasantly surprised when I first saw the message: "congratulations , etc.etc."
Thanks for sharing this info. I will definitely watch carefully for my hubs to come out of idle status.
This is a really cool feature. It definitely takes the guess work out of the equation for the authors. Which gives us more time to research and write hubs!
Thanks for your kind words, feedback, and comments everyone!
By the way, Simone, I used the new feature to see when my "best" hub was last crawled by search engines. Not good news. Even though my title uses words from engine searches, Google last crawled the article THREE DAYS AGO, and Bing, two days ago!
Great idea! Thanks you Simone and the rest of the HP team. Thank you very much for your efforts and keeping us informed.
This is a really nice feature. As soon as I read this, saw it in slow hub, I used it and made some modifications to existing.
Thanks for the info Simone.