Some additional details of my discussion w/ reporter John Bohannon for his Science story on Google Scholar

There is a story in today’s Science magazine on Google Scholar by John Bohannon.  Entitled “Google Scholar Wins Raves, But Can It Be Trusted” the article discusses some of the pros and cons of using Google Scholar.  The author of the article interviewed me on and off over a few weeks about Google Scholar because I have written multiple blog posts on how I use it.  For example see:

• Mini journal club: staged phage attack of a humanizes microbiome of mouse
• Weird things coming up from automated Google Scholar searches pointing to hea1thandfitness.com
• Playing with Impact Story to look at Alt Metrics for my papers, data, etc
• Thank you Google Scholar Updates for finding me two new papers to read
• Google Scholar has search by date option – a good thing
• Thanks Google Scholar Updates for new article to read .. wish you had an #openaccess only setting
• Wow – Google Scholar “Updates” a big step forward in sifting through the scientific literature
• Playing w/ Google Scholar’s “Citations” system – some nice features but in need of a little fine tuning

And also a diverse array of posts on Twitter which I will not rehash here.

Anyway – the new article covers some interesting points but is very very short (ahh — the fun with page length restrictions).  So I thought I would post here some of the comments I made about Google Scholar in emails with the reporter.

Bohannon wrote to me on December 1, 2013

Dear Dr. Eisen-

I’m writing a news story for Science about Google Scholar. Have you continued using their article recommendation engine? (I saw your blog post about it from last year.)

(and then he wrote some details about what he was working on which I am not sure he would want me to post here and I have not asked so I am leaving them out)

I’d very much like to hear your thoughts on how Google Scholar has developed, and how well it works as a replacement for traditional library/proprietary/non-open literature databases.

cheers and thanks in advance,
John Bohannon

I wrote back, that same day (unusual for me):

Well, was just hacking around with Google Scholar this AM.

Have you dug into their new function – Scholar Library? I am playing around with it but have not quite figured it out. What I am hoping to do is to figure out how to get recommendations based on lists in the Library. Currently, the recommendation engine has one very very big limitation. It bases recommendations on one’s own publications. And if you are trying to move into a new area – well that is pretty useless.

So – some comments

1. I find the recommendation engine to be very very useful still. One of the best ways to find out about new papers. With the limitation mentioned above.

2. I use automated Google Scholar searches to find all sorts of papers of interest. This helps cover topics more broadly than the recommendation engine.

3. There are other recommendation systems out there – but I have not used them too much.

4. As for “free and open” – I don’t think I would use such terminology here. Yes, Google Scholar is free. But is it open? I don’t think so. For example, I am not sure if they publish / release all their code that works behind the scenes. And I am not sure how open the results are (not saying it is not open – I just don’t know).

5. The citation information is a wonderful tool – and it is great to have this information be freely available. I use it routinely for all sorts of purposes including getting around the massive limitations of IF. One issue however with GS is that it takes citations from ALL sorts of sources including non peer reviewed material and even material that people may not have been aware was even publicly available. So – citation counts are generally higher – sometimes much higher – in GS than with other metrics. I note – I put info for my Citations based on GS on my CV and various other places.

And this allows GS to be seriously gamed in terms of citation counts.

6. GS is still clunky in a few ways and I hope that Google puts more effort into it. It could become THE tool for academic scholarship searches and tracking but it has some bugs and minor annoyances still.

Just some quick thoughts. Let me know if you have any other questions.

Jonathan Eisen

He then wrote back with some questions. Again, I won’t share the whole email here at this time, but I will share the specific questions he asked. And I then wrote back with some answers:

Question:

** Care to share a recent example? Something nicely illustrates its usefulness.

Answer:

Open source software tool for the week: AXIOME: automated exploration of microbial diversity

Thank you Google Scholar Updates for finding me two new papers to read 

Question regarding my comment on other systems

Me: There are other recommendation systems out there –  but I have not used them too much.

Bohannon: ** Are there?  For example?  I should take a gander.

Me: Mendeley has one – called recommended or something like that

 

Question regarding my comment on free vs. open

 

Bohannon: Right, good point. Free but not really open. Do you feel like this is a worrying limitation? Is it realistic to make it open?  Or at least, more open?

 

Me: Well, I am certainly not one to tell others how to run a business.  So I am thankful for any free or open material released / produced by for profits.  But there is a major worry here which is – if we do not know how the system works – we do not know if it is biased or how it can be gamed, etc.  And if it is not fully open then if we invest in making use of it, Google could simply kill it at any time and we would have no source material to use for other purposes.

 

Question regarding my comment on Impact Factor

 

Bohannon: What is an example of a massive limitation of IF that GS solves? And I wonder why GS can’t tell the difference between peer-reviewed and non-peer-reviewed sources. I’ll add that to questions below.

 

Me: GS helps with article level metrics as opposed to journal level metrics like IF.  I can look up citations to all my papers quickly.  And I can track metrics like H index and I10 and others.  IF – being a journal level metric – is not really that informative in my opinion (and that of many others)

 

Question on other things GS could do:

Questions for Google

Bohannon: * Any other questions you’d like Google to answer, or features to request?

Me:

Well many features to request.

1. Better hot linking of all authors of a paper. Right now they only seem to link the 1st author to their google scholar page.

2. Better handling of long author lists.

3. Better ability to upload collections exported from other tools.

And many more …