Google Scholar is Manipulatable - Slashdot

Posted by msmash

• ›

  This is an example of Goodhart's Law [wikipedia.org]:

  "When a measure becomes a target, it ceases to be a good measure"

  There is an article in this week's Economist about how the peer review system and promotions based on publishing and accumulating citations are failing. Companies that might benefit from research mostly see research from universities as worthless, yet also do much less of their own research because of perverse incentives. This leads to less innovation and lower productivity growth.

  Universities are failing [economist.com]

  • Re:

    "Of course, there is a place for speculative research unconstrained by economic concerns, but perhaps we have gone too far in that direction". But that's the very opposite of what's happening. It's getting harder and harder to get funding for research unless you can demonstrate the likelihood of short-term commercialization.

    • Re:

      Do you have a citation for that? That's the opposite of what the Economist says.

      Or, more accurately, the Economist says that the research isn't actually useful, but not necessarily that it wasn't expected to be useful.

  • Re:

    For-profit companies do their own research because they want proprietary innovations. If companies just relied on public research, then they become manufacturers of commodities, which is not a great position for those companies.

    Most published work is either incremental or not all that relevant. That's true for both academia and corporate research. However, a small percentage is useful, and an even smaller percentage is seminal. Missing out on the useful and especial seminal work is a bad thing.

    Most good

• Apart from doing important, relevant, useful research, there's other ways for academics to increase their citation counts. In the social sciences at least, one is to make provocative claims that will more than likely draw the ire & indignant outrage of other academics who, in turn, will publish papers with rebuttals & counter-arguments, all citing the offending paper.
  
  Another is to take one research project but publish several papers on it, each reporting a different feature or aspect, rather than

• The notion that a paper has merit because it has a lot of cites is intrinsically faulty. It's easy to posit a paper that is so advanced that no one cites it all all, since it obviates all existing research. This is more of the "closed shop" mentality working in academia, it's the same with the peer review process, which has shown time and again to be easily manipulated. This is just another and obviously flawed example of how easy it is to show "measurables" to non-technical people.

• Google Scholar is incredibly inaccurate because there is no way to control the algorithm and tell it to exclude papers that have nothing to do with you. I tried for a while to clear out all the papers it added to my profile that had nothing to do with me but I work in a field with large publication rates from big collaborations and there was just no way to keep up with the deluge. As a result, I would not trust Google Scholar's citation record for anything important - if you want accurate information you need to use something like ORCID.

  • Re:

    I'm not sure the field you work in is relevant. It attributes at least one of my papers to someone with the same name in a different field, and it won't let me correct the data because I don't have an email address which matches its whitelist.

• The system is not broken, except for the incompetent. No one relies solely on citation counts to hire or to give awards. It is a first-level weed-out mechanism. After that first step, the specific papers are examined, and the quality and impact of those papers will be considered along with other things.

  In a way, the citation count is like a GPA. No one hires a student based solely on the basis of a 4.0 GPA. The 4.0 can lead to an initial interview, but it doesn't replace the interview.

• Even worse for academic publications is that on average claimed results in peer reviewed journals are slightly less than 50% reproducible for many fields. See for instance https://royalsocietypublishing... [royalsocie...ishing.org]... and the principals in these fields are well aware of this problem.