Home | News & Events | Events | 2014 Events | NISO Virtual Conferences | June 18: Transforming Assessment: Alternative Metrics and Other Trends

NISO Virtual Conference: Transforming Assessment: Alternative Metrics and Other Trends

June 18, 2014
11:00 - 5:00 p.m. (Eastern Time)

  • About the Conference
  • Agenda and Presentation Slides
  • Event Q&A
  • Registration
    Can't make it on the conference day? Register now and gain access to the archive for one year.
  • System Requirements:
    • NISO has developed a quick tutorial, How to Participate in a NISO Web Event. Please view the recording, which is an overview of the web conferencing system and will help to answer the most commonly asked questions regarding participating in an online Webex event.
    • You will need a computer for the presentation and Q&A.
    • Audio is available through the computer (broadcast) and by telephone. We recommend you have a set-up for telephone audio as back-up even if you plan to use the broadcast audio as the voice over Internet isn't always 100% reliable.
    • Please check your system in advance to make sure it meets the Cisco WebEx requirements. It is your responsibility to ensure that your system is properly set up before each webinar begins. 

About the Virtual Conference

For decades, the landscape of assessment has been reasonably stable with traditional metrics, primarily the Journal Impact Factor (JIF), now provided by Thomson-Reuters. The JIF has served the community reasonably well for many decades, but with the growth of social media, open access mandates, and greater emphasis on articles over the journal "package," new methods of assessing quality and impact have exploded on the scene. The current metrics focus on the journal also leaves out a wide variety of non-traditional output of research such as data sets, software, visualization tools, or performance recordings, which are as—or even more—important than the journal article in some disciplines.

The new "alternative metrics" are not without their own issues. As with any new measures, different interpretations of the definitions exist, the metrics are inconsistently applied, or data from comprehensive ranges of sources are limited. NISO initiated a project in 2013 to identify issues around the new altmetrics that could be solved with standards or best practices. Phase 2 of the project is to develop the prioritized standards and recommended practices identified by the community.

This Virtual Conference will explore the range of new metrics forms and the infrastructure necessary to create reliable measures across the range of platforms, publishers, and authors. It will also discuss the current status of the NISO Altmetrics initiative and what standards or best practices may be forthcoming in this area.


11:00 a.m. – 11:10 a.m. – Introduction
Todd Carpenter, Executive Director, NISO

* * * * * * * * *

11:10 a.m. - 12:00 p.m.
Keynote Address: Altmetrics at the Portfolio Level
Paul Groth, Ph.D., Assistant Professor at the VU University Amsterdam

In this talk, Dr. Groth will look at the current status of altmetrics with respect to understanding scholarly impact at the portfolio level. In particular, the importance of contextualiziing almetrics in a larger picture or narrative around a body (or portfolio) of scholarship. In particular, he will discuss the potential benefits for using altmetrics not just for individual scholars but also particular research projects. Research projects are an important area of interest for the use of altmetrics because of their short lifespan, multitude of outputs, and diversity of membership.

Paul Groth is an Assistant Professor at the VU University Amsterdam in the Web & Media Group, and a member of the Network Institute. He researches approaches for dealing with large amounts of diverse contextualized knowledge with a particular focus on the web and e-Science applications. Specifically, his research covers data provenance, Web Science, knowledge integration and knowledge sharing.

* * * * * * * * *

12:00 p.m. - 12:30 p.m. Snowball Metrics: University-owned Benchmarking to Reveal Strengths within All Activities
Dr. Lisa Colledge, Snowball Metrics Program Director, Elsevier

Snowball Metrics are defined and agreed by higher education institutions without being imposed by organizations with potentially distinct aims such as funders and government groups. This presentation will cover the aims and approach of Snowball, and end by focusing on the importance of agreed measures across the entire spectrum of activities that a university does. This includes getting published, read, and cited. It is also important to be able to look at these activities but in a broader way than just Impact Factor and h-index: it also includes getting funding and commercializing that institutions' research and having a positive impact on the region. Altmetrics will be discussed - as they are typically defined - as part of this breadth, and also highlight some other key areas that it’s important to get insight into.

Dr. Lisa Colledge is an expert in the use of metrics that provide insights into research performance of universities, researchers, journals, and other scholarly output. She developed this knowledge by working with editors and learned societies to build understanding of journal Impact Factors relative to their competitors, and to develop strategies to improve journals’ standings. She then joined the SciVal group, which develops tools to support performance evaluation, where she was responsible for developing SciVal Strata, and for introducing two independent journal metrics in Scopus.

Lisa currently applies her expertise to Snowball Metrics, for which she is Program Director; in this initiative, a group of universities agrees a single robust method to generate metrics that are consistent, regardless of the data source used to generate them, and which thus support benchmarking. She also consults on metrics and their use to the SciVal group.

Prior to joining Elsevier, Lisa conducted postdoctoral research at the University of Edinburgh. She holds both a DPhil and an MA from the University of Oxford.

* * * * * * * * *

12:30 p.m. - 1:00 p.m. Assessing and Reporting Research Impact – A Role for the Library
Kristi L. Holmes, Ph.D., Director, Galter Health Sciences Library, Northwestern University, Feinberg School of Medicine

This presentation will be a broad discussion of alternative metrics and how they can be considered and contextualized in the academic environment. Particular emphasis will be placed on the perspective of the library, including possible roles for libraries in supporting these efforts locally, strategies for identifying needs and developing services, and a discussion of related local, national, and international efforts which offer libraries an opportunity to get involved and play a significant role.

Kristi Holmes is the Director of Galter Health Sciences Library and an associate professor in Preventative Medicine-Health and Biomedical Informatics at Northwestern University’s Feinberg School of Medicine. Dr. Holmes is an Associate Director in the Northwestern University Clinical and Translational Sciences Institute (NUCATS) and is particularly interested in issues related to clinical and research informatics, as well as the representation and the dissemination of research findings and resulting impact. She is the Engagement lead and serves on the Steering group for VIVO and serves on the Outreach Steering Group of ORCID. She earned her PhD at Iowa State University before joining the Bernard Becker Medical Library at Washington University in St. Louis, where she had worked as a bioinformaticist since 2007.

* * * * * * * * *

1:00 p.m. - 1:45 p.m. Lunch Break

* * * * * * * * *

1:45 p.m. - 2:15 p.m. What Do Alternative Metrics and the 'Flight of the Bumblebee' Have in Common?
Martha Kyrillidou, PhD, Senior Director of Statistics and Service Quality Programs, Association of Research Libraries (ARL)

Research libraries are engaged in developing new measures, staying competitive and at the same time wanting to be open and transparent while respecting privacy and individual creativity. So, how do we balance all the tensions in this complex and interconnected world? Kyrillidou will offer some perspectives from the Association of Research Libraries with examples that capture the development and use of alternative metrics by library leaders.

Martha Kyrillidou, senior director of ARL statistics and service quality programs, has been leading the Statistics and Assessment capability since 1994.

She pioneered the development of the StatsQUAL gateway, offering assessment products and services to the library, higher education, and government communities. The StatsQUAL products and services range from descriptive statistics to evaluative tools focusing on service quality improvements. Martha's analytical expertise covers both strategic and policy issues and she has widely disseminated findings through numerous publications, training events, consultations, and speaking engagements. Prior to her affiliation with ARL, Martha worked in the Library Research Center at the Graduate School of Library and Information Science at the University of Illinois at Urbana-Champaign and the Bureau of Research at the School of Education at Kent State University. 

Martha holds a PhD in library and information science from the University of Illinois at Urbana-Champaign (2009). Her dissertation is the most heavily downloaded dissertation at the IDEALS institutional repository. Martha holds an MLS and an MEd with specialization in evaluation and measurement from Kent State University (1990); and a BA in English from Aristotle University, Thessaloniki, Greece (1987).

* * * * * * * * *

2:15 p.m. - 2:45 p.m. Identifying the Invisible Impact of Scholarly Publications: A Multi-Disciplinary Analysis Using Altmetrics
Ehsan Mohammadi, Ph.D. Candidate, Wolverhampton University

The field of Altmetrics is concerned with alternative metrics for the impact research publications using social web data. In this talk, the results of three quantitative altmetrics studies will be presented with the aim of discovering the suitability and reliability of two altmetrics resources: Mendeley, a social reference manager website, and Faculty of F1000 (F1000), a post- publishing peer review platform. The correlations between the new metrics and citations at the level of articles for several disciplines and the contexts in which the new metrics can be useful for research evaluation across different disciplines will be discussed.

Ehsan Mohammadi is a PhD candidate currently working on his thesis at the Statistical Cybermetrics Research Group of Wolverhampton University. He is interested in validating new metrics rather than citation analysis for research evaluation. Ehsan aims to find and evaluate the social web-based metrics for research assessment, namely scientometrics 2.0, altmetrics, beyond Impact, and then validate these variables via comparison to traditional measures. He also holds a master of library and information science.

* * * * * * * * *

2:45 p.m. - 3:15 p.m. – Three Publishers' Perspective on Using Altmetrics: PLOS, PKP, and eLife

Jennifer Lin, Senior Product Manager, PLOS
Juan Pablo Alperin, PhD Candidate, Public Knowledge Project, Stanford University
Ruth Ivimey-Cook, Web Application Developer, eLife sciences

Altmetrics data can be a powerful and flexible tool for publishers in scholarly communications. They support new approaches to the structuring, presentation, use, and evaluation of research literature. They can also inform internal editorial and business processes. This panel will showcase the myriad of ways in which three publishers - PLOS, eLife, and PKP - are employing altmetrics.

Jennifer Lin is Senior Product Manager at PLOS. She currently leads the organization's Article-Level Metrics program as well as the data program. As a former business consultant with Accenture, she worked with Fortune 500 companies as well as governments to develop and deploy new products and services. Jennifer received her PhD in political philosophy and has served as an instructor at Johns Hopkins University.

* * *

Juan Pablo Alperin (@juancommander) is a PhD candidate in the Stanford Graduate School of Education and a researcher with the Public Knowledge Project (PKP). Juan leads several research and development projects (http://flacso.org.br/oa/category/proyectos/?lang=en) on improving the quality, impact, and reach of Latin American research, and is currently studying the alternative and public impact of open access.

* * * 

Ruth Ivimey-Cook is a Web Application Developer at eLife sciences.

* * * * * * * * *

3:15 p.m. - 3:30 p.m. Afternoon Break

* * * * * * * * *

3:30 p.m. - 4:00 p.m. Virginia Tech’s Response to Research Data Needs: The Center for Digital Research and Scholarship
Julie G. Speer, Associate Dean for Research and Informatics, University Libraries, Virginia Tech

* * * * * * * * * 

4:00 p.m. - 4:30 p.m. NISO Altmetrics Initiative: A Project Update
Martin Fenner, Technical Lead for the PLOS Article-Level Metrics project

Last year, the Alfred P. Sloan Foundation awarded NISO a grant to undertake a two-phase initiative to explore, identify, and advance standards and/or best practices related to a new suite of potential metrics in the community. This project is an important step in the development and adoption of new assessment metrics, which include usage-based metrics, social media references, and network behavioral analysis.

Martin Fenner, Technical Lead for the PLOS Article-Level Metrics project, has been enlisted by NISO to lead this project which will explore potential assessment criteria for non-traditional research outputs, such as data sets, visualizations, software, and other applications.

Martin Fenner is the technical lead for the PLOS Article-Level Metrics project. Before taking this position in 2012 he worked as a medical oncologist at the Hannover Medical School Cancer Center in Germany. He has served on the ORCID Board from 2010 to 2012 and is a member of the ORCID Outreach Steering Group.

* * * * * * * * *

4:30 p.m. - 5:00 p.m. Conference Roundtable: Finding Acceptance and Value: Why Altmetrics Matter
Moderated by: Todd Carpenter, Executive Director, NISO 



If paying by credit card, register online.

If paying by check, please use this PDF form.

Registration closes on June 17, 2014 at 4:00 p.m. (ET).

Registration Costs

  • NISO Member
    • $185.00 (US and Canada)
    • $225.00 (International)
  • Non-Member
    • $245.00 (US and Canada)
    • $285.00 (International)
  • Student
    • $80.00

Additional Information

  • Registration closes on June 17, 2014 at 4:00 p.m. (ET). Cancellations made by  June 11, 2014 will receive a refund, less a $35 cancellation. After that date, there are no refunds.
  • Registrants will receive detailed instructions about accessing the virtual conference via e-mail the Friday prior to the event. (Anyone registering between Monday and the close of registration will receive the message shortly after the registration is received, within normal business hours.) Due to the widespread use of spam blockers, filters, out of office messages, etc., it is your responsibility to contact the NISO office if you do not receive login instructions before the start of the webinar.
  • If you have not received your Login Instruction email by 10AM (ET) on the Tuesday before the virtual conference, please contact the NISO office or email Juliana Wood, Educational Programs Manager at jwood@niso.org for immediate assistance.
  • Registration is per site (access for one computer) and includes access to the online recorded archive of the conference. You may have as many people as you like from the registrant's organization view the conference from that one connection. If you need additional connections, you will need to enter a separate registration for each connection needed.
  • If you are registering someone else from your organization, either use that person's e-mail address when registering or contact the NISO office to provide alternate contact information.
  • Conference presentation slides and Q&A will be posted to this event webpage following the live conference.
  • Registrants will receive an e-mail message containing access information to the archived conference recording within 48 hours after the event. This recording access is only to be used by the registrant's organization.