Interesting research papers on documentation for transparency

This is a list of research papers proposing documentation formats for machine learning for the goal of transparency. This is a wiki article open to anyone’s contribution.

  • Bender, E. M., & Friedman, B. (2018). Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics, 6, 587-604.
  • Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumeé III, H., & Crawford, K. (2018). Datasheets for datasets. https://arxiv.org/abs/1803.09010 https://arxiv.org/abs/1803.09010
  • Holland, S., Hosny, A., Newman, S., Joseph, J., & Chmielinski, K. (2018). The dataset nutrition label: A framework to drive higher data quality standards. https://arxiv.org/abs/1805.03677; Kelley, P. G., Bresee, J., Cranor, L. F., & Reeder, R. W. (2009). A nutrition label for privacy. In Proceedings of the 5th Symposium on Usable Privacy and Security (p. 4). ACM. http://cups.cs.cmu.edu/soups/2009/proceedings/a4-kelley.pdf
  • Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., … & Gebru, T. (2019, January). Model cards for model reporting. In Proceedings of the Conference on Fairness, Accountability, and Transparency (pp. 220-229). ACM. https://arxiv.org/abs/1810.03993
  • Hind, M., Mehta, S., Mojsilovic, A., Nair, R., Ramamurthy, K. N., Olteanu, A., & Varshney, K. R. (2018). Increasing Trust in AI Services through Supplier’s Declarations of Conformity. https://arxiv.org/abs/1808.07261

Hi @jingying

Thanks for compiling this useful list and the great work the ABOUT ML team is doing at PAI!

I wanted to let you know 2 things

  1. There is an updated version of the Hind et al paper listed above:
    M. Arnold, R. K. E. Bellamy, M. Hind, S. Houde, S. Mehta, A. Mojsilovic, R. Nair, K. Natesan Ramamurthy, A. Olteanu, D. Piorkowski, D. Reimer, J. Richards, J. Tsay, K. R. Varshney, IBM Journal of Research and Development, 63(4/5), 2019.
    link: https://ieeexplore.ieee.org/document/8843893

  2. We’ve recently written a follow-on paper about our experiences with completing real FactSheets and posted it on arXiv.
    Experiences with Improving the Transparency of AI Models and Services.
    M. Hind, S. Houde, J. Martino, A. Mojsilovic, D. Piorkowski, J. Richards, K. R. Varshney. Nov, 2019,
    link: https://arxiv.org/abs/1911.08293

Thanks,
Michael Hind