b'Viewpointsthe most sophisticated and accurate algorithms in use todayas psychology, behavioral economics, human-centered (e.g., neural networks) are exceedingly complicated anddesign, and ethics. 10Such an approach is necessary in light cannot be effectively described through language. In short,of the fact that no complete list of wrong practices exists in these types of algorithms come at the cost of constraining anmachine learning. Rather, the goal for an auditor must be to auditors ability to parse out the inner-workings of a model,ask if any steps of the development process are approached though providing the benefit of improved accuracy. Second,in a manner that does not give sufficient attention to the as often claimed by companies that rely on algorithms forissue of bias, with the definition of sufficient obviously revenue, open models allow for the possibility of gamingvarying across contexts.by the constituent population. While it is not necessarily inevitable that the results of an algorithmic audit are madeThe public sectors potential for leadershippublic, organizations with this worry may hesitate in fullyAs society begins to grapple with the potential drawbacks documenting internal procedures. of machine learning, perhaps with some of the initial fervor surrounding big data subsiding, public sector organizations Notably, even in cases where the exact contours of anare presented with an opportunity. Rather than waiting for audit are subject to debate, the idea that machine learningthe issues of bias to be solved by technology companies applications should be developed with a certain degreeor relying on legislators to push regulation, algorithmic of formal documentation is crucial for risk mitigation.auditing serves as a middle ground, balancing progress Regardless of the algorithms complexity, organizationswith caution. Governments should pursue innovative data can still commit to transparency around factors such asanalysis methods that will empower them to better serve optimization criteria, data inputs, sampling processes, andand understand their constituencies, but in a manner that feedback loops, all of which can limit the potential forpromotes accountability and equity. unintentional bias to become entrenched.FootnotesKey takeawaysTo summarize how the above-mentioned principles of1.Hawking, S. (2018). A Brief History of Time. Random House USA.auditing translate to practice, an algorithmic audit involves2.Audit. (n.d.). Retrieved from https://www.merriam-webster.com/dic-examining each part of a models lifecycle, using ationary/auditcombination of standardized best practices and discretionary judgment calls, all of which are informed by the available3.Suchman, M. C. (1995). Managing Legitimacy: Strategic and Institutional Approaches. The Academy of Management Review, documentation and social context. In the words of one20(3), 571. doi:10.2307/258788article from Harvard Business Review, the process must be interdisciplinary in order for it to succeed, relying on4.Burrell, J. (2018, August 13). Report from the first AFOG Summer Workshop (Panel 3). Retrieved from http://afog.berkeley.social science methodology and concepts from such fieldsedu/2018/08/13/report-from-the-first-afog-summer-workshop/5.Dirsmith and Haskins (1991). Inherent risk assessment and audit firm technology: A contrast in world theories. Accounting, Organizations and Society, 16(1), 61-90.6.Power, M. K. (2003). Auditing and the production of legitimacy. Accounting, Organizations and Society, 28(4), 79-394. doi:10.1016/s0361-3682(01)00047-27.Breiman, L. (2001). Statistical Modeling: The Two Cultures (with comments and a rejoinder by the author). Statistical Science, 16(3), 199-231. doi:10.1214/ss/10092137268.Burrell, J. (2018, August 13). Report from the first AFOG Summer Workshop (Panel 4). Retrieved from http://afog.berkeley.edu/2018/08/13/report-from-the-first-afog-summer-workshop/9.Power, M. (1996). Making things auditable. Accounting, Organizations and Society, 21(2-3), 289-315. doi:10.1016/0361-3682(95)00004-610. Guszcza, J., Rahwan, I., Bible, W., Cebrian, M., & Katyal, V. (2018, December 01). Why We Need to Audit Algorithms. Retrieved from https://hbr.org/2018/11/why-we- need-to-audit-algorithmsWINTER 2019 / 2020 IBM Center for The Business of Government 91'