14. Doing the Right Thing
Chapter 14 of Designing Data-Intensive Applications — 7 sections.
"Feeding AI systems on the world's beauty, ugliness, and cruelty, but expecting it to reflect only the beauty is a fantasy." — Vinay Uday Prabhu and Abeba Birhane
The premise:
"Every system is built for a purpose; EVERY ACTION WE TAKE HAS BOTH INTENDED AND UNINTENDED CONSEQUENCES. The purpose may be as simple as making money, BUT THE CONSEQUENCES MAY BE FAR-REACHING. We, the engineers building these systems, have a RESPONSIBILITY to carefully consider those consequences and to ensure that our decisions do not cause harm."
"We talk about data as an abstract thing, BUT REMEMBER THAT MANY DATASETS ARE ABOUT PEOPLE: their behavior, their interests, their identity. WE MUST TREAT SUCH DATA WITH HUMANITY AND RESPECT. USERS ARE HUMANS TOO, AND HUMAN DIGNITY IS PARAMOUNT."
On professional ethics as actually practiced:
"There are guidelines to help software engineers navigate these issues, such as the ACM CODE OF ETHICS AND PROFESSIONAL CONDUCT, BUT THEY ARE RARELY DISCUSSED, APPLIED, AND ENFORCED IN PRACTICE. As a result, ENGINEERS AND PRODUCT MANAGERS SOMETIMES TAKE A CAVALIER ATTITUDE TO PRIVACY and potential negative consequences of their products."
"A technology is NOT GOOD OR BAD IN ITSELF — what matters is HOW IT IS USED AND HOW IT AFFECTS PEOPLE. This is true of a software system like a search engine IN MUCH THE SAME WAY AS IT IS OF A WEAPON LIKE A GUN. THE ETHICAL RESPONSIBILITY IS OURS TO BEAR; IT IS NOT SUFFICIENT FOR SOFTWARE ENGINEERS TO FOCUS EXCLUSIVELY ON THE TECHNOLOGY AND IGNORE ITS CONSEQUENCES."
"In contrast to much of computing, the concepts at the heart of ethics are NOT FIXED OR DETERMINATE in their precise meaning; THEY REQUIRE INTERPRETATION, WHICH MAY BE SUBJECTIVE. What makes something 'good' or 'bad' is not well defined, and SERIOUS DISCOURSE ON THE SUBJECT AMONG COMPUTING PROFESSIONALS IS LACKING."
"ETHICS IS NOT GOING THROUGH A CHECKLIST TO CONFIRM YOU COMPLY; IT'S A PARTICIPATORY AND ITERATIVE PROCESS OF REFLECTION, IN DIALOG WITH THE PEOPLE INVOLVED, WITH ACCOUNTABILITY FOR THE RESULTS."