14.2 Privacy and Tracking
Retention & blast radius
- 1825M
- records held
- 1825M
- identifiable
- 9 mo
- past purpose
9 months of data serve no stated purpose. It cannot help the product and it can still leak, be subpoenaed, or be repurposed for something the user never agreed to. Deleting it is the cheapest security control available.
2.1 The relationship shift
- The service performs a service for the user. “When a system stores only data that a user has explicitly entered, because they want the system to store and process it — the user is the customer.”
- Behavioral tracking that genuinely helps the user.
- tracking which search results are clicked → better ranking
- “people who liked X also liked Y” → discovery
- A/B tests and user flow analysis → better UI
- The advertisers are the customers. “If the service is funded through advertising, the advertisers are the actual customers, and the users’ interests take second place. Tracking data becomes more detailed, analyses become further-reaching, and data is retained for a long time to build detailed profiles of each person.”
- “The user is given a free service and is coaxed into engaging with it as much as possible. The tracking of the user primarily serves not that individual but rather the needs of the advertisers who are funding the service.”
⇒ “This relationship can be appropriately described with a word that has more sinister connotations: surveillance.”
The thought experiment:
"Try REPLACING THE WORD DATA WITH SURVEILLANCE, and observe whether common phrases still sound so good:"
"In our surveillance-driven organization we collect real-time surveillance streams and store them in our surveillance warehouse. Our surveillance scientists use advanced analytics and surveillance processing in order to derive new insights."
"This thought experiment is UNUSUALLY POLEMIC FOR THIS BOOK, DESIGNING SURVEILLANCE-INTENSIVE APPLICATIONS, BUT STRONG WORDS ARE NEEDED TO EMPHASIZE THIS POINT."
2.2 The scale of it
"IN OUR ATTEMPTS TO MAKE SOFTWARE 'EAT THE WORLD,' WE HAVE BUILT THE GREATEST MASS SURVEILLANCE INFRASTRUCTURE EVER SEEN."
"We are rapidly approaching a world in which EVERY INHABITED SPACE CONTAINS AT LEAST ONE INTERNET-CONNECTED MICROPHONE — smartphones, smart TVs, voice-controlled assistants, baby monitors, and EVEN CHILDREN'S TOYS that use cloud-based speech recognition. MANY OF THESE DEVICES HAVE A TERRIBLE SECURITY RECORD."
"Surveillance of our LOCATION AND MOVEMENTS, our SOCIAL RELATIONSHIPS AND COMMUNICATIONS, our PURCHASES AND PAYMENTS, and our HEALTH DATA has become ALMOST UNAVOIDABLE. A surveillance organization may end up KNOWING MORE ABOUT A PERSON THAN THAT PERSON KNOWS ABOUT THEMSELVES — for example, IDENTIFYING ILLNESSES OR ECONOMIC PROBLEMS BEFORE THAT INDIVIDUAL IS AWARE OF THEM."
"EVEN THE MOST TOTALITARIAN AND REPRESSIVE REGIMES OF THE PAST COULD ONLY DREAM OF PUTTING A MICROPHONE IN EVERY ROOM AND FORCING EVERY PERSON TO CONSTANTLY CARRY A DEVICE CAPABLE OF TRACKING THEIR LOCATION. YET THE BENEFITS WE GET FROM DIGITAL TECHNOLOGY ARE SO GREAT THAT WE NOW VOLUNTARILY ACCEPT THIS STATE OF TOTAL SURVEILLANCE. THE DIFFERENCE IS JUST THAT THE DATA IS BEING COLLECTED BY CORPORATIONS TO PROVIDE US WITH SERVICES, RATHER THAN GOVERNMENT AGENCIES SEEKING CONTROL."
Why we accept it — and why that reasoning is a privilege:
"Perhaps you feel YOU HAVE NOTHING TO HIDE — in other words, YOU ARE TOTALLY IN LINE WITH EXISTING POWER STRUCTURES, YOU ARE NOT A MARGINALIZED MINORITY, AND YOU NEEDN'T FEAR PERSECUTION. NOT EVERYONE IS SO FORTUNATE."
"Or perhaps it's because THE PURPOSE SEEMS BENIGN — not overt coercion, MERELY BETTER RECOMMENDATIONS AND MORE PERSONALIZED MARKETING. HOWEVER, COMBINED WITH THE DISCUSSION OF PREDICTIVE ANALYTICS, THAT DISTINCTION SEEMS LESS CLEAR."
It's already not benign:
- "Behavioral data on CAR DRIVING, tracked by cars WITHOUT DRIVERS' CONSENT, affecting their INSURANCE PREMIUMS"
- "Health insurance coverage that depends on people WEARING A FITNESS TRACKING DEVICE"
- "The MOVEMENT SENSOR IN A SMARTWATCH can be used to work out WHAT YOU ARE TYPING (E.G., PASSWORDS) WITH FAIRLY GOOD ACCURACY. Sensor accuracy and algorithms for analysis are ONLY GOING TO GET BETTER"
2.3 Consent and freedom of choice — four reasons "they agreed" fails
Is the tracking necessary?
“Some forms Directly feed into improving features for users — click-through rate improving result ranking; products bought together suggesting related products. However, when tracking for content recommendations or to build user profiles for advertising, it is less clear whether this is genuinely in the user’s interest. is it necessary only because the ads pay for the service?”
Consent requires understanding, and there isn’t any
“Most users have Little knowledge of what data they are feeding into our databases or how it is retained and processed — And most privacy policies do more to obscure than to illuminate. without understanding what happens to their data, users cannot give meaningful consent.”
- ★ “Often, data from one user also says things about other people who are not users of the service and who have not agreed to any terms.”
- ★ “The derived datasets we discussed in the last few chapters — in which data from The entire user base may have been combined with behavioral tracking and external data sources — Are precisely the kinds of data that users cannot meaningfully understand.”
It is not an exchange
“Data is extracted from users through A one-way process, not a relationship with true reciprocity or a fair value exchange. there is no dialogue, no option for users to negotiate how much data they provide and what service they receive. The relationship is asymmetric and one-sided; the terms are set by the service, not by the user.”
“Just don’t use it” is not a free choice
“If a service is so popular that it is ‘Regarded by most people as essential for basic social participation,’ then It is not reasonable to expect people to opt out — its use is effectively mandatory. Especially when a service has Network effects, there is a social cost to people choosing not to use it.”
- ★ “These platforms are Designed specifically to engage users. Many use game mechanics and tactics common in gambling to keep users coming back.”
- ★ “Declining to engage is an option for Only the small number of people who are privileged enough to have the time and knowledge to understand the privacy policy, And who can afford to potentially miss out on social participation or professional opportunities. for people in a less privileged position, there is no meaningful freedom of choice; surveillance becomes inescapable.”
What GDPR actually requires of consent:
Consent must be "FREELY GIVEN, SPECIFIC, INFORMED, AND UNAMBIGUOUS," and the user must be able to "REFUSE OR WITHDRAW CONSENT WITHOUT DETRIMENT" — otherwise it is not considered freely given. Any request must be written "IN AN INTELLIGIBLE AND EASILY ACCESSIBLE FORM, USING CLEAR AND PLAIN LANGUAGE," and "SILENCE, PRE-TICKED BOXES OR INACTIVITY [DO NOT] CONSTITUTE CONSENT."
(Consent is not the only lawful basis — others include complying with other legislation, protecting somebody's life, and legitimate interest, which permits e.g. fraud prevention — "which fraudsters would presumably not consent to." Nevertheless, consent is the most frequently used basis in internet services.)
2.4 What privacy actually means
"Sometimes people claim that 'PRIVACY IS DEAD' on the grounds that some users are willing to post all sorts of things about their lives to social media. HOWEVER, THIS CLAIM IS FALSE AND RESTS ON A MISUNDERSTANDING OF THE WORD PRIVACY."
"HAVING PRIVACY DOES NOT MEAN KEEPING EVERYTHING SECRET; IT MEANS HAVING THE FREEDOM TO CHOOSE WHAT TO REVEAL TO WHOM, WHAT TO MAKE PUBLIC, AND WHAT TO KEEP SECRET. THE RIGHT TO PRIVACY IS A DECISION RIGHT: IT ENABLES EACH PERSON TO DECIDE WHERE THEY WANT TO BE ON THE SPECTRUM BETWEEN SECRECY AND TRANSPARENCY IN EACH SITUATION. IT IS AN IMPORTANT ASPECT OF A PERSON'S FREEDOM AND AUTONOMY."
The illustrative case:
"Someone with a RARE MEDICAL CONDITION might be VERY HAPPY to provide their private medical data to researchers if it might help develop treatments. HOWEVER, THIS PERSON MUST HAVE A CHOICE OVER WHO MAY ACCESS THIS DATA AND FOR WHAT PURPOSE. If information about their condition could HINDER THEIR ACCESS TO MEDICAL INSURANCE OR EMPLOYMENT, this person would probably be MUCH MORE CAUTIOUS."
The transfer — this is the chapter's sharpest reframe:
"When data is extracted from people through surveillance infrastructure, PRIVACY RIGHTS ARE NOT NECESSARILY ERODED BUT RATHER TRANSFERRED TO THE DATA COLLECTOR. Companies that acquire data essentially say, 'TRUST US TO DO THE RIGHT THING WITH YOUR DATA' — WHICH MEANS THAT THE RIGHT TO DECIDE WHAT TO REVEAL AND WHAT TO KEEP SECRET IS TRANSFERRED FROM THE INDIVIDUAL TO THE COMPANY."
"The companies in turn CHOOSE TO KEEP MUCH OF THE OUTCOME SECRET, BECAUSE TO REVEAL IT WOULD BE PERCEIVED AS CREEPY AND WOULD HARM THEIR BUSINESS MODEL (which relies on KNOWING MORE ABOUT PEOPLE THAN OTHER COMPANIES DO). Intimate information is revealed ONLY INDIRECTLY — in the form of TOOLS FOR TARGETING ADVERTISEMENTS TO SPECIFIC GROUPS (such as those SUFFERING FROM A PARTICULAR ILLNESS)."
"EVEN IF PARTICULAR USERS CANNOT BE PERSONALLY REIDENTIFIED FROM THE BUCKET OF PEOPLE TARGETED BY A PARTICULAR AD, THEY HAVE LOST THEIR AGENCY ABOUT THE DISCLOSURE OF SOME INTIMATE INFORMATION. IT IS NOT THE USER WHO DECIDES WHAT IS REVEALED TO WHOM ON THE BASIS OF THEIR PERSONAL PREFERENCES — IT IS THE COMPANY THAT EXERCISES THE PRIVACY RIGHT WITH THE GOAL OF MAXIMIZING ITS PROFIT."
On "creepiness" as a managed perception, and what engineers owe:
"Many companies want to AVOID BEING PERCEIVED AS CREEPY, AVOIDING THE QUESTION OF HOW INTRUSIVE THEIR DATA COLLECTION ACTUALLY IS AND INSTEAD FOCUSING ON MANAGING USER PERCEPTIONS. And even these perceptions are often managed poorly — SOMETHING MAY BE FACTUALLY CORRECT, BUT IF IT TRIGGERS PAINFUL MEMORIES, THE USER MAY NOT WANT TO BE REMINDED ABOUT IT."
"With any kind of data, WE SHOULD EXPECT THE POSSIBILITY THAT IT IS WRONG, UNDESIRABLE, OR INAPPROPRIATE IN SOME WAY, AND WE NEED TO BUILD MECHANISMS FOR HANDLING THOSE FAILURES. Whether something is 'undesirable' or 'inappropriate' is DOWN TO HUMAN JUDGMENT; ALGORITHMS ARE OBLIVIOUS TO SUCH NOTIONS UNLESS WE EXPLICITLY PROGRAM THEM TO RESPECT HUMAN NEEDS. AS ENGINEERS, WE MUST BE HUMBLE, ACCEPTING AND PLANNING FOR SUCH FAILINGS."
And the limit of privacy settings:
"Privacy settings are A STARTING POINT for handing back some control. HOWEVER, REGARDLESS OF THE SETTING, THE SERVICE ITSELF STILL HAS UNFETTERED ACCESS TO THE DATA and is free to use it in any way permitted by the privacy policy. Even if the service promises NOT TO SELL the data to third parties, IT USUALLY GRANTS ITSELF UNRESTRICTED RIGHTS TO PROCESS AND ANALYZE THE DATA INTERNALLY, OFTEN GOING MUCH FURTHER THAN WHAT IS OVERTLY VISIBLE TO USERS."
"This kind of LARGE-SCALE TRANSFER OF PRIVACY RIGHTS FROM INDIVIDUALS TO CORPORATIONS IS HISTORICALLY UNPRECEDENTED. Surveillance has always existed, BUT IT USED TO BE EXPENSIVE AND MANUAL, NOT SCALABLE AND AUTOMATED. Trust relationships have always existed — between a patient and their doctor, a defendant and their attorney — BUT IN THESE CASES THE USE OF DATA HAS BEEN STRICTLY GOVERNED BY ETHICAL, LEGAL, AND REGULATORY CONSTRAINTS."
2.5 Data as assets and power
The reframe from "exhaust" to "labor":
| The comfortable story | The accurate story |
|---|---|
| “Behavioral data is a byproduct — sometimes called ‘data exhaust’ — suggesting the data is worthless waste material. Viewed this way, behavioral analytics is a form of recycling that extracts value from data that would have been thrown away.” | “From an economic point of view, if targeted advertising is what pays for a service, then the user activity that generates behavioral data could be regarded as a form of labor.” |
| ⇒ “More correct would be to view it the other way around.” | “One could go even further and argue that the application with which the user interacts is merely a means to lure users into feeding more and more personal information into the surveillance infrastructure. The delightful human creativity and social relationships that often find expression in online services are cynically exploited by the data extraction machine.” |
Who wants the data, beyond the company that collected it:
"DATA BROKERS OPERATING IN SECRECY, purchasing, aggregating, analyzing, and reselling people's personal data. STARTUPS ARE VALUED BY THEIR USER NUMBERS, OR 'EYEBALLS' — THAT IS, BY THEIR SURVEILLANCE CAPABILITIES."
"GOVERNMENTS want it too, and they may seek to obtain it by means of SECRET DEALS, COERCION, LEGAL COMPULSION, OR SIMPLY THEFT. WHEN A COMPANY GOES BANKRUPT, THE PERSONAL DATA IT HAS COLLECTED IS ONE OF THE ASSETS THAT GET SOLD. And because data is difficult to secure, BREACHES HAPPEN DISCONCERTINGLY OFTEN."
"These observations have led critics to say that data is NOT JUST AN ASSET, BUT A 'TOXIC ASSET,' or at least 'HAZARDOUS MATERIAL.' MAYBE DATA IS NOT THE NEW GOLD, OR THE NEW OIL, BUT RATHER THE NEW URANIUM."
The five ways it goes wrong even with good intentions:
"Even if we think we are capable of preventing abuse, whenever we collect data WE NEED TO BALANCE THE BENEFITS WITH THE RISK OF IT FALLING INTO THE WRONG HANDS:"
- "Computer systems may be COMPROMISED BY CRIMINALS OR HOSTILE FOREIGN INTELLIGENCE SERVICES"
- "Data may be LEAKED BY INSIDERS"
- "The company may FALL INTO THE HANDS OF UNSCRUPULOUS MANAGEMENT THAT DOES NOT SHARE OUR VALUES"
- "The country may be TAKEN OVER BY A REGIME THAT HAS NO QUALMS ABOUT COMPELLING US TO HAND OVER THE DATA"
- Bankruptcy — the data is sold as an asset
"When collecting data, WE NEED TO CONSIDER NOT JUST TODAY'S POLITICAL ENVIRONMENT, BUT ALL POSSIBLE FUTURE GOVERNMENTS. There is NO GUARANTEE that every government elected in the future will respect human rights and civil liberties. As Bruce Schneier observes: 'IT IS POOR CIVIC HYGIENE TO INSTALL TECHNOLOGIES THAT COULD SOMEDAY FACILITATE A POLICE STATE.'"
On power:
"'KNOWLEDGE IS POWER,' as the old adage goes. And furthermore: 'TO SCRUTINIZE OTHERS WHILE AVOIDING SCRUTINY ONESELF IS ONE OF THE MOST IMPORTANT FORMS OF POWER.' This is why totalitarian governments want surveillance: IT GIVES THEM THE POWER TO CONTROL THE POPULATION. Although today's technology companies are NOT OVERTLY SEEKING POLITICAL POWER, the data and knowledge they have accumulated — MUCH OF IT SURREPTITIOUSLY, OUTSIDE OF PUBLIC OVERSIGHT — NEVERTHELESS GIVES THEM A LOT OF POWER OVER OUR LIVES."
2.6 Remembering the Industrial Revolution
| The Industrial Revolution | The information age |
|---|---|
| “brought sustained economic growth and significantly improved living standards in the long run — yet it also came with major problems:” | The internet, data storage and processing, and software-driven automation are having a major impact on the global economy and human society. |
| “Just as the Industrial Revolution had a dark side that needed to be managed, our transition to the information age has major problems that we need to confront and solve. The collection and use of data is one of those problems.” |
“It took a long time before safeguards were established: environmental protection regulations, safety protocols, laws prohibiting child labor, health inspections for food. Undoubtedly, the cost of doing business increased when factories were no longer allowed to dump waste into rivers, sell tainted foods, or exploit workers. But society as a whole benefited hugely from these regulations, and few of us would want to return to a time before.”
Bruce Schneier's framing, which is the moral center of the chapter:
"DATA IS THE POLLUTION PROBLEM OF THE INFORMATION AGE, AND PROTECTING PRIVACY IS THE ENVIRONMENTAL CHALLENGE. ALMOST ALL COMPUTERS PRODUCE INFORMATION. IT STAYS AROUND, FESTERING. HOW WE DEAL WITH IT — HOW WE CONTAIN IT AND HOW WE DISPOSE OF IT — IS CENTRAL TO THE HEALTH OF OUR INFORMATION ECONOMY. JUST AS WE LOOK BACK TODAY AT THE EARLY DECADES OF THE INDUSTRIAL AGE AND WONDER HOW OUR ANCESTORS COULD HAVE IGNORED POLLUTION IN THEIR RUSH TO BUILD AN INDUSTRIAL WORLD, OUR GRANDCHILDREN WILL LOOK BACK AT US DURING THESE EARLY DECADES OF THE INFORMATION AGE AND JUDGE US ON HOW WE ADDRESSED THE CHALLENGE OF DATA COLLECTION AND MISUSE."
"WE SHOULD TRY TO MAKE THEM PROUD."
2.7 Legislation and self-regulation
What GDPR demands, and the direct conflict:
GDPR: personal data must be "COLLECTED FOR SPECIFIED, EXPLICIT AND LEGITIMATE PURPOSES AND NOT FURTHER PROCESSED IN A MANNER THAT IS INCOMPATIBLE WITH THOSE PURPOSES" and be "ADEQUATE, RELEVANT AND LIMITED TO WHAT IS NECESSARY."
"However, THIS PRINCIPLE OF DATA MINIMIZATION RUNS DIRECTLY COUNTER TO THE PHILOSOPHY OF BIG DATA, WHICH IS TO MAXIMIZE DATA COLLECTION, TO COMBINE IT WITH OTHER DATASETS, AND TO EXPERIMENT AND EXPLORE IN ORDER TO GENERATE NEW INSIGHTS. EXPLORATION MEANS USING DATA FOR UNFORESEEN PURPOSES — WHICH THE GDPR STATES IS THE OPPOSITE OF THE 'SPECIFIED AND EXPLICIT' PURPOSES FOR WHICH THE DATA MUST HAVE BEEN COLLECTED."
"While this regulation has had SOME EFFECT ON THE ONLINE ADVERTISING INDUSTRY, IT HAS BEEN WEAKLY ENFORCED AND DOES NOT SEEM TO HAVE LED TO MUCH OF A CHANGE IN CULTURE AND PRACTICES ACROSS THE WIDER TECH INDUSTRY."
The counter-argument, given fairly:
"Companies broadly OPPOSE REGULATION as being a burden and a hindrance to innovation. TO SOME EXTENT, THAT OPPOSITION IS JUSTIFIED. For example, sharing medical data creates clear risks to privacy BUT ALSO POTENTIAL OPPORTUNITIES: HOW MANY DEATHS COULD BE PREVENTED IF DATA ANALYSIS WERE ABLE TO HELP US ACHIEVE BETTER DIAGNOSTICS OR FIND BETTER TREATMENTS? OVERREGULATION MAY PREVENT SUCH BREAKTHROUGHS. IT IS DIFFICULT TO BALANCE THE POTENTIAL OPPORTUNITIES WITH THE RISKS."
The call to action:
"FUNDAMENTALLY, WE NEED A CULTURE SHIFT IN THE TECH INDUSTRY WITH REGARD TO PERSONAL DATA. WE SHOULD STOP REGARDING USERS AS METRICS TO BE OPTIMIZED, AND REMEMBER THAT THEY ARE HUMANS WHO DESERVE RESPECT, DIGNITY, AND AGENCY."
- "We should SELF-REGULATE our data collection and processing practices in order to ESTABLISH AND MAINTAIN THE TRUST of the people who depend on our software."
- "We should take it upon ourselves to EDUCATE END USERS ABOUT HOW THEIR DATA IS USED RATHER THAN KEEPING THEM IN THE DARK."
- "We should allow each individual to MAINTAIN THEIR PRIVACY (i.e., their control over their own data) and NOT STEAL THAT CONTROL FROM THEM THROUGH SURVEILLANCE."
"OUR INDIVIDUAL RIGHT TO CONTROL OUR DATA IS LIKE THE NATURAL ENVIRONMENT OF A NATIONAL PARK: IF WE DON'T EXPLICITLY PROTECT AND CARE FOR IT, IT WILL BE DESTROYED. IT WILL BE THE TRAGEDY OF THE COMMONS, AND WE WILL ALL BE WORSE OFF FOR IT. UBIQUITOUS SURVEILLANCE IS NOT INEVITABLE. WE ARE STILL ABLE TO STOP IT."
"As a FIRST STEP, WE SHOULD NOT RETAIN DATA FOREVER, BUT PURGE IT AS SOON AS IT IS NO LONGER NEEDED, AND MINIMIZE WHAT WE COLLECT IN THE FIRST PLACE. DATA YOU DON'T HAVE IS DATA THAT CAN'T BE LEAKED, STOLEN, OR COMPELLED BY GOVERNMENTS TO BE HANDED OVER."
"AS PEOPLE WORKING IN TECHNOLOGY, IF WE DON'T CONSIDER THE SOCIETAL IMPACT OF OUR WORK, WE'RE NOT DOING OUR JOB."