git ssb

0+

Grey the earthling / gkn.me.uk



Commit 93551f2df87e794a7fbd1278d35788bbcfaeda86

Browse Files

Add entry: Self-driving cars

Greg K Nicholson committed on 2/25/2018, 11:29:02 PM
Parent: fc5a0e958c95593c2959382c9db90e473b608713

Files changed

content/selfdrivingcars.mdadded
content/selfdrivingcars.mdView
@@ -1,0 +1,159 @@
1 …+---
2 …+title: >
3 …+ Self-driving cars: software crashes
4 …+date: 2018-02-25 23:28
5 …+description: “It just works automatically!”
6 …+
7 …+tags: software, Free Software, technology, trust, law, politics, capitalism
8 …+
9 …+links:
10 …+ - url: https://www.youtube.com/watch?v=nFZGpES-St8
11 …+ title: >
12 …+ Karen Sandler's talk about having a pacemaker-defibrillator
13 …+ that runs proprietary software
14 …+ (video)
15 …+ description: >
16 …+ It's literally screwed into her heart,
17 …+ but she can't legally fix it, or even see how it works
18 …+ - url: http://fortune.com/2016/04/12/self-driving-cars-safety-study/
19 …+ title: >
20 …+ It's Impossible to Find Out If Self-Driving Cars Are Safe: Report
21 …+ description: >
22 …+ “Even if autonomous vehicle fleets are driven 10 million miles,
23 …+ one still would not be able to draw statistical conclusions
24 …+ about safety and reliability.”
25 …+ — if only you could inspect the instructions…
26 …+ - url: https://opensource.org/osd-annotated
27 …+ title: The Open Source Definition
28 …+ rel: related
29 …+ type: text/html
30 …+ - url: https://reproducible-builds.org/
31 …+ title: Reproducible builds
32 …+ rel: related
33 …+ type: text/html
34 …+ - url: https://www.washingtonpost.com/news/energy-environment/wp/2015/09/18/epa-volkswagen-used-defeat-device-to-circumvent-air-pollution-controls/
35 …+ title: Volkswagen used ‘defeat device’ to illegally skirt air-pollution controls
36 …+ rel: related
37 …+ type: text/html
38 …+
39 …+---
40 …+
41 …+No car is self-driving.
42 …+
43 …+A “self-driving” car is piloted by software,
44 …+which is ultimately written by a person.
45 …+You don't know who that person was;
46 …+only that they were employed by a particular company.
47 …+
48 …+They were probably sitting in an office somewhere in California
49 …+when they wrote the code driving your car.
50 …+Maybe it was 17:30 on a Friday and,
51 …+despite caring sincerely about the work they were doing,
52 …+they happened to be distracted by the prospect of going home.
53 …+Maybe not. You don't know.
54 …+
55 …+**Do you trust that person with your life?**
56 …+
57 …+Well, the company hired them, so they can't be *completely* useless.
58 …+You trust the company's recruitment procedures.
59 …+…What *are* the company's recruitment procedures?
60 …+
61 …+Anyway, presumably there are processes in place to review the code,
62 …+and stop mistakes from making it into the final software.
63 …+Presumably.
64 …+You trust that there are, and that they work, and never fail.
65 …+
66 …+Now imagine the company has made it illegal
67 …+for you to see how the software works.
68 …+**Are you *sure* you trust this company with your life?**
69 …+
70 …+---
71 …+
72 …+There should be a law saying that if a vehicle can be piloted by software,
73 …+and it's capable of containing or hurting a human,
74 …+then all installed software must be [open source],
75 …+and you must be able to *prove* that
76 …+the source code corresponds to the software running in the car.
77 …+
78 …+[open source]: https://opensource.org/osd-annotated
79 …+
80 …+It has to be legally possible for the vehicle's owner (or prospective owner)
81 …+to discover how their car might behave in a life-or-death situation,
82 …+so they can decide whether they want to be responsible for the car's actions.
83 …+
84 …+#### Responsibility
85 …+
86 …+Logically, the manufacturer who wrote the software would be responsible,
87 …+but they have no incentive to take responsibility
88 …+for their cars' imperfections. Doesn't make money.
89 …+Why admit your own flaws while your competitors keep schtum,
90 …+look better, and rake it in?
91 …+Any goodwill from better transparency will evaporate
92 …+as soon as someone dies in an accident.
93 …+
94 …+It's much safer to claim that
95 …+the human pilot should have taken control at the critical moment.
96 …+Capitalist governments won't argue with rich, profitably-taxable businesses.
97 …+
98 …+Car makers will only be transparent about how their cars behave
99 …+if they're obliged to by law.
100 …+
101 …+
102 …+#### Open source
103 …+
104 …+Merely having access to the software's source code isn't enough.
105 …+It must be legal to reuse the source code, for several reasons.
106 …+
107 …+Morally, if Non-Specific Engines Ltd writes an algorithm
108 …+that's better at saving lives than any other algorithm,
109 …+shouldn't Acme Motors be *obliged* to used the safer algorithm
110 …+in their cars, rather than forbidden?
111 …+
112 …+Practically, you need software experts to audit the code.
113 …+You want the code checked by an independent expert
114 …+in the field of vehicle automation
115 …+— not a business partner of the manufacturer —
116 …+and that person will be a software developer.
117 …+
118 …+If they use a similar concept in their own work later,
119 …+Mom's Friendly Car Company could threaten to sue them,
120 …+claiming they copied the code illegally.
121 …+Software developers are rarely as rich as car companies;
122 …+even the threat of a lawsuit would mean that in practice
123 …+the code would go unchecked.
124 …+
125 …+And again, morally, you can save lives here,
126 …+by letting the developer reuse the good code.
127 …+
128 …+
129 …+#### Reproducible builds
130 …+
131 …+Lastly, it needs to be possible to prove that
132 …+the audited code is actually the code running in the car.
133 …+You want an independent auditor to build the software for themself,
134 …+in a development environment they trust,
135 …+and get the exact same output as what's in the car.
136 …+It must be possible to [build the software reproducibly].
137 …+
138 …+[build the software reproducibly]: https://reproducible-builds.org/
139 …+
140 …+Otherwise checking the code is pointless —
141 …+you still have to trust the car manufacturer,
142 …+and you can't be sure the software's behaviour doesn't
143 …+[deviate in subtle ways in very specific situations].
144 …+Maybe you don't care about any subtle differences, but maybe you do.
145 …+The driver should at least be honest with you, and you can decide for yourself.
146 …+
147 …+[deviate in subtle ways in very specific situations]:
148 …+https://www.washingtonpost.com/news/energy-environment/wp/2015/09/18/epa-volkswagen-used-defeat-device-to-circumvent-air-pollution-controls/
149 …+
150 …+---
151 …+
152 …+None of this will make sure a self-driving car is perfectly safe.
153 …+All software has bugs.
154 …+But at least you'll know the driver was acting in good faith.
155 …+
156 …+Trade secrets and competitive advantage are not worth dying for.
157 …+
158 …+…Or you *could* just trust the big friendly company… right?
159 …+

Built with git-ssb-web