In this episode with David Colarusso, we talk about Suffolk University Law School’s Legal Innovation and Technology Lab, what it is and what it hopes to achieve, as well as a concept known as “Chesterton’s fence” and how you can better use risk assessment tools in your firm.
David Colarusso is the Director of Suffolk University Law School’s Legal Innovation and Technology Lab. An attorney and educator by training, he has worked as a public defender, data scientist, software engineer, and high school physics teacher. He is the author of a programming language for lawyers, QnA Markup, an award-winning legal hacker, ABA Legal Rebel, and Fastcase 50 honoree. In 2017 he was named one of the ABA’s top legal tweeters.
Thanks to ARAG, Ruby Receptionists, and TextExpander for sponsoring this episode!
Don’t miss an episode of The Lawyerist Podcast! Subscribe now in your favorite podcast app.
Transcript generated by Rev.com.
Voiceover: Welcome to The Lawyerist podcast, a series of conversations about law practice. Each week we talk with legal entrepreneurs and innovators about building a successful law practice in today’s challenging and constantly changing legal market. Now, here are your hosts.
Sam Glover: Hi, I’m Sam Glover.
Aaron Street: And, I’m Aaron Street, and this is episode 225 of The Lawyerist podcast, part of the Legal Talk Network. Today, we’re talking with David Colarusso about Chesterton’s Fence and the importance of humility when attempting to solve legal problems with technology or anything else.
Sam Glover: Today’s podcast is brought to you by ARAG, Ruby Receptionist, and TextExpander. We wouldn’t be able to do this show without their support, so stay tuned. We’ll tell you more about them later on.
Aaron Street: Before recording this intro, I tried to place a wager on what the odds where that I would pronounce Chesterton’s Fencer correctly on the first take-
Sam Glover: You did great!
Aaron Street: … and now I’ve done it twice.
Sam Glover: I know. Three times even.
Aaron Street: It’s amazing. Lucky me.
Sam Glover: Yeah, that’s amazing.
Aaron Street: A couple of episodes we had Allison Shields on 200 episodes after her first appearance and David Colarusso is now appearing exactly 205 episodes after his first appearance.
Sam Glover: That’s the buffer. If you want to come back, it’s 200 episodes.
Aaron Street: Apparently. Maybe not.
Sam Glover: Nice. That feels excessive.
Aaron Street: I thought it would be fun to take a few minutes this morning and chat about threat modeling.
Sam Glover: Oh, threat modeling. I love this because it’s the it depends answer to the question of what level of security do I need that lawyers hate.
Aaron Street: Plus, it allows us to talk about hackers.
Sam Glover: Yeah.
Aaron Street: And, terrorists and whatever, which is always fun.
Sam Glover: Absolutely. Here’s the thing about threat modeling and what it is. Threat modeling is a framework to help you think about the answer to the question, how much tech security do I need to be doing.
Aaron Street: Right, so we talked, I think, two episodes ago about data security needs generally and some of our default best practices law firms can use to make sure that they are protecting their law firm and client data in an appropriately confidential way. There’s some generic best practices there, but generic obviously only applies as a default standard, but not a specific standard to you. The importance is in being competent with protecting your clients’ data, is understanding what specific threats might actually apply to your firm and your clients’ issues.
Aaron Street: For some firms, the default standard potentially might even be more sophisticated than the actual threats that that firm faces, but for some firms, particularly those that deal with a lot of financial data or personally identifiable information or have opposing parties who might know a lot about your client, like in divorce matters or if you’re dealing with really sophisticated criminal matters or international business … There are any number of-
Sam Glover: There’s so many variables.
Aaron Street: … case types or practice areas where a really sophisticated understanding of who might want to try to destroy this data, capture this data, steal and share this data, or how they might do it. These are things that all firms need to at least spend a little bit of time thinking through so that if those risks ever become realized, that you’ve made sure that you’re protecting yourself as best you can.
Sam Glover: One of the places I like to start with on this, because I like to help lawyers understand that a threat model is not just security hacker geek stuff, is I like to start by talking about the paper on your desk and the lock on your front door. What are the risks to somebody reading confidential client information that you might leave out on your desk in the evening? Well, anybody who wanders into your office, anybody with a key to your office, anybody who, if the front door is unlocked, anybody who’s passing by and gets curious. That’s a starting point for one piece of your threat model.
Sam Glover: Another piece of it is obviously going to be your email and your file server and things like that, but you need to be thinking holistically about what are the specific characteristics about your law practice and how you store information and where and how you secure it, and then how can you actually take the necessary steps, the reasonable necessary steps to make sure everything is secure.
Aaron Street: Yeah, at it’s most basic, this threat model analysis dovetails a lot with understanding basic data security and with protecting your files from destruction too. Part of your threat model is if my office burned down, then what? Some of these are risks that have a low likelihood of occurring but would have tremendous harm if they did, and others have a really high likelihood of occurring, but maybe less harm if they did.
Sam Glover: You’re doing the calculus of risk for data security edition.
Aaron Street: Yeah, and unless you are a very interesting boutique firm in the news for representing international terrorists or you represent some IP holder in a Chinese espionage case or something, this doesn’t have to be an extended analysis that you’re investing consultants and time and money into.
Sam Glover: You can do it yourself.
Aaron Street: You need to spend some time thinking about it.
Sam Glover: We’ve got a worksheet to help you think through your threat model, and you can find it in our Insider Library. We also have the security download, the whitepaper that we talked about before, the Four-Step Security Upgrade, which will walk you through four things that we think are really easy and simple that everyone should do no matter what your threat model looks like. If you want a more holistic view of how your law firm stands both technologically and otherwise, then you should use the Small Firm Scorecard. You can find the link on the front page of Lawyerist or go to lawyerist.com/scorecard, and evaluate your firm on your own scale on threat modeling and a number of other factors about business health, technology savviness, financial health, all the kinds of things.
Sam Glover: With that, here’s my conversation with David, and we’ll explain what Chesterton’s Fence is and go into some detail around some of the implications of it.
Aaron Street: You pronounced it right the first time too!
Sam Glover: I know. We’re so good.
David Colarusso: Hi, I’m David Colarusso. I’m a practitioner in residence and the director of the Legal Information and Technology Lab at the University Law School. I’m looking forward to talking all things tech and legal.
Sam Glover: Hey, David. Welcome to the podcast. You are currently the practitioner in residence, which I understand is not actually a professor, but we’ve had you before as a public defender, and before that you were, I think, a science teacher. Is that right?
David Colarusso: Yep, and I had a small software company on the side, and that probably rounds out all the things for which I’ve had to pay taxes on.
Sam Glover: That sounds about right. You’ve also built some software. You’ve got some open source projects. You’re kind of all over the place, I guess. Not in a bad way, but in a good way.
David Colarusso: Yeah, I haven’t had much chance to work on my woodworking, though, lately, so I’m a little sad.
Sam Glover: Well, let’s talk about your work at Suffolk University. What is the Legal Innovation and Technology Lab?
David Colarusso: It’s a great excuse for me to use the initials L-I-T. Of all things that it is, it’s definitely LIT. The Legal Information and Technology Lab, we’re a joint program of our clinical programs at the law school and our Legal Information and Technology Concentration. We basically operate as a consultancy inside the law school, providing data science and legal tech consulting to external and internal clients. That could be governments, nonprofits, sometimes small firms, Palace Law, which I know your listeners know. We’ve worked with them on some stuff.
David Colarusso: Then, we also work with internal clients. So, we have students that are in our clinics who are normal clinical students with a reduced caseload, and they actually will work basically as part of the consultancy but working for the clinics as their clients, so internal innovation folks in our clinical programs.
Sam Glover: So, is the goal to make money for Suffolk or to … What’s the reason for, especially with the external stuff, doing that external work?
David Colarusso: It’s really the same goal as clinical programs. It’s experiential education. It’s the understanding that you never really learn something as well as when you’re doing it. What we end up having students do is work as part of teams identifying issues and finding technical solutions for those issues when appropriate. Sometimes those solutions might, however, be less technical than some people might suppose. It might just be process improvement, but really it’s about finding the right tools to solve a problem.
David Colarusso: Also, in addition to our quasi-consulting work, which most of it is at no cost because we tend to work with governments and nonprofits [inaudible 00:08:38], we also have a class that goes along with the lab that I teach called Coding the Law, and that helps to acquaint students with the actual ins and outs of the technology and actually have them get their hands dirty and actually build some stuff. There the idea is that you never really understand something until you can create it yourself. It’s to gain them an understanding that they can use to go out into the world in their practice and help understand when someone’s saying something about technology. Whether or not it be someone trying to sell them something for their practice or someone saying something as it affects an issue that they’re dealing with for a client and that they’re going to be able to … Whether or not that person’s actually telling them how things go, or if they’re just trying to sell them some BS.
Sam Glover: You and I were chatting with someone whose name escapes me at the moment from Harvard’s Legal Innovation Lab or whatever they call it. The Library Innovation-
David Colarusso: Yeah, Adam Ziegler.
Sam Glover: … thing. Anyway …
David Colarusso: Yeah, LIL, their Library Innovation Lab.
Sam Glover: Yes, Adam Ziegler. He pitched the idea that a lawyer who has a background in technology such as they can learn at Suffolk is really a lawyer of a higher skill level because you have new, more robust tools in your toolbox for bringing to law practice problems. Not necessarily just business problems, although those too, but let’s say you get a crazy batch of discovery. Maybe if you understand how to work with large volumes of data and information, you have a different skillset for analyzing that than a lawyer who might not have those skills.
David Colarusso: Yeah. [inaudible 00:10:06] actually a … That’s one of Adam’s colleagues. That was probably a conversation with Jack Cushman, who is one of Adam’s colleagues. He teaches a similar class where they teach coding to lawyers as well, and that’s the entire premise. Actually, there’s a handful, maybe not quite two dozen classes in the U.S. where people are teaching coding to attorneys in law school, and they have a range of different focuses. Actually, Jack and mine are probably two of the ones that are closest to each other in that we’re looking at using technology as a tool to help filter a student’s understanding of their legal practice.
David Colarusso: Some people are just looking at it as we’ll just provide you with these skills, and we’re teaching you a class as we teach anyone a coding class absent that context. But I know one of the things that I tend to … Then, some are doing it on the very theoretical side. They’re looking at what are the consequences of technology legally. What do we need to know about machine bias and other issues that might come up through the use of technology? What we’re trying to do is actually say, “Well, the way to really understand those issues is to understand how things work. So, we’re going to give you a foundation and have you get your hands dirty, and then also talk about these things.”
David Colarusso: For example, we have, as one of our required readings in my course, Kathy O’Neil’s Weapons of Math Destruction, which is an exploration of how algorithms can get misused in society and the consequences of not actually understanding what it is that’s behind that black box.
Sam Glover: You’ve just teed up the pivot perfectly, which is the reason you and I decided we wanted to do a podcast was to talk about Chesterton’s Fence. We can get to that in a minute, but I just want to set this up where one of the things we keep seeing, it seems like, is people who are trying to … Whether they’re trying to innovate around the legal industry, from the legal industry, within it, whatever, or they’re trying to disrupt it, whatever it is, there’s a lot of banging heads together when … I’m thinking of, for example, the DoNotPay app, or a number of the things that I’ve seen at hackathons for law students or whatever.
Sam Glover: Often, what it seems like, is there’s a disconnect between the actual problem that they’re trying to solve and their understanding of the problem. So, they end up building a solution that is actually solving a problem that’s already been solved or that isn’t an actual problem that needs solving. As a result, nothing really happens with it. You get a lot of noise about how great this thing is or how much it can do, and then there’s a lot of sound and fury signifying nothing it seems like. Or, you get something like, let’s say, Bird and Lime, where they’re following in the footsteps of Uber and Lyft, but you drop a solution into a market without really considering the legal problem, and all of a sudden, your biggest problem is your legal problems because you didn’t really account for them.
Sam Glover: Tell us about Chesterton’s Fence.
David Colarusso: Yeah, so Chesterton’s Fence. It’s interesting. It’s often people, if they know it, they know it from a quotation from, I think, Bartlett’s Quotation has John Kennedy citing it as Chesterton. His encapsulation was, “Don’t ever take down a fence until you know the reason why it was put up.” Chesterton’s statement was a little-
Sam Glover: Which is about half of it, right?
David Colarusso: Yeah, yeah. And Chesterton’s statement’s a little more complicated than that. It’s interesting historic history because it’s John F. Kennedy, refracting Frances Chesterton’s comment from one of his apologetics on Catholicism in the early part of the 20th Century. There basically he was talking about reformers and social reformers and saying, basically, if someone comes and says, “I want to blow up the system, because I don’t see any point to why it’s this way,” then he’s going to say, “No, don’t. I’m not going to let you blow up the system,” except here he’s talking about take down the fence.
David Colarusso: When you come back, and you tell me, “Oh, now I see why that fence is there, then maybe I’ll let you take it down.” The point being is that you should be humble about the notion that maybe some of the things that you don’t understand why they’re there are there for a reason, and it’s worth understanding that before you go tearing things down.
Sam Glover: I think about that, in the sense of the legal system. It’s really easy to listen to a lot of criminal, true crime podcasts and say, “Our legal system’s broken,” which is true. But it’s not broken in the ways that many people think it’s broke. It’s not broken because it produces bad outcomes. The bad outcome’s a result of the way it might be broken, but it’s you’re missing so much by just saying it’s broken. There are good reasons why it is the way it is, and there’s some reasons that maybe it should be changed, but you can’t start to address those until you actually have taken it apart and understood it.
David Colarusso: Yeah. Well, the rules of evidence, are probably the place where most people run into this because they say, “Well, why-
Sam Glover: Because they’re nuts. From an outside view, they’re nuts.
David Colarusso: Right. On the outside, you get a trial going up in the court of public opinion, and everything is fair game. You get hearsay. You get all this speculation, and then you wonder, “How is it that the jury came to a certain decision?” You recognize, “Well, they only saw part of the evidence.” Some people are saying, “Well, that’s silly. Why didn’t they see everything?” It turns out there are really good reasons why they didn’t see everything because we don’t trust the process if certain things can get in that might have an undue effect on people’s thinking.
David Colarusso: It turns out that working fast and breaking things only works as a strategy when the things you break aren’t really that important. So that you really have to think proportionally to, “If I break this thing, what’s the worst that could happen?” This is a model that is adopted in software development and put out a minimum viable product. Nicole [Braddock 00:15:32] talks about this a lot, the idea that you can put out a minimum viable product for legal. Well, you got to understand what is really that minimum viable product for legal because you can’t just say, “Oh, well, this didn’t work out for you. So, I’m sorry.” You have to make sure that it actually …
David Colarusso: It’s like the minimum viable product for an airplane. You want the airplane to fly, right? It’s no good for you if you kill half your users.
Sam Glover: You can’t A/B test within the legal industry, right?
David Colarusso: Yeah. Well …
Sam Glover: It’s pretty hard to say to people, “Okay, so we’re going to test these different evidentiary rules and different courtrooms and see if we think the results are good or better in one than the other.” You can’t redo the whole trials for the B test if it doesn’t win or something. It’s harder. It’s not that it isn’t already broken and doesn’t need work, it’s just that it’s way more complicated than people who are trying to unbreak it are trying to think through it in many cases.
David Colarusso: Yeah, and I would I push back a little bit on saying you can’t A/B test in law because I think one of the things that’s missing is real, randomized controlled trials in law. The Access to Justice Lab across the river at Harvard, that’s their big thing. They want to, say, set up real, randomized controlled trials on things and seeing what actually works, not just figure this all out by gut. People say, “Well, you can’t do that ethically.” Well, except this is done all the time in the medical profession where it’s people’s lives on the line. That’s [inaudible 00:16:54] often people’s lives and their liberty interest on the line in courts too, so I don’t think that those are out of sync.
David Colarusso: Obviously, they found ways to do that ethically and responsibly because they realized that the knowledge you gain from real, randomized controlled trials actually is very useful to knowing whether or not this thing you think is working is working. Because sometimes the thing you think is working is not.
Sam Glover: Well, and I suppose-
David Colarusso: Yeah, so I just push back. Yeah.
Sam Glover: I guess that makes a ton of sense.
David Colarusso: Yeah. My point is, it’s just more complicated.
Sam Glover: Yeah. It is just more complicated, but the alternative of it’s complicated and throw up your hands and do nothing also doesn’t … That’s not good either.
David Colarusso: Yeah, I think, well, that’s the easy way out because you don’t have to think about what’s hard or try to improve upon things. That’s a very defeatist attitude that … I think that one of the reasons I’m drawn to innovation and its use in law is the possibility to make things better. There’s a very clear point that the lab … It’s the Legal Innovation and Technology Lab, not just the technology lab. The idea, how do we actually make changes that are going to improve things? That’s really what we’re all about, and you have to know why things are the way they are before you actually can do that.
David Colarusso: It’s Chesterton’s Fence. There we go. We tied it all together.
Sam Glover: There we go. So we’ve teed up Chesterton’s fence, and now we’re going to take a quick break. When we come back, we’re going to follow it down the path through some of the consequences that may or may not be meaningful within law and within some related subjects. So, we’ll be back in a minute.
Sam Glover: With TextExpander you don’t have to waste time retyping things you’ve already worded perfectly. Instead, just use gathered snippets of information using simple keyboard shortcuts or custom abbreviations. You can capture the important pieces of your emails, directions, messages, and data as snippets, so you never have to retype them again. From correcting your personal typos and defining industry terms to whole email templates, reusing your info has never been faster, and it works everywhere you type. TextExpander is available for Mac, Windows, iPhone, and iPad, and now Chrome too. Listeners can get 20% off their first year by visiting textexpander.com/podcast.
Sam Glover: How cool would it be to grow your practice in your chosen area of law without spending time or money on business development? Now you can with ARAG. ARAG is a leader in legal insurance, and it works a lot like medical insurance. When you become a provider on the ARAG network, you consult with and represent clients for various legal issues from writing a will to dealing with bankruptcy or divorce. ARAG does the rest. Seriously. They’ll connect you with new clients. They’ll pay you directly. They’ll even collect client feedback and share it with you so you can keep growing your business.
Sam Glover: Visit araglegal.com/lawyerist. That’s A-R-A-Glegal.com/lawyerist to join the network for no fee and start growing your practice. It is all about the growth. In fact, more than 90% of ARAG members say they are more likely to consult with an attorney when something goes up than if they didn’t have legal insurance. Check it out at araglegal.com/lawyerist. That’s A-R-A-Glegal.com/lawyerist.
Sam Glover: Support for today’s episode comes from Ruby Receptionist, helping legal professionals like you deliver legendary service and grow your practice with live receptionist and chat services. At a fraction of the cost of a full-time hire, Ruby’s live, U.S.-based team greets your clients personally when they call or visit your website. Ruby can route calls to you or connect chats to call based on your customized directions. Your live receptionist can collect new client intake, answer frequently asked questions, and make follow up calls. Ruby streamlines billing with call tracking and Clio, Rocket Matter and Clio Grow integrations. Ruby can send messages to you via the mobile app, email, or text and much more, helping you grow your firm.
Sam Glover: Thousands of solo and small firm attorneys across the country rely on Ruby to turn callers and website visitors into clients, and now you can try Ruby for free. Call 844-715-7829 today or visit callruby.com/lawyeristpodcast to get started with your 14-day free trial. That’s 844-715-7829, or callruby.com/lawyeristpodcast.
Sam Glover: Okay, we’re back. So, we introduced Chesterton’s fence, which you’ve summed up in the sense of be humble. In a way, I think this means approaching legal problems or any problems like a designer would where you don’t presume to understand it until you’ve done your research into it.
David Colarusso: Yeah, and I think the thing that’s interesting is that it’s often positing the existence of a pre-designed system. It’s not just that you have to understand your user’s need and address it, but you need to understand what user needs were being addressed by the prior design. This is easiest to see when you come upon a system that has design components that are not really called into question by the existence of a new technology, or just part of a process or procedure.
David Colarusso: Actually, there the court systems, for the most part, are a really good case of that when we’re talking about rules of evidence, for example. When it gets tricky is when technology introduces a new ability that didn’t exist at the time of the original design, and then what you’ve got to do is ask not just why is it this way … Because maybe the way it’s this way is because there wasn’t a possibility for … If we talk about Uber and Lyft, there wasn’t the possibility for people just to call cars with their phone. This massive regulatory apparatus and taxi medallions was created as a way to help control that, but now we can just have people on our phones calling cars. Well, the question is, is there a reason why someone might want to keep that regulatory framework in place even in light-
Sam Glover: Well, let’s tee this up. Let’s talk about open data, like public access to court records. This is something that I’ve talked about a few times before. There are all kinds of reasons why open data would be a good thing. It opens the door to innovation. It drops barriers to public access, who should have a right to get information and stuff. What we haven’t really talked about before is what might happen if you actually do take down the barrier, if PACER wasn’t so difficult to access and didn’t cost money or if state court records weren’t hidden behind archival walls and things like that. So, keep going.
David Colarusso: Well, so it’s interesting. You do have some [inaudible 00:22:48] the laboratories in democracy in all the 50 states, and multiple other legal jurisdictions in the US are interesting to see how they deal with this because you have different jurisdictions in the state court context dealing with this differently. You have some places like Florida where they just effectively … although it still comes down to your county by county … but you have some counties that basically throw their entire docket up online, including just about everything. Sometimes there’s a nominal attempt to maybe not show names of parties, but sometimes there, it’s just everything up there.
David Colarusso: That creates a bunch of unintended consequences because there was traditionally a barrier to, in aggregate, getting all that information and getting it easily. That barrier prevented certain abuses. The one that most people really resonates with them are tenants being blackballed by potential landlords because they’ve had some matter against a prior landlord. The idea that you’re unlucky enough to get someone who’s not doing their job as a landlord, and so you have to get to the point where you take some legal action, and now, because of that, you’re never going to be able to rent again, is something that people really don’t like.
David Colarusso: That’s does abuse that-
Sam Glover: Another example of it is redaction of things like social security numbers. If you have to go into the basement and copy a piece of paper to get security numbers, it’s not as big of a deal if people are doing what they’re supposed to be doing to properly redact documents. But lawyers are terrible at redacting PDFs properly, and so PACER records, for example, have been shown to be rife with private data that shouldn’t be in them, but it is. There it is.
David Colarusso: Just ask Manafort’s lawyers how hard it is to redact them.
David Colarusso: Yeah, and that’s another potential problem there. Of course, then you have a lot of legislatures trying to be very thoughtful about access to criminal records and doing things like banning the box on job applications or limiting who has access to criminal records, allowing people to expunge their records after a certain amount of time. All of that gets short-circuited if someone can just go in and Hoover up today the docket information from the court and put it in a database and then sell that back out to folks. It means you really can’t ever have moved past that point in your life.
Sam Glover: It’s a bit of a tangent, but are there good examples of opening up public records?
David Colarusso: This is actually interesting. I didn’t preview this bit for you, but I think I actually have some good solutions to this. This is probably going to be the topic of a paper coming up.
Sam Glover: Oh, cool.
David Colarusso: The notion is that the benefits that we want to see from public open records come in two classes. One, at the particular, individual case, which is that you have a need to find out more about a particular case, so you’re doing … You’re an investigative journalist, and you’re researching your particular company and its practices, and you want to see what’s going on. You want to see their litigation history. Or, in the aggregate, you’re looking at something that’s going on in the industry or in the court system in general, and you’re trying to look at some trend. Maybe you’re looking for disparate impact against some protected classes, and you just want to know what are the statistics of individuals from this protected class and the sentences they get, something like that.
David Colarusso: You have these statistical questions in the population level and these very particular questions. The particular questions are not that really well addressed now by most court systems. You can walk into a court clerk. You can get the file. You can get the information you need for a particular case quite easily. Every jurisdiction’s a little different about how they interpret their own rules around that, but, for the most part, they’re public records, and you can just walk in and request them.
David Colarusso: The question comes when you get to the move to the aggregate. Should I just be able to go in and say, “Show me all the records for all time?” and then let me just dive in through the data and find patterns there because that then opens up to all these prior abuses we were talking about. With that accessibility in the aggregate, that becomes the difficulty. You can blackball an entire set of folks.
David Colarusso: What I think would be a really responsible way to deal with this is to do something … I’m still trying to play with the name. I’ve been talking with some individuals over at the Chan Zuckerberg Initiative about this oddly enough, and there’s the idea of how can you share the data in a way that will make it useful for statistical inquiry, but without opening up all these can of worms of actually getting people’s personalized information.
Sam Glover: I can see why Facebook would be interested in that.
David Colarusso: Yeah, although Chan Zuckerberg, to be fair, is separate from Facebook. It just happens that the money came from the same place.
Sam Glover: They might share some interests. Let’s just be clear.
David Colarusso: They have a large criminal justice group there that’s working on helping improve the criminal justice system, and they’ve been working a lot with some of the progressive DAs that have been elected recently around the country to help them get a handle on their data situation. The thing that people often say is they say, “We want to do transparency. We want to just open everything up.” As a former public defender, that makes me a little anxious when the thing you’re talking about opening everything up is everyone’s criminal records, basically.
David Colarusso: Because you do have these protections and we do … This is codified in law in a lot of places where we want people to be able to move past. Once you’ve paid your debt to society, we want you to be able to move past that. So, a lot of people talk about, well, maybe the technical solution is just to anonymize the data somehow. That becomes very problematic because there’s a lot of research that says that you can very easily de-anonymize that data. If I know someone’s gender, their zip code, and their age, I can probably tell you who they are with just other records that I can connect with quite easily.
David Colarusso: Anonymization isn’t quite the right answer because it’s hard to do that properly, and you don’t want just to give out aggregate reports about data because then you have to identify what all the possible relevant aggregate reports are. It’s nice to do reporting and say, “These are the percentages here. These are the percentages there.” In fact, actually, there was a criminal justice reform bill here in Massachusetts, which recently mandated some of this aggregate reporting in regards to racial disparities in sentencing. But you might actually have much more particular questions you’d want to ask then just telling me the number of people who this or that happened to.
David Colarusso: So the idea is to take the data, the real data, and basically to obfuscate it in some way. That could be just if you imagine it as a giant table in Excel, every row is an entry in the database. You could just imagine for each column just randomizing the rows, then you break that connection between someone’s birthday and their zip code, et cetera. Or figuring out what the ranges are in those columns and then basically republishing the data but now in a way that it doesn’t actually contain any information.
David Colarusso: What you’ve done is basically you’ve published the Excel spreadsheet, as it were … hopefully, it’s not going to be published as an Excel spreadsheet, but you can publish the information as it were … but it actually doesn’t have any real content in relationship to real things except for you can see the range of outcomes and you can see the possible different options for a value. Now, of course, before this, you do have to scrub things like social security number, personal identifying information that’s entirely unique to an individual. Those fields would not be-
Sam Glover: Let me stop you there.
David Colarusso: … shared. Yeah.
Sam Glover: Because it sounds like you’ve got a solution to this, and it’s super interesting, but I want to bring us back to Chesterton’s Fence.
David Colarusso: Ah, yes, to Chesterton’s Fence.
Sam Glover: So, we talked about thinking about opening up court data in the sense of that is where there is a fence, and it’s not clear why there’s a fence there. But now that we’ve thinking about taking it down, what might happen? It turns out that because there was a fence there, somebody started grazing their cattle on the land, and if we removed the fence now, then they’re going to run rampant, and that’s a problem. It’s a similar issue with respect to court data.
Sam Glover: Before we started talking, you mentioned one that I hadn’t even thought about, which was lie detectors are actually a bit of a Chesterton’s Fence problem.
David Colarusso: Yeah, that’s an interesting one because most people I think, assume, to the extent that they know that lie detectors are not admissible as evidence, they assume that’s because they’re really bad at telling whether or not someone lies. You’ve seen this portrayed all the time by people doing additional work on new lie detection technology. There seems to be this notion like, “Oh, if we could just get a lie detector that worked, then that would be good enough to have it introduced in court.” At least I think that’s the implicit assumption most people have.
Sam Glover: Wait, so you’re saying they do work?
David Colarusso: No, I’m not saying they work, but I’m saying the reason they’re excluded from evidence is not because they do or don’t work, although I suppose you could make an argument that they don’t. But I think most of the arguments have been around the fact that they usurp the fact-finding mission of the jury. They basically replace the juror’s judgment and don’t allow them to do that final fact-finding.
Sam Glover: Makes sense, right? If somebody has said this person is lying or not, then the trier of fact is pointless.
David Colarusso: Yeah, exactly. Why do you need a jury? So, they usurp that powers, I think the stronger rationale for the exclusion.
Sam Glover: Very interesting.
David Colarusso: Which means that when you have this new lie detector and it says that it works, then it starts to bring up all these interesting questions like, “Okay. Well, now, do we want to admit it or wait a second?” It’s not because it was bad that we weren’t admitting it, although they are horribly bad in really actually predicting if people are lying. But let’s assume that we had a lie detector that worked. Well, now the question is, do we really want to tear down that fence? Because juries are not just about fact-finding. They are about fact-finding, but they’re about community accepting the outcome of a court case.
David Colarusso: They’re about establishing the validity and the valid nature of the execution of justice. Without involving a jury of your peers, is our court system really the thing that we think it is?
Sam Glover: Right. In a way, the jury stands for the possibility that there’s more to resolve in cases than justice, or at least the justice is more than did this person do it or not. Guilt and innocence are just one component of it, which sounds a little bit similar to another issue that you raised, which is risk assessment tools like say, finding out the probability that somebody is going to recidivate.
David Colarusso: Yeah, so this is one that has showed up, at least in the popular press, more and more. ProPublica had a piece a couple years ago that they blew up looking at some of the risk assessment tools that were being used in pre-trial detention. There the idea is, could these things appropriately predict whether or not someone would return for the next court date? We’re talking, again, pre-trial. This gets all conflated in very interesting ways in the popular press because I don’t think most people understand that the rationale behind bail is returning to court, not whether or not you’re going to commit another crime or if all sorts of [crosstalk 00:33:16].
Sam Glover: Fair enough. Yep.
David Colarusso: But to the extent that you cannot conflate those issues, if we just talk about how it’s just meant to be a tool to help determine whether or not that person’s going to return to court, it’s not clear that it’s necessarily the criteria that you want to use. You might want to be fair about that. What I mean is … and this has to do with issues of algorithmic fairness … it may be that historically speaking, there might be disparities in whether or not someone gets picked up, say, for a new offense based upon their race, not actually based upon whether or not they commit any crime. To the extent that you create a model that’s based upon historic data, that data itself is flawed and has bias, and you can be baking that bias in.
Sam Glover: That makes a ton of sense. You can’t just plug a machine in and ask it to describe things because it lacks this cultural-social awareness that is also relevant.
David Colarusso: Yeah, or another way of putting it is, it will only optimize for those things that you tell it to optimize for. If you say, “Figure out whether or not this person is likely to show up,” and it can produce a prediction as to whether or not they’ll [inaudible 00:34:18] … maybe not show up for court because the way it’s usually ended up being about recidivism. So, tell me if this person’s going to get picked up on another offense. Well, the likelihood they might be picked up on another offense is not just a question of whether or not they’re likely to commit an offense. It’s whether or not there are cops in their neighborhood.
David Colarusso: I’m always reminded of this great image from Michelle Alexander’s The New Jim Crow, where she points out that police could have decided to prosecute the drug war by breaking up coke and ecstasy parties at the lacrosse team in the suburbs.
Sam Glover: Right, or riding up and down the elevators in your favorite skyscraper in downtown.
David Colarusso: Yeah, but they never repelled off of helicopters into these suburban parties when mom and dad are away. So, a lot of this has to do with historical patterns of policing and a bunch of things that are bigger than just do we see them in this particular number because that number that you’re measuring might not actually measure the whole story.
David Colarusso: Now, it’s not to say that people are going to get it right. In fact, people have their own biases and make their own mistakes. The point is that we should be very careful. I think this goes back to the Chesterton’s Fence notion of replacing flawed, human decision-makers with differently flawed algorithmic decision-makers.
Sam Glover: Especially if we can’t really tell how they’re making their decisions.
David Colarusso: Yeah, you can go back and forth on that one because people, you can’t really … You can ask them, and they can tell you what they think.
Sam Glover: I suppose that’s true.
David Colarusso: But the point is, it really just goes back to being humble and back to this notion that you have to understand why the thing that’s there is there. Then I’m always reminded of a wonderful … if we’re on the quote wagon today … a great quote from the statistician, George Box, which says, “All models are wrong, yet some models are useful.” He was talking about statistical models, but this is generalizable. You think about a map. A map is a model of the world. It’s not perfect. It doesn’t tell you every nook and cranny, so it’s wrong, but it’s useful because it can help you get from one place to another.
David Colarusso: Whenever we’re talking about taking down a fence or using a model, the thing I always like to say is, “Well, we know that this model is not 100% right.” So it should start the conversation, not end the conversation.” So, let’s have it. Start the conversation. Then you always have to ask, compared to what? We’re talking about, is this useful? Yes, yes. Compared to what? Sometimes you’re comparing it to nothing, so it’s a really easy bar to clear. But you have to ask, are we doing harm or we doing good?
David Colarusso: A lot of times [inaudible 00:36:38] work fast and break things is really just about let’s move ahead and see where things settle. That’s fine when the stakes are low, but when the stakes are high, it just requires a little more thought.
Sam Glover: We started out by talking about this in the sense of big changes, big changes at the level of trying to solve the access to justice problem or things like that, and legal startups. It strikes me that this also has some real implications for more mundane questions like, why the hell doesn’t my law practice management software do the things that I want it to do? Everything that’s designed is designed with some intention behind it, and this is something I encounter all the time where we get impatient with our products and our software and things like that. I find it is beneficial to, before I start blowing things up, to try and figure out why was it designed to do things in this way.
Sam Glover: That might reveal something to me about how this software wants to be used, or the intention behind it, which may or may not be right. But, it can help me in figuring out how I want to design my systems and my processes around the tools that I’ve chosen to use. Consumers are going to be demanding without worrying too much about why something was designed the way it was, but when you’re talking about how to design your business or your practice or how to use your tools, I think Chesterton’s Fence has a lot of important lessons there for us for how we approach those problems as well.
David Colarusso: Yeah, contract drafting is a wonderful example of this, right?
Sam Glover: Sure.
David Colarusso: Why is that clause there? Well, that clause is there because something happened at some point in the history of that drafter who thought that was reasonable to be there. Is that still something we should be thinking about? Yeah, you have to understand the state of play before you move forward and change things, or at least if you want to be responsible about it.
Sam Glover: Yeah, that’s a really good point. Because it’s always been done this way is a super uncompelling argument, but you can assume from the fact that it’s always been done this way that there was a reason and it probably wasn’t just because I like that language. Although, it could have been. Yeah, before you decide to simplify a 19-page contract into one page, which is always my instinct, it’s pretty important to understand why it might have been 19 pages in the first place. Yeah, makes sense. Again, not an excuse to not do that work.
David Colarusso: No, not an excuse at all. In fact, it makes everyone have to do the work. It’s not like [inaudible 00:38:53] was saying this is the way we did it. I think it’s probably incumbent upon the person who’s saying that to justify why it is that they did that. Yeah, you have to put the work in. There’s just no way around that.
Sam Glover: Here’s my question for you because I see this a lot. I think you and I are the type of people who see a problem, want to understand it, and then want to improve on it. A lot of people either don’t see problems or see a problem, begin to understand that it’s complicated, and give up, or just figure that there was a good reason in the first place and don’t act on it. How do you think someone can work on changing their mindset from being someone who just accepts the status quo to somebody who looks at things and fixes them? Or, is that just something innate in people like you and me?
David Colarusso: Oh, that’s a big one. Okay. I’m going to have to think for more than two seconds.
Sam Glover: It’s okay if you don’t have an answer, but it’s … I watch people struggle with this. They click the same button 15 times rather than just figure out a way not to click that button 15 times and just to click it once. I see people struggle with these little things over and over and over again, that range from just everyday little things to …
Sam Glover: I always hate to portray My wife in any way negative on the podcast, but she was asking for some help organizing files the other night. She has a folder, and it only has one folder in it that has everything else in it so that there’s one extra step she has to go through every single time and there’s no reason for that folder to be there. And I’m like, “Why do you waste these clicks every time when you could just move everything out of this folder into the parent folder?” It never occurred to her to do that. It’s just a small example of people who don’t fix problems.
David Colarusso: My suspicion is that it might, in part, be about the permission structure to think about it. Just because my working theory now for a lot of when I see people not making decisions that I would make, one, I have to ask, “Well, maybe they have a better reason for doing it than me.” Sometimes I think people are just … They’re operating from a place of scarcity, right? Their time is scarce. Their ability to think about things is scarce. It’s much easier to just, as you say, go with the flow.
David Colarusso: There’s often value in that because there is presumably the accumulated wisdom of those who came before. I think in order to question things, you have to be in a space where you feel free enough to exert some of that effort because I think it is effortful. I find a lot of joy in following my curiosity, but that doesn’t happen on nights when I’m tired or days when I’m tired-
Sam Glover: Yeah, good point.
David Colarusso: … or right at the end of the semester when I’m doing 100 different things. Although, sometimes I get distracted by things because I’m procrastinating and I want to do other stuff. Maybe the answer is to somehow relieve some of that cognitive load so that you have the ability to ask those questions. I know you guys, at Lawyerist, you’re always talking about how you go, and you do your reviews of your policies and stuff, and have very regular, built-in times where you’re reassessing the way you’re doing things. I think just making the space for that is probably the first step.
David Colarusso: That’s based upon the fact I was ambushed with this question.
Sam Glover: Yeah, I’m not asking you completely random though, because one of the things I have noticed about people with scientific backgrounds … Maybe this comes from the scientific training where nothing is true unless it can be replicated, basically. I have noticed about people with scientific backgrounds that the moment there’s a big news report that comes out about whatever in the universe or discoveries or whatever, the people that I know who are scientific will often sit down with their calculators and try and replicate those calculations, or go and dig into Wolfram|Alpha for the kinds of data that they need to see if they can understand why it was calculated in that way.
Sam Glover: I love that. It seems to be that kind of curiosity about, “Huh, can I understand that?” that I think we could all benefit from doing more of. I’m not totally asking you a random because you have that background, and I’ve noticed you doing those things.
David Colarusso: There, I think, a lot of that probably it is the tendency to question received wisdom and to say prove it. I can remember the motto of the Royal Institute, one of the science institutes that Newton belonged to. The Latin translation is something like, “Truth not by man, but by argument,” or something. That’s a really bad translation. I’m now going to have to go find that, and I’m tempted to just Google it here for the listeners.
David Colarusso: That’s really central to the idea of scientific claims are justified based upon their arguments, not upon who’s making the argument. That’s very common, something you find in law too. It’s funny you bring this up, though, because I was thinking about this today before our call. I was thinking about how both physics and the law are full of people who are sure that they understand everything and can figure out any problem, and that that actually is the antithesis of people who are using Chesterton’s Fence. It’s the idea of like, “Oh, I’m a smart person. I can figure this out.” That often gets us in a lot of trouble.
Sam Glover: Unless you’ve actually put in the effort to figure it out.
David Colarusso: Yeah, unless you’ve actually put in the effort. I guess it’s this two sides of the same coin there. There’s a competence to try to figure it out, and then the question is whether or not you then actually do the work that you really need to do to figure it out. Both of those, I think, come from a place of privilege, right? The notion that yeah, I can take the time to figure out whether or not this is right because I don’t have to be doing x, y, and z. It’s not going to cause a problem for me. No one’s going to question my ability either.
David Colarusso: Certainly, one thing privilege does give you is space to think about these things. So, maybe that does come back to how do you get people-
Sam Glover: Yeah, your first impulse was ownership, but I think it’s also competence, the idea that I feel competent to address this problem, and I’m someone who can take responsibility for it. Those seem like they go hand in hand. I don’t necessarily see a big math problem and think that I am competent to replicate that solution. I don’t know that I can solve it. Maybe that’s similar when somebody looks at a technology problem and thinks about it that way. If they don’t think of themselves as a tech-savvy person, then they may not think that this is a problem that I have ownership over or that I’m competent to solve.
Sam Glover: I guess what we’re saying is, newsflash, maybe you are.
David Colarusso: Yeah. We deal with experts all the time, and we help them explain things to us so we can explain them to a jury. You have the tools you need to figure out problems, but the truth is that yeah, you’re not going to know everything from … Well, this is interesting because this is something that comes up with my students all the time when we teach them a little programming. I think most people think that computer programmers have all of a computer language memorized like someone might memorize French. But the amount of times that you have to go back and look at your old code and cut and paste, or you go to Stack Overflow and see how someone does something … Yet, it’s a crutch to rely on that too much.
David Colarusso: The truth is, you don’t memorize it in the same way. You always have to …
Sam Glover: Wait, so that’s normal?
David Colarusso: What you do is you learn to code. Yeah, that’s normal.
Sam Glover: God, I thought I was just an idiot.
David Colarusso: That’s the realization that a lot of people have when they’re doing that. You mean, “Wait. Now, this is how I do this?” There are their programmers who’ve been working in the same language for decades, and they always have to look up this syntax for a particular thing. That’s because they-
Sam Glover: That’s a relief to hear that.
David Colarusso: Yeah, well, that’s because they’ve made an important distinction between the memorization of something and the ownership of that knowledge. They know how to find the answer, and they know how to find that answer in a way that’s going to get them the right answer that doesn’t have to hit upon their cognitive load.
Sam Glover: I guess that describes law too, right?
David Colarusso: Yeah, exactly. You know where to find information. That’s transferable to other domains, and when you recognize that, then you can be curious about things because it’s not so overwhelming. You don’t have to understand everything. It’s dangerous here to think that you’re going to … This is the tension, right? You have to understand the rationale, but the point is it’s freeing to know that you can ask someone, or do research. I don’t want people to go off and try to disprove the moon landings because they can find stuff on Google. That’s the danger, right? That’s the second side.
David Colarusso: I did happen to Google, the Royal Society’s motto, and …
Sam Glover: Excellent.
David Colarusso: It roughly translates to, “Take nobody’s word for it.”
Sam Glover: Gotcha. I like that.
Sam Glover: Well, I think that’s a nice place to end. David, thank you so much. If I can wrap us up for listeners, it is, be humble when you approach problems, and try to understand them before you try to fix them. “Don’t take anybody’s word for anything,” says Sir Isaac Newton. Thank you so much for being with us today.
David Colarusso: Thank you.
Aaron Street: Make sure to catch next week’s episode of The Lawyerist podcast by subscribing to the show in your favorite podcast app, and please leave a rating to help other people find our show. You can find the notes for today’s episode on lawyerist.com/podcast.
Sam Glover: The Lawyerist podcast is produced with help from Lindsey Calhoon and edited by Paul Fischer. The views expressed by the participants are their own and are not endorsed by Legal Talk Network. Nothing said in this podcast is legal advice for you.