The Legal Tech Audit Proves Lawyers Are Terrible at Technology

At last year’s LegalTech West Coast Conference, D. Casey Flaherty, Kia Motors America’s in-house counsel, made the provocative assertion that many lawyers are technologically incompetent, and this incompetence leads to wasted time and money. In an attempt to address this, Flaherty developed a legal tech audit (LTA) designed to test basic competencies in working with PDFs, Word documents, and Excel spreadsheets. Flaherty first administered the audit to nine outside counsel firms. According to Flaherty, all of them failed spectacularly.

After using an early version of his legal tech audit in house, Flaherty teamed up with Suffolk University’s Institute on Law Practice Technology and Innovation to make the LTA available for lawyers and law students.

Recently, both Sam Glover and I1 had a chance to take the audit, and here’s what we found.

Lawyers Really Are Terrible at Basic Office Productivity Technology

First, there is no question that Flaherty is right about the fact that many, if not most, lawyers are shockingly bad at the things really are very basic skills, like making changes in Word documents, de-duplicating Excel spreadsheets, and redacting PDF files. Indeed, Flaherty’s own administration of the audit to outside firms makes that clear. Make no mistake — these are skills that should be expected of every attorney. Being able to do things like editing your own documents in order to make them presentable and preparing documents for e-filing are legal work.

That said, the Legal Tech Audit should be considered an office productivity software skills audit, rather than a legal technology audit. This audit won’t test you on whether you know how to share client files securely via cloud storage or FTP, or whether you know how to properly encrypt client files. This is about Microsoft Word, Excel, and Adobe Acrobat, basically.

The LTA website doesn’t give away all the secrets of what is tested, and I won’t do that here either, so I’ll only be talking about a few of the tasks. In Word, you will do things like move text around, delete comments, renumber contract sections, and remove identifying metadata. In Excel, you will be expected to do things like de-duplicate a spreadsheet and perform basic calculations. Finally, in Adobe Acrobat, prepare yourself for redacting documents, combining pages, and preparing for e-filing.

The Audit is Not Ready for Prime Time

Unfortunately, while the idea of a legal tech audit is great and this one does test critical skills, the software for the testing is less than ideal. Flaherty definitely designed this with BigLaw in mind, assuming that the attorneys that take the test will be in a firm that has its own training department, learning management system, and IT staff that will facilitate taking the LTA. Perhaps that setting may change some of the problems both Sam and I had getting into the audit and working through it, but that setting will not fix all the problems with the user experience.

The audit runs only on Windows, and only in IE (or possibly Chrome or Firefox with an IE plugin). You’ll need Microsoft Word and Excel. In theory, you could do this in or something similar, but you’ll likely run into trouble because the audit expects you to do certain things in an exact way. You will also need Adobe Acrobat Professional or a similar PDF program that allows for redaction. (Not all do).

You’ll also need full administrative access to your computer and and network so that you can make the training page a trusted site in IE and let it do things like run executable files on your machine. The audit also records every keystroke you make during the test.

The invitations Sam and I received didn’t specify the system requirements (but invites to others would have) so as a result I took it in less than optimum conditions. I only have Mac and Linux machines at home, so I ran the entire test in Microsoft XP running in Parallels on Mac, which resulted, regrettably, in most of my keystrokes not being tracked. (This is probably due to Parallels.) I used Microsoft Office 2011 for Mac, but I did not have a PDF program installed that does redaction.

Full disclosure: I have no doubt that I did horribly anyway. I don’t need to see my scores to do that. The Excel portion was a killer, and I’m not ashamed to admit I did terrible at it.

Sam took the audit on a Windows 7 machine with Microsoft Office 2010 and Adobe Acrobat X Standard (which does not do redaction, either).

Assuming you have a compatible system (and obviously you will if you take the test in what Flaherty considers an ideal testing environment), you will find that the interface itself is odd. The buttons are detached from the instructions and the program bogged down and stopped running at one point. And there is no back button if you make a mistake and want to go back through previous instructions. This becomes a big issue when the instructions are vague, which they are in several places. For example, the Excel instructions do not specify which of two worksheets you should be modifying. And in the PDF section, you have to attach an exhibit that doesn’t exist in your sample documents. It may or may not be a PDF page you extracted earlier, but the instructions are not helpful.

At the end of the audit, you will get what is essentially a raw score. Your ultimate “grade” on the audit will be expressed as a function of the time it takes you to finish it, including time penalties for things you did wrong. This should arrive within a week, and the report will show you exactly where you went wrong. If you perform a task wrong or skip it altogether, you will be assessed a time penalty that is pegged to how long it would take the slowest person to complete the task.

Our concerns with the audit’s missteps may appear petty, but what those concerns generally mean is that the bad (and outdated — IE, really?) technology driving the audit and the clunkiness the user experiences end up overshadowing the usefulness of the audit itself.

The Audit Could Have a Bright Future — For Some Types of Attorneys

Because we think the Legal Tech Audit is not ready for market (even though it is already available to attorneys who would like to learn how little they know about field codes in Word), that does not mean we think the audit should be written off. First, Flaherty and Suffolk are very responsive to suggestions about necessary changes, and have already modified one part of the Excel portion of the test after Sam ran into errors.

In the right situation — BigLaw, lots of training resources, a decent amount of money and time to spend on the audit — the LTA could become a meaningful way for general counsel to benchmark the efficiency of outside counsel. Law firms will be charged $250 per user for a one-year subscription, and can pay $150 per user for an additional tutorial. Additionally, Suffolk and Flaherty have partnered with training companies that can work with firms to increase proficiency on these skills. The hoped-for (and likely) outcome is that corporations who are shopping around for outside counsel will be able to request that the firm members take the test and provide the scores so that the company can choose a cost-conscious and efficient provider.

This scenario sounds great, but it also highlights why the audit is not for everyone. At the law school I teach at, more than half the students each year have Macs, and will probably stay with that platform if they are able to (even if their firms are officially using Windows). Solos might not have $400 to spare on learning material and are probably more likely to be running what the audit considers non-standard software, such as Google Docs or Preview. Small firms simply do not have the training department or learning management platform that the audit envisions. But for those people who work at firms that fall into the category of firms that Flaherty and Suffolk are targeting, this audit may very well be something you find yourself taking quite soon. Brush up on those Excel skills.


  • 2014-09-09. Originally published. The license costs $250 per user for a one-year license, not $250 a year for a firm-wide license. Thanks for pointing out our mistake, MG.
  • 2014-09-12. Article revised with information we learned after publication.

  1. Both Sam and I would be considered tech proficient under any rubric. 


  1. Avatar Paul Spitz says:

    What a nightmare! Of course, they lost me when you said I would have to pay $250 for my firm (a solo practice, so $250 just for me), while a multi-billion dollar corporation would only have to pay $25/lawyer.

  2. Avatar MG says:

    Article is misleading – reads $250 for a one-year firm-wide subscription, but clicking through to the link reveals that the price is actually $250 per user for a one year subscription for firms.

  3. Avatar Jeff Taylor says:

    So basically, I have to use technology I don’t care for to actually tell me that I suck at using the technology I don’t care for? Pass.

  4. Avatar Sam Glover says:

    Casey Flaherty gave me access to the audit, as well, but I haven’t tried to take it, yet. It sounds like taking the audit will be nearly as hard as scoring well, which is unfortunate. I hope they will work out the kinks over time, because lawyers really need to be shown how terrible they are at tech.

    My understanding is that there will also be a companion course that will teach the skills tested. I assume that is why the audit is sold as a subscription, not as a one-time thing. That way, you can take the test, fail, learn to do better, and try again.

    2014-09-10 Update: I’ve taken the audit, and you can read my reaction in today’s email newsletter.

  5. Avatar topsully says:

    Not having access to the audit I have to accept at face value what the author said about the need to have unrestricted access to the computer, the ability to install a keylogger and the requirement to have Acrobat installed. Three things which are a no-sell in our firm. (With all of the better, less costly programs for manipulating pdf files that are available to law firms the need to use Acrobat is puzzling. Hopefully the audit will allow you to use the tool of your choice for pdf manipulation.)

    I wonder how long it will take before someone with malicious intent creates a look alike ” LTA audit” as cover for their own keystroke recorder to breech the security at a law firm?

    I wonder how long before BigLaw Firm has F-500 Client insists on this audit and then because of the keylogging technology installed BigLaw Firm fails the SOC II required by their biggest FinancialFirm client?

    I wonder how long before some of these people wake up and realize that there’s more to being a lawyer than knowing how to properly number paragraphs? Being a lawyer is more than manipulating the electronic documents. There’s much more at stake with your legal problems than whether or not your lawyer is manually numbering paragraphs. Of course SOMEONE has to know how to create and manipulate the documents, and many times the right person is a lawyer. But if there are clients out there seriously considering allowing law firms to handle their business solely based on a technology audit then they probably aren’t going to be getting the best legal advice and service for their dollar.

    Yes, lawyers need technology skills, but not every lawyer needs the same technology skills. Not every client needs to use a one size fits all approach to testing lawyer technology skills.

    Clients should be able to measure whether they’re receiving the best value for their money and if their work is being done in a timely fashion. But the goals of this audit seem way too shallow to me and won’t fully answer that question.

  6. Avatar moedogs says:

    It seems to me that training to the test is the last things firms want or need. I think what Flaherty is trying to emphasize is that these are the skills attorneys (all professionals really) should be using in their every day practice. Companies like Payne Consulting, BEC Legal and Microsystems make MILLIONS of dollars selling programs (programs based on the native Word capabilities) to help lawyers organize their thoughts on paper. Apparently the lawyers working at all these firms buying those products don’t use the products themselves. I learned to organize research papers using flashcards when I was in the 7th grade. Now I can do the same this using Outline view in Word, where my Styles not only build my TOC but they also automatically bookmark my PDF when I publish it. And no, I don’t have Acrobat, I just publish to PDF directly from Word and Excel. Why would anyone NOT want to be more efficient on a daily basis, especially with more and more clients requesting AFA? Your firms spend hundreds of thousands of dollars on technology, you are doing yourselves and your clients a disservice by not embracing the tools to their full potential.

  7. Avatar qning says:

    Did you dog this guy’s beta? If so, that’s a bit of a party-foul. On the other hand, if dude has put this thing out there and is ex-post crying “beta!” well, that’s on him. Plus, he might know better than to invite you to try it out. What with you being brutally honest and all.

    Flaherty does it in 30 minutes, and the mean is 5 hours – I’d like to see the raw scores to see what the distribution is. And what is “Actual (Mean) Completion…” meant to communicate?

    • Avatar Sam Glover says:

      Nothing in the email giving us access to the audit indicated that we were taking a beta, and the word beta does not appear anywhere on other than in a white paper referring to a two-week trial by students at Colorado Law School back in June. Plus, you can absolutely register and take the audit now, if you’ve got $250 to spare.

      However, when he followed up with us after this post went live, Flaherty said:

      You took the beta version of the LTA. The content is not beta. But the delivery mechanism is beta.

      So did we dog the beta? I guess so.

      • Avatar Paul Spitz says:

        Isn’t the point of beta to identify the faults?

      • Avatar Alex says:

        That’s fascinating that his defense is you used the beta version. Anything on the Internet with a retail price and no beta disclaimer, is not beta, in my opinion. I’m guessing that he read about lean start-up principles and undershot his minimum viable product. The fact that he was planning to iterate his product post-launch doesn’t mean he gets to call it beta now.

        Neat concept, though. I wonder if his secondary market are law school writing classes and paralegal programs.

  8. Avatar Ann Fisher says:

    This will sound cranky. It isn’t any more than the statements and implications made that sufficient numbers of lawyers are so stupid that they need this audit. so…

    To the gentlemen (Flaherty?) who was originally irritated at the amount of money that was squandered by what you considered uneducated technophiles – you are pretty disingenuous to complain when your beta audit still doesn’t work and requires things (like open internet access for governmental agencies) that are in appropriate. It just show that the only entity that doesn’t understand technology is YOU. I am offended that you would waste your company’s time and the time of lawyers unfortunate enough to do work for you just so that you can show them up. Your pettiness and lack of understanding of technology yourself gets in the way of making things better. You impute stupidity and evil doings without any actual understanding what outside counsel do for your company and the technology they employ. Shame on you.

    Do lawyers need the ability to manage the software they use? Yes of course. Are you the final arbiter of what’s good and what’s not – no – and maybe you should get out of that field and do something you actually know something about.

  9. Avatar celia says:

    $250 to tell me whether I suck at technology? I can’t figure out for myself whether I’m struggling with my software and need to brush up on my skills myself? Sorry, I pass.

    • Avatar Sam Glover says:

      The point isn’t to show you that you suck at technology. The point is to show your clients that you suck at technology. If a client is making decisions about who to hire based in part on your score, you’re going to put a higher priority on brushing up on those skills (I don’t know if you actually suck, but most lawyers do, and aren’t doing anything to change that).

      Overall, I think the legal tech audit is a good thing. The audit itself needs to be improved before I would advise anyone to rely on it, of course. But when it is polished up, I think clients will and should use it to help assess their lawyers.

Leave a Reply