Lawyers are in denial to the point of delusion with respect to the reliability of keyword search and human review. U.S. District Court Magistrate Judge John Facciola put it best when he quipped that lawyers think they’re experts at keyword search because they once found a Chinese restaurant on Google.
We trust keyword search because we understand it. We trust manual review of documents because we grossly overestimate reviewers’ abilities to make sound, consistent decisions about relevance. “To err is human,” the bar seems to say, “but forgive us if we’d rather not divine just how error-prone reviewers really are.”
Better approaches to search are arriving as so-called “predictive coding” or “technology assisted review” products. Still, it will be years before the rank and file embraces TAR, if only because those hawking TAR tools remain resolutely uninterested in positioning the technology for use by anyone but big corporations and “white shoe” law firms. Worse, the fervor among vendors to sell something, anything that they can label predictive coding ensures that tools little different from ordinary keyword search will be given a dab of lipstick and pushed out to market as TAR tools. It’s messy down in the TAR pit.
Even those adopting predictive coding tools will need to compile “seed sets” of relevant documents to train their tools. So clunky-but-comfy keyword search and manual review are likely to remain the means to cull seed sets from samples. Despite serious shortcomings, keyword search and manual review will be with us for a while.
Keyword search is the art of finding documents containing words and phrases that signal relevance followed by page-by-page (linear) review of those documents. It’s often called the “gold standard” of e-discovery.
That’s ironic, because extracting and refining gold relies less on finding precious aurum than it does on dispersing all that isn’t golden. Prospectors use water and chemicals to flush away all but the gold left behind. So a true “gold standard” for keyword search would incorporate both precise inclusion (smart queries) and defensible exclusion (smart filters).
To illustrate, in one e-discovery dispute over search, the plaintiff submitted keywords to be run against the defendant’s email archive for a three-month interval. Unfortunately, the archive held all email for all custodians, and the defendant adamantly refused to segregate by key custodian or deduplicate before running searches. The interval was narrow, but the collection was vast and redundant.
The defendant tested the agreed-upon keywords but shared only aggregate hit rates for each. Thinking the numbers too high, but unwilling to look at the hits in context, the defendant rejected the search terms. The plaintiff agreed the hit counts were daunting but asked to see examples of hits on irrelevant documents before furnishing exclusionary (AND NOT) modifications to flush away more of what wasn’t golden.
The defendant refused, insisting it wasn’t necessary to see the noise hits in context to generate more precise queries. The parties were at an impasse, with one side grousing “too many hits” and demanding different search terms and the other side uncertain how to exclude irrelevant documents without knowing what caused the noisy results.
A lawyer who dismisses a search because it yields “too many hits” is as astute as the Emperor Joseph dismissing Mozart’s Il Seraglio as an opera with “too many notes.” Mozart replied, “There are just as many notes as there should be.” Indeed, if data is properly processed to be susceptible to text search and the search tool performs appropriately, a keyword search generates just as many hits as there should be. Of course, few lawyers craft queries with the precision Mozart brought to music; so when the terms used seem well chosen for relevance, it’s crucial to scrutinize the results to learn what tailings are cropping up with the gilt-edged, relevant documents.
Keyword search is just a crude screen: “Show me items that contain these words, and don’t show me items that contain those.” High hit counts don’t always signal a bad screen. If search terms merely divide the collection into one pile holding relevant documents and one without, you’re closer to striking gold. Then, you look at what you can reliably exclude with the next screen, and the next, drawing ever closer to that elusive quarry, documentum relevantus.
But you must see hits in context to refine queries by exclusion. That seems so manifestly obvious, it’s astounding how often it’s not done. When lawyers delegate keyword search, they often get back only aggregate hit counts and mistakenly conclude that’s enough information to judge searches noisy or not. If, instead, counsel get their hands dirty with the data, as by personally exploring representative samples using desktop or hosted tools, the parties could work quickly, effectively and cooperatively to zero in on relevant material. Good queries are best refined by knowledgeable people testing them against pertinent, small collections. Lousy outcomes spring from lawyers thinking up magic words and running them against everything.
It’s not just a theory. Recently, as part of an early case assessment effort, I sought to rapidly isolate relevant documents from a half million email items culled from four key custodians. That’s a volume where you’d expect to see bids from service providers and mustering of review teams. It’s a project most firms would see as much more than a weekend’s work for one lawyer. We tried something different. To start, the client exported the four key custodians’ email messages for the time period of interest from its email archives. Those 50 gigabytes of messaging went into a desktop processing and review tool. Extracting and indexing the data overnight, I flagged exception items (e.g., images without extractable text and encrypted files) for further processing, then exported spreadsheets reflecting the most used email addresses. I asked the custodians to flag addresses with no connection to the dispute.
Meanwhile, I compiled the customary list of search terms and phrases expected to occur in relevant documents and tested these. Documents with false hits were examined for characteristics permitting mechanical exclusion. Testing, re-testing and re-examination soon produced reliable inclusion and exclusion term lists. Weeks of evaluation took just days because the iterations and results were instantaneous.
The discards were tested, too. For example, material excluded by addresses but containing inclusion terms was carefully checked to ensure the hits weren’t relevant. Defensible exclusion proved as powerful as inclusion, and potentially relevant material that couldn’t be excluded as tailings stayed in the collection as ore. A true “gold standard.”
Did it produce a perfectly parsed set of material? Certainly not. Keyword search and human review still fall short of expectations. But it was fast, relatively cheap, and afforded cautious confidence that the set produced was more relevant and less riddled with junk than what would have emerged from the usual game of blind man’s buff. It was fast and cheap because the person creating and testing the inclusive and exclusive filters was elbows-deep in the data and hands-on with the search tool. Feedback was immediate. Quality checks could be done at once.
Ideally, EDD tools don’t put distance between the lawyer and the evidence but, instead, extend our reach and help us get our arms around big data. A lawyer who is hands-on with the evidence and who tests and refines his or her choices is a lawyer who can explain and defend those choices. That’s the real golden future of e-discovery. Welcome back, counselor.
——————————————————————————–
TAKEAWAYS
■A lawyer who dismisses a search because it yields “too many hits” is as astute as the Emperor Joseph who dismissed Mozart’s Il Seraglio as an opera with “too many notes.”
■If counsel get their hands dirty with the data, as by personally exploring representative samples using desktop or hosted tools, the parties could work quickly, effectively and cooperatively to zero in on relevant material.
Austin’s Craig Ball is a trial lawyer and computer forensics/e-discovery special master. Email: craig@ball.net.
