Jeffrey Lewis: Welcome everyone, I am Jeff Lewis.

Tim Kowal: And I'm Tim Kowal. Today our guest is John Stobart, senior counsel at Capstone Law in Los Angeles and a certified appellate specialist. Also, I'm jealous, he's a member of the California Academy of Appellate Lawyers. His practice is wage and hour and PAGA appeals, but that's not the reason that we invited John today. Most of the conversation about lawyers and AI falls into one of two camps: the horror stories about fake citations or the vendor pitches promising that software will write your brief. And John is doing something less common. He's an appellate specialist who uses Claude in his own practice. And he's been showing other lawyers how to use Claude more effectively on TikTok of all places. That's where all the appellate specialists are gathering these days, I hear. John's going to talk to us about what responsible AI use looks like. John, welcome to the podcast.

John Stobart: Thank you, Tim. Thanks for having me and thanks, Jeff.

Tim Kowal: Now, the timing on this one turned out to be good because just about a week ago, on September 30th, Governor Newsom signed SB 574, the first state statute in the country to regulate how lawyers use generative AI. Jeff and I talked about this one a little bit, and we were both a little bit pessimistic, took a little bit of a dim view of the statute that maybe it discourages creative but appropriate use. Maybe too many attorneys are taking creative liberties with AI and are not disciplined about it. So we want to talk with you about some of your tips on how to make creative but appropriate use to give value to clients and value to the legal system as a whole. So that's we're going to talk about today, hopefully. Where does using AI end and delegating to it begin? So I don't know. Jeff, you asked to take the laboring oar on this one. Usually, I stumble around and ask the questions. Well, I'm going to hand this one over to you.

Jeffrey Lewis: All right. All right. I'll start. Yeah. And here's the big one, John. You know, without insulting anybody else on this podcast today, most appellate lawyers write stiff scholarly pieces, and they publish them on LinkedIn or Substack. You do short, useful, helpful videos on TikTok. So, my question is, why TikTok and who's your audience?

John Stobart: Yeah. So well, TikTok wasn't the intended method of getting this information out. I saw that, you know, I was really excited about AI and what I was doing for my practice. I wanted to dig deeper into it, and I wanted to tell other people about it. And I actually started on Instagram, and I thought carousels would be a good way because you could just kind of I could put forth information, do this, prompt that, check this, you know. And I mean, I would post it and I'd get like five views, you know, and like two of them were family members. And then I started doing a couple of videos, and my wife kept saying, Post to TikTok, post to TikTok. And I was like, why would I do that? I didn't even have a TikTok account. So, I opened up a TikTok account. I've been through a couple of handles, Tim, as you mentioned earlier. I landed on @AI4Lawyers and started posting videos and they would just take off. And I've got videos that I recorded months ago that are still getting views, like daily views today. So, there is an audience out there. What I mostly see is they're mostly like solo practitioners. If I look at it there, it'll be like, you know, law dog or something, you know, something like that. And its mostly people out there trying to use TikTok to like to push their profession. And that's what I found. That was the audience I found. That's the community I found on TikTok.

Jeffrey Lewis: Interesting. Yeah.

Tim Kowal: That's an interesting point. Other people groping around like you trying to get business and they happen to find an attorney. You know that's interesting, because I wondered how many people who are looking for an attorney are going to TikTok, but you mentioned that that your prospective clients might already be there pitching their own wares and then they happen to learn about an appellate service.

John Stobart: Right. And just to be perfectly clear, I don't offer anything for sale. I don't do any kind of consulting. This is all just an educational project for me. If I did want there are some people who do AI consulting that use this as a platform and that's what their intended audience is. I'm just I'm just out there telling trying to tell people, you know, better ways to use this and ethical ways to use it. And aside from who's on there, like the thing that I found really interesting is how little number of people talking about AI there are on TikTok. There’re actually only two other accounts that genuinely do what I do. And they're both excellent. And if you go to my account and you send me a DM if you're a TikTok person, I will get you both of their accounts. One's meta council and the other one's Carolyn. And they do, they speak all around the country and they're kind of like I kind of follow their guidance and get information from them. But they still do it from a different perspective. My perspective is just like one-minute-long videos, here's how to do this certain thing, you know?

Jeffrey Lewis: Yeah, I noticed the majority of videos out there and articles are about horror stories of lawyers getting sanctioned for dumb things, or just warnings to people about don't do this with AI. What I find refreshing about your TikTok feed is that most of the videos are helpful for, you know, things you can do, things appropriate uses, short informative clips. And I want to know. Was it your intention to stay away from the sanctioned horror story misuse category of docs?

John Stobart: Well, you know what? If I was trying to get views, then I would have taken that path because as I noticed from those other accounts, when they talk about that, you they get lots of views and lots of engagement. But you know, I'd rather talk about, you know, how to avoid that, you know, those kinds of things and a productive use of AI. I think there's enough out there and a lot of misconceptions on, you know, these horror stories, and they do exist. I mean, we're talking thousands of attorneys who have been sanctioned across the country for the misuse of AI. And you know, I could report on that, or I could, you know, try to push an ethical and responsible use of AI, and I chose that.

Jeffrey Lewis: You know, for those lawyers who are listening to the podcast who are reluctant to adopt new technology, you know, the guys who didn't want to put their stuff in Dropbox when cloud computing came out and they never wanted to file documents electronically. They insist on serving documents by mail. For those lawyers who are listening to the podcast, why should those kinds of lawyers take a chance and try AI.

John Stobart: Well, I mean, the reason why: they will be amazed at what and how it can help. And the advice I would give is you just start small. Okay. It would be it's it would be irresponsible to say, go get Claude and have it written a brief because you're not there yet. I see it as a kind of driving the LLM. You're not there yet. You don't know how to use the tool yet, right? And the tool doesn't know how to react to you yet. So that is an irresponsible use of AI. And that's actually what you see a lot. Happening in these horror story cases. Somebody gets AI, they have them do something that they shouldn't have it do. They trust it and they file it and they get sanctioned. I would say start small. A great place to start is you go get a paid account, $20 account, either ChatGPT or Claude. I use Claude. And you could hook it up to your Microsoft Word and have it edit a document for you. Just do a check doc and it'll go through and sort of like a like a proofreader. And it'll, but when it does that, it does so much more and it might tell you some citations that are wrong. If you're numbering paragraphs and you go first, second, second, third, it'll say, Hey, do you really mean to have two second paragraphs? Because it would make more sense if you went one, two, three, four. And it'll pick up on all those little things. And then you then you know, after you see what it's capable of, then you start building on that. And so, I would suggest like if somebody wanted to get into AI and start learning how to use it, you start very small with either something like that or summarizing. Documents are a great way. That's how I got started. Yeah, I had I had a trial coming up. I had a 500-page PDF and I had to separate it up into, I don't know, 70 or so documents. And I'm looking at it and I didn't have time to get a paralegal to do that for me. It was too short notice. And our firm has just started using Claude. It was the first assignment I ever gave Claude and I said, Can you index this? And it shot back an index. And I said, Can you separate this into it? I came back with 70 documents. Can you separate this into 70 PDFs? I opened the folder; I had 70 PDFs named with file numbers and all that. And I was like, okay. And it said, great, John. Good luck with your trial. Would you like anything else? So, it read from the documents that had a trial coming up. And I said, as a matter of fact, you know, and that that's how it got started. So, you know, start small and then, you know, if you like what it's doing, ask it to do some more.

Tim Kowal: You're talking about using Claude directly, and that's the that's the same platform that's available to everybody. And there are a lot of legal tech companies that are trying to create their own legal AI services. So, there's Harvey, and now Clio has their own Clio Work or Vincent, I think, is their AI platform. And they're going to advertise that to lawyers who are skittish about using AI, and they'll be able to have an easy sell because they'll say, don't worry, we made it safe for your lawyers. We put up legal guardrails so there won't be any hallucinations. I think maybe some of these cases with attorneys getting sanctioned for you know AI generated hallucinations is probably the best marketing tool that some of these legal AI companies ever had. But what do you tell attorneys who are cautious about? What do you mean by just using Claude? That's available to the masses, to the hoi polloi. Don't I need a special legal AI service?

John Stobart: Yeah. So first of all, Tim, there's no more than I've read at least two cases where people are using Harvey and got sanctioned. And that's on, yeah, if you go on my website, I brief all the major sanction cases and all the major discovery. There's two kinds of camps of cases right now. There you've got the sanction cases, which everybody's interested in, and also the discovery of prompts. Discovery of your clients' prompts, discovery of work product, which is obviously going to be protected, but it's still out there, and then also discovery of experts. But aside from that, so Harvey will not save you. A lot of those use the same ChatGPT Claude LLMs and they're just a wrapper. So, I would say.

Tim Kowal: A wrapper meaning is still using the same underlying model. It just they've included their own prompts basically around it and told it what context to use and what not to use.

John Stobart: Yeah, and you know, its probably softer landing, but you pay for it. But there's nothing wrong with it. If you are the same it's the same strategy, you know, you'd start small, see what it can do, and then build, you know, like a rapport with the with the tool, with the LLM. But you could do all of that with Claude. So, Claude is the is the one that my firm uses, so that's what I use. That's the only thing that our AI policy allows us to use. I use ChatGPT a lot too. I use it for my website for my TikTok videos. I use it to get ready for this podcast. It

Tim Kowal: You use ChatGPT in instead of Claude for certain things?

John Stobart: Yeah, so anything non-work related, because my Claude account is tied to my firm and I only use it for work. That's it. And so, everything else that I use AI for, I use ChatGPT and Gemini. I wouldn't use Gemini for legal work. It's not there yet, but ChatGPT and Claude, $20 plans are all you would need to get started for sure.

Tim Kowal: But so, the so with Claude, you feel that it's safe enough to use for legal services. Are there any safeguards that you that that you tell attorneys to use out of the box? You know, you turn off the training, you know, don't use my tr my training data for instance. Anything else to make sure that you're compliant? I think you have to be on a certain paid plan to maintain certain confidentiality. If you're on one of the free plans, certainly it's a no go.

John Stobart: Yeah, for sure. The twenty-dollar plan, turn off the training is a good way to start. I still don't, I mean, they say they don't put confidential information. I don't know, I've never put any confidential information in the AI. I mean, we're talking pure client communications. I've had that with just me and my client. That's not going in there. I don't understand why it ever would. Now, you know, documents and stuff like that that are non-confidential, sure. And anything filed with the court; you have no confidentiality problems there. Arbitration is a little bit stickier because those are confidential proceedings. So that takes an extra level of precaution. But that's the confidentiality side. On the ethical use and the verification side, I have lots of safeguards up for that. One would be I create a sandbox. So, whenever I use a project, I'll open up a brand-new folder. It's separate. I have a firm drive, I have a personal drive that's supported by my firm, and then I that I put all my case files in and then on that drive I have a Claude drive and it has all the same cases in it, but it's the only folder that Claude's allowed to go to. And so, all my interactions with Claude are in this sandbox or in this Claude drive. And whenever I want to work outside of it or start handing it out to the firm, I move it over to my personal drive and then it makes it to the firm drive, right? So that's one thing. And by that I can tell what it has and what it doesn't have. So, if I want to limit its analysis on something to say just the complaint and the answer. And I don't want it to look at all the discovery responses. I'll limit it sandbox and I'll just give it those two things and I'll say, look at that. Throughout my use over the last, you know, several months, I've actually now I originally said I would never use it for research. And the reason is because of this hallucination. And quite frankly, I've been doing research for 20 years. I like it. You know, I feel like I'm confident at it, and I don't need Claude to come in here and find the case, right? But that being said, now what I do is, and this is just another example of how you can put together safeguards to protect, you know, against some of these horror stories, is I have a case law corpus. And so, if I'm using cases in a brief, so now I will use it for writing, I do it in sections, or if I'm using cases for an analysis for a memo, I make sure I get all those cases from Westlaw. It is within their terms of use, as long as I'm using it for that, I get the rich text format. Downloaded and I give it to Claude, and it converts it to markdown and then that's what it uses. And if it's I can tell if it's looking for a case because it'll start searching the internet for it. And you can see what it's doing. And now it's got a rule. Don't if you need a case, ask me for it. Right. And so, I give it the case. So, I'm controlling where it's getting the cases from. And then I throw everything it outputs into Westlaw into a Westlaw check. And then I just take that report and give it back to Claude and I say here's the report and it says okay Westlaw suggests these cases. We should look at those. Westlaw said this citation's off. Look at that. This quote's off. This case doesn't exist. You know, I've got three different things. I'm using case law, I'm using Westlaw, I'm using verified sources, and I'm keeping it all in a sandbox, so my AI doesn't start straying, you know, and going out and looking for other things.

Tim Kowal: Yeah, you got kind of AI checking AI, or you got and you've got Westlaw checking the AI. And you're making sure that it's using verified sources and is do you have it do you give it an instruction? You know, do not go on the web to find the case law, tell me if you need the case.

John Stobart: I wouldn’t harness it like that because you never know what it might find out there. I just want to know if that's what it's doing, right? So, if it's going out there because there's a lot of really good blogs out there. It likes blogs more than anything else. So, they'll attach themselves to blogs. And if you look under the hood, you click and see what it's actually looking at, or you ask it to tell you what it's looking at. You can go to those blogs and say, Hey, you know, these are this is a good firm, and I actually know that attorney. You know, this is probably good information. Let me look at this case, you know? Or if you get if it's looking at something that's trash, you know, you can tell like somebody who just is posting things, be like, Yeah, we're not going to look at that bar, you know.

Jeffrey Lewis: Well, I guess we need to make sure we say nice things about AI in case AI starts scraping podcasts for possible content during research.

Tim Kowal: So, I might start citing our own transcripts back to us in our legal research.

Jeffrey Lewis: Yeah. John, let me ask you this. We started the program talking about this new law passed in California, first of its kind in the nation. Do you have any quick hot takes about the new law and how it's going to impact what we do?

John Stobart: Yeah, so two things. You guys covered it pretty well last week. I watched that podcast. She did a good job on that. So, some of the more controversial things. First of all, let's talk about the verification provision. And that's coming out of 128.7. So that's not actually, you know, it that the need of that's not in the 6068.1. And the interesting thing here is now is that regardless of whether you use AI, you have to verify every cite and every source. If you look at 128.7, it doesn't say if you use AI. So, this is an example of where these AI guardrails are actually adding to people who don't use AI. So, like if you I've thrown briefs that I filed you know last year, a couple years ago with AI in there to analyze them and try to use the same arguments again. And it will say, I'll say, hey, pull these arguments out, and I'll come back and say, I've got those arguments. By the way, you have three missed cites, one pin cite is wrong, and one quote is wrong. Right. And this is stuff that I filed with the court that I verified myself. At the Academy, we have quarterly meetings. We just had one in San Francisco last weekend and we were talking about this law. And a lot of people were saying that don't use AI or are using it very sparingly are spending more time verifying cases now than they ever have. Right. Because if you give it to an AI model, you're afraid. What if it changed a word? What if it changed a number in the citation, you know? And so now what you have is whether or not you use AI, cite verifications now fall under 128.7. So that's interesting. That's across the board. Now the verification you guys were talking about last week, that's the next big thing. And I think you really hit on it. It's going to be like a word count certificate. And I think you said you can't over-disclose, right? It doesn't require telling you what part or what you use or what part of your brief you use it for. So, I envision it and I and I my firm's going to put in a practice of just including it at the end of everything. And the big philosophical discussion on this is the signal versus noise ratio. It just becomes noise because everybody's doing it, right? You just perfunctory sign it at the end. So, it's really almost like a Rule 11 signing in federal court. You're singing it, you're saying you used it. So, they can track so if they are finding hallucinations, they can say, Well, you used it, you've got hallucinations, put one and one together. So, I think it's more of like an after-the-fact, you know, provision. So, you know, they got all the gotchas, you used AI, you hallucinated, you know, instead of amazingly, I just thought of this. A lot of the cases that involve AI hallucinations, probably 30% of them, the attorney denies using AI in the first place, especially the earlier cases. And so a lot of the if you go on my website and look at the sanction cases, I'd say about a third of them the attorney denies using AI off the bat. And then one of them even came back and said, I was so nervous I'd lose my job, like in a declaration after explaining to a judge he didn't use AI. So like it gets rid of that, right? We've got you signed this thing, you submitted it, you said you used AI, that's out of the way. Right. So I think that's kind of might be secondary purpose of it, but I don't think it discourages. I don't think, especially as we move further along, I don't think there's going to be any negative connotations put on using AI. I proudly use it obviously, use it responsibly. And if it did hallucinate, you know, I might come up with a bad argument myself and they can say that's so bad AI must have hallucinated it. And I might just say, yeah. Yeah, you're right.

Jeffrey Lewis: Yeah.

Tim Kowal: Well you well, John, you say that you use AI proudly and responsibly. I think a lot of lawyers would like to be able to say that, but maybe they don't know what it means to use it and to use it responsibly. And maybe they're using it in some ways responsibly, in some ways maybe they don't know they're using it irresponsibly. How do you how do you draw that line and how does an attorney say that accurately that I use it proudly and responsibly?

John Stobart: So yeah, I would think the best way to do that is starting again with this small build-up steps, right? So I would never, I still to this day, I would not say write a brief, read it through, edit it, and file it. Or even worse, just take it and file it. I have it like write, like I'll start off with a case memo. Okay. So if I have a case, I'll put in the pleadings, I'll put in whatever documents are there, whatever kind of depositions are available, and I'll just say. Start off with the facts. Okay. What is the procedural facts? I do employment law. What are the facts of this employee? When did they start their employment? What was their and timelines, right? Summary stuff. So now that I've got that document and that memo, now AI knows my basic case facts. So if later on I'm asking it something, it doesn't have to make up his start date, right? It doesn't have to guess as to what his pay was in September of 2017. So you start with a basis like that. Now from that document, I can build a depo outline. I can build an argument section in a brief. I can build a motion to compel discovery. I don't do discovery, but I imagine you could, right? But

Tim Kowal: Yeah. But you're saying it starts it starts with context, right? You're so you it starts with a with the right source of truth. You're gathering your information, you got to think like the like the rules of evidence. You got to lay your foundation. You almost have to think like a like a lawyer. How would you build your case at trial? You have to give good evidence with foundation. You got to do the same thing with your AI. Give it good stuff, because it's garbage in, garbage out. So make sure you're feeding it. It's like giving it a diet, you know. You don't want to fill it full of you know of preservatives, GMOs, and stuff, you want to give it good healthy food. You give it good, healthy, accurate information and you're more likely to get good information out of it.

John Stobart: Yeah, and then you then you work through it and you get suggestions and a lot of them you'll say no to. I'll give a great example. I was I had it for an ex parte to oppose a motion to continue trial, right? Very quick turnaround, very easy. And I asked it to just I just gave it the ex parte application. It already had my case memo, so I've been working on this case. And it came back and it said, John, no problem. This is going to get denied because a year ago the judge said, No more continuances, and that's an order. And I was like, listen, Claude how do I tell you this? The judge said no more continuances a year ago. That's not winning the day. You know? And so and it kept on suggesting that. And I just had to, you know, I am polite with my AI, like that's not it. Or I ignore it. Let's go with option number two and three, you know. So you definitely have to drive it. You have to you have to be in on the process. And so that's like kind of the delegation. I'm not delegating it to make any kind of strategic decisions. I'm not delegating it to, you know, come up with arguments. I'm giving it information. I'm giving it a basis and a backbone. And I'm having it, I'm working with it to develop that. And it and it really come into play when you're dealing with large amounts of information and documents, because it can look at all that at once. I can't. I was just saying I'm working on like a, you know, an 80-page appellate brief right now. It's 17,000 words. I'm over 3,000 words. If I make a change in the first section and it affects a change later, the chances of me catching up on that are hard, or you know, adding something later that's already up ahead. And if I have AI look at that whole thing, it'll say, you know, you use this quote twice, maybe paraphrase it the second time. It says you're repeating this argument in section three and four. And like for me to look at an entire brief and see that, that's nearly impossible. So yeah.

Tim Kowal: What about how did you get how did you get there? I won't hold you to this specific brief, but in general, you've got an opening brief. Are you still writing the same way? Are you is every word of it typed by you or dictated by you? Are you using AI to generate any part of the of that brief?

John Stobart: So the argument sections I'm writing. So I actually limit it to bullet points for that. So when I ask it to help me with a with an argument section, I'll have it do bullet points. And then I'll take that and I'll write it myself because it's like I said, I don't I guess I might have said this beforehand, it's not to the point where it can replace a lawyer. You can tell it's AI-written, it's not there. It's not it can't it can mimic writing like a lawyer, but it's kind of like a TV lawyer. You know, if you really look at it, it's like you can't do that kind of stuff. So definitely with all argument sections, that's all written by me. I might let it help me edit it later. A fact section, I'd probably go I'd go backwards with that. I'll let it take a shot at the facts and then edit those because that's a little drier, right? And actually it helps write it, you know, you want a plain nice recitation of the facts. So and then of course, intro is for me, intro is just all I do those at the end and that's all it. So yeah, depending on the section, either way.

Jeffrey Lewis: So, so John, I think you just answered this question, but I'll ask it anyway. AI is moving so fast, it's improving fast. Some say it's even improving itself, you know, this recursive AI. There are like three camps of people I talk to about AI. You know, one group says AI is coming for all our lawyer jobs, they're going to put us all out of work. Another camp says this is a great boom for lawyers. Lawyers who are with it are going to be able to provide access to justice to people who can't afford lawyers. And then there's a third camp that thinks, well, all this discussion's irrelevant because AI is going to, you know, cause a plague, and kill all of us. Which camp do you think you're going to fall in, not with where AI is today, but where AI is heading in the future?

John Stobart: Yeah. I mean, so we're seeing we're seeing lots of like very fast growth with AI, but it still has very to me I see very the same limitations every time. It cannot supplant a lawyer's thinking. It's never going to do a lawyer's job. I don't think. Not this. If maybe if we get to that next level where it starts thinking on its own, maybe. I still don't think it's I don't think it's not there now and I don't think it'll get there. What it does do is it gets better at helping. Right. It's it gets more intuitive. It takes direction better. It learns how I like to do things better. So in that way, even in the last six months, I've seen it get better with each release. You know, this Opus 5.5 that's out on Claude right now is amazing. It uses very little tokens and it's and it writes better than anything else. It's they've gone through and got rid of all the dashes and all the you know words that you'd see. If you are going to use it to generate text.

Tim Kowal: RIP dash. Yeah.

John Stobart: Yeah. It's funny because I have a rule

Tim Kowal: It's poisoned it for everyone else now.

John Stobart: I have a rule that says no dashes and when I made a brief bank I put in all my appellate briefs from the past five years and it said, John, you have six hundred and seventy two dashes in these briefs.

Jeffrey Lewis: That's funny. Hey, have you have you thought about using that brief bank of all your old briefs as a as a source of truth for AI to mine certain arguments or to find authority? Yeah.

John Stobart: That's exactly what I do with it. Yeah. So I not only have a brief bank, I have a deposition bank. And so I've taken all the PMQ depositions I've done in the last three years. HR, payroll, supervisors, and customer service, right? And I have them all in a corpus and no page line, no numbers, because that's not it's not for citing, just the questions and answers. And it's all meta tagged out to different causes of action. And so if I'm give it a case memo that I create and I say I'm going to depose this person and I give it the documents I want to use for the deposition, it will create an outline now that uses my questions in a format that I like to use. So the first time I had it help me with the depo outline, I was amazed. It came back, but like there's like three sentence long questions. A lot of them were leading. Like you couldn't use it, but it was a good start. This last one that I just did using this corpus, I went in and asked the deposition and I got ahead of myself. You know, I went off and was asking follow-up questions and you know, thought I was, you know, getting some really good stuff. And I looked down when I that road ended in the deposition questions and I looked at my outline and six of the questions I just asked were written down. So it knew where they were written down because I'd done that before. I'd literally gone down that path before. And so had I not picked up on it in real time, it was going to lead me there anyways. So that's like an example of, you know, when I first started using this, the deposition outlines, although good, were not, were not usable. They were what you'd say a good start, right? By training it and giving it, you know, this, like you said, source of truth, this sandbox, this, these are the questions I like to ask and how I like to ask them. It can now, I'll have it put together a deposition outline and I'll read through it once and mark it up. And then read through it a second time to make sure it flows and then go to the depo. And it they're all my questions. And same thing with arguments, you know. You guys last week you were talking about standard of review. I love it when standard review becomes a battleground. I've got one of those right now. It's you know, you're trying to get to de novo from substantial evidence, right? And I've argued that so many times. And now I'm on a respondent's brief where they come in and it's a bench trial. It's substantial evidence and they're arguing de novo and I said, look back at these three cases I did where I argued like they are, and then read the opinion where the court of appeal shot me down and said, No, this is substantial evidence and reverse engineer that so I can get in front of it. I'm not blaming them for doing it. I've done it myself. But, you know, and it came back and it's nothing I haven't written before, but it was right at my fingertips, you know.

Jeffrey Lewis: Yeah. Hey, do you have young lawyers at your firm and do you have any like different policies or different guidance for young up-and-coming lawyers about, hey, learn how to do this crap first before you have AI do it for you?

John Stobart: So that's interesting. At my firm, like I don't know if I said this, but we just introduced AI to the firm six months ago. That's when my journey began. I never used it before. So I never had Claude, ChatGPT or anything. I think it was in April or May. And then after using it for a month, I got a TikTok account. I never had that before either. So I kind of hit the ground running with this. So a lot of people at my firm are still learning. And I actually like sitting back in the meetings when we discussed this and hearing how other people you know, how they are trying to handle these things. And you know, is and I like hearing that's what I like about the TikTok too, is I get to hear how other people are trying to use it. And you got some people who are like, I've got the most perfect prompt, you know, and it's this prompt that's like this long and it goes through like all these things. And I'm like, that's cool, you know. I'll caution you because the more information you put in the prompt, if anything is wrong in your prompt, you're going to get a wrong answer back. But that's cool. But I also maybe just try like a one sentence prompt, you know, and compare them. But like so, like I think everybody's just kind of trying to figure this out on their own. And there is no, you know, manual for it. You can you can pick up as much stuff as you can from where, but nobody really knows how it works. The only way you really get a feel for it is by using it.

Jeffrey Lewis: There is a manual. There's that dude on TikTok who gives you all these tips. That's my manual. Hey, John, have you branched out at all? You know, ChatGPT is pushing out this thing called dots and Meta is pushing out this thing called Muse, these kinds of proactive AI. And instead of you going to AI with a chat, it comes at you. Have you have you played around with any of those products?

John Stobart: No, I would like to, but not for legal work. So a lot of those are where you've got agents running around the clock. My legal work is very, I don't I don't have any kind of sub agents or agents or anything like that. My work with is very interactive. It's me interacting with the bot and a sandbox where it has all the context it needs and coming together to get a product right now for other things. Yeah, I just put a Google speaker. I everything in my house is Siri, but I put a Google speaker in my kitchen so I could so it can tell me recipes as I'm cooking. Okay, what's the next step? What's the next step? And I load all the recipes into Google Keep and it I tell it what I'm cooking and it tells me how to cook it. So now I don't have to carry my phone around. So like there are things like that that I could definitely see it useful for, but I don't see how agents or having things running around the clock would ever help a legal argument. Because I mean there's nothing it can really do without me. Right. Like I'm not going to say work on this brief until tomorrow morning and then let me see what you got. You know? It's a it's a in it's in a you know, I've got thirty-five briefs, oppose them all, see you tomorrow morning. Like that's not useful, you know.

Jeffrey Lewis: I'll be at the beach. Right. No, there are some use cases though. Let me suggest this. I know we talked before we hit record that you like to keep abreast of AI topics and you search for things that are like AI in the news and whatnot. I think these bots can on their own search every day for an AI story and then deliver to your inbox a summary of hey, you might want to write about this or talk about on TikTok. There's a harmless use case.

John Stobart: Yes, that's great, but that wouldn't be my legal work. That's my hobby of being a TikTok. But yeah, or I guess you could set it up to look for cases, but although I don't think it gets the most updated it's not like Westlaw I don't know if it'd get the updated cases.

Jeffrey Lewis: Yeah. Interesting. I'm a little afraid to use that stuff for work stuff, but I did put my toe in the water and tried Muse for booking a recent hotel and a recent airline trip to San Francisco. And instead of giving it my credit card, which kind of scared me, I typed it in manually at that step. But I'll say this Muse product did a great job of finding me a good rate, you know, with leg room and the right hotel and it typed in my member ID number so I get my points and I could see how you could get hooked on this pretty easily.

John Stobart: Yeah, it's worked its way into a lot of crazy little corners of my life. I play guitar and I have a MIDI pedal board. And over the years I've I guess I came up with twenty-two hundred MIDI commands that I've programmed into it. And I gave it to chat and it went through and cleaned them all up in an afternoon and instead and made it even better. So five years of programming a MIDI controller, chat cleaned it up in an hour. So

Jeffrey Lewis: Hey, you mentioned a few minutes ago that you had some involvement in arbitration or you were talking about arbitrations. And I want to ask you, have you heard about the American Arbitration Association's pilot program of doing AI generated results for construction law disputes? Have you heard about that?

John Stobart: I so I heard about I've heard about things like that. Okay. And I also heard your talk last week. Look, just like attorneys can't delegate the practice of law to AI, which is, you know, by staying in the process. And by the way, we mentioned earlier, you know, AI checking AI and citations. That is all the way up until the final step before you file its I also have it keep the actual versions before you file its human eyes on the verifications. And then in that folder that it can't touch and then you send it off. But yeah just like you know attorneys can't delegate you know the practice of law I don't think arbitrators should be able to delegate the decision making on these to a bot either. I mean you kind of joked last week like who wrote the bot? You know, like is it left thinking is it I have no problem with an arbitrator using AI to help come to a decision. I do that myself. I have almost all my arbitrations go through an AI and I'll tell it that I'm I'll even tell it that I'm the other side. I'll be like, hey, I'm the respondent. You know, are we going to win this? That's like a post-mortem type thing, a pre-mortem type thing. So I have no problem with that, but the AI making those calls and those decisions and coming up with the judgment without human interaction is that's not it.

Jeffrey Lewis: You know, the and I and I agree with you, but I have to tell you the interesting thing about this AAA product is it's with full disclosure and full agreement, post dispute, party one and party two agree, you know, we don't want to pay a hundred thousand dollars to Tim to litigate this. We just want to submit our documents to a machine and press a button and get an arbitrated AI decision. And you know, it's even though the parties have agreed to that, it still strikes me as wrong.

John Stobart: It's not there. It would unless it was a very simple dispute. It's not there. It would pick up on some nuance that wasn't important, like the judge saying a year ago no more continuances. Yeah, it's not nuanced enough. It's not there. It would I would be too afraid of it making the decision on something like that. Now doing that with an arbitrator reviewing it and paying that arbitrator, you know, a day's worth of work to review. The decision, now you're talking, right? No hearing, no depositions. Put everything in there and arbitrator review and verify that it's not making a decision on something that's not important. It could get very technical, you know.

Jeffrey Lewis: Yeah. No right of appeal.

John Stobart: Right. And I mean with the technical stuff too, like I'm a I'm a plaintiff's employment lawyer. We a lot of our lawsuits are technicalities, you know, pay stubs, minutes and so AI loves my cases. You know, you got dead to center, you know, like but like juries

Jeffrey Lewis: Yeah. Big cheerleader for you, John.

John Stobart: But juries when you sue, you know, over thirty-five dollars a person, they don't, you know, on the class action, juries are like, You're wasting my time, you know? Or can be. So yeah, I don't think it's there yet. To just a turnkey push button, but you could. I mean, you can also flip a coin.

Tim Kowal: Yeah, at some level it's no different than gambling, right? Unless you know are confident in the in the judgment at the end, which has been the watchword, that you've got to have the human judgment layer involved. But they even the garbage in, garbage out that we were talking about, you have to make sure that the foundation, all the facts, the sources of truth that that the AI is using is on the level and that it's appropriate. And as you mentioned, it's not taking into account, you know, improper factors. And yes, that would be a lot to program and to check and to make sure you know how would you make sure? Do you have to have a human at every one of those checkpoints to make sure that that the programming's being followed? Because AI is different than regular coding. Coding is based on principles of mathematics ultimately. It's all designed to get down to a one or a zero. But AI is different because no one knows what the heck is going on with it.

Jeffrey Lewis: A mysterious black box. Yeah. Yeah.

Tim Kowal: Yeah, you program prompts and skills and everything, but it's all in language that even the prompts, you know, I go back and read some of my prompts and I think to myself, I don't even know what I meant by that. But the AI is going to take a shot at it.

Jeffrey Lewis: Yeah.

John Stobart: I have the skills I do use are updated about every time I use them. So the deposition corpus, the brief bank, every time I put a case in, it's like, hey, this case puts numbers on the bottom left corner. It's the only case I've ever seen do that. So it miscounted everything. So I'm going to add to your skill now that if it's in the bottom left corner, do this. So like I don't have a single skill. I thought that was going to be the next thing too, is turnkey skills. It's I my skill is refined every time I use it. And it and you know, there's nothing; it's a moving target. So like to even if you did set it up to analyze the facts and do all these things, it's not going to do it the same way every time, you know.

Tim Kowal: I have a member of my team, what one of his core functions is to integrate all of the new skill updates that I've stumbled across through a given day. Every time I do something and it's like, no, that that's not the way I want it. Update the skill backlog and have Edwin implement it when he gets around to it. And so he has this growing database that grows by like a dozen new entries every day of things that I've updated with my skills because they don't do it right.

John Stobart: Yeah, and if you have a GitHub, you'll find out I'm on like version a hundred and seventy-two of my markdown maker. Every time it sees something new, it wants to update. Yeah. So I don't think it's to a point of like a turnkey point it and go type situation yet. But I tell you this, man. And you know what? I'm not sure I've saved any time with AI. You know, that was my very first thing I came out and said, you know, that was with better briefs. It's not to save time, it's to it's to make it better, right? And you might save time on certain tasks, but like I mean, a lot of times the way I work, I've got a certain amount of time to finish something before I move on to the next thing. If I have three days to prepare for a PMQ deposition, I'm going to spend all three days, right? Now instead of spending a day and a half putting together the exhibits, I can do that in the morning and spend that day and a half going through them and really finding, you know, the good information out of them. So I've I d I'm still don't think I've and now and at the end, of course, you've got to you know, bake in the time to check everything. So that's an additional, you know, time that I mean, I guess we just copied and pasted from old briefs and filed it in the past. Is that what we is that what's going on? Is that

Jeffrey Lewis: Right. Standard of review anyway. Yeah.

Tim Kowal: The old days was what, six months ago?

John Stobart: Yeah.

Jeffrey Lewis: Yeah. Hey, John, don't embarrass anyone in public. Don't name any names. Don't get me sued for defamation. But do you have a story of feeding an opponent's brief or work product into ChatGPT or Claude and discovering something that you wouldn't have discovered but for AI?

John Stobart: Well, yeah. On that jury trial we just had, we lost. All right. We've got a great appeal. I like it better as an appeal as I did, right? But their judgment, they you know, they sent the judgment over and we're just like, whatever, you know, like who's really going to fight [inaudible] a judgment that says you lose, right? It's to protect them. And after they put through like three of them, I we were curious to like why is this the third one? I threw it through and had it look at everything. They had the parties wrong, they had dates wrong, they had there was like six, five or six things in that judgment that as an appellate attorney, as you guys know, could be very problematic, especially for res judicata purposes in the future. And so they obviously did not run it through AI because it would have caught it immediately. It's just the kind of things they it loves looking for that. It loves looking for names, it loves looking for. Any kind of little technical like difference. If you ask it to find problems, that's what it's looking for. You know, it's looking for the first, second, second, third paragraph, numbered paragraphs, things that it can really claw into and say come back to you and be like, look what I found. This is wrong. And yeah, you know, and I ran it through there and I had to tell, you know, the other attorneys, I was like, I think we should probably bring this to their attention. I don't care because I'm never going to try to enforce this thing. But you know, what do you do? So there's lots of times where I'll put opponents' briefs or things in there and I'll pull out errors. I think that's pretty significant error though. When you're talking, I mean, their client paid a lot of money for that judgment.

Jeffrey Lewis: Yeah. Yeah, that's something you want to get right. You know, the day is coming when standard of care experts and legal malpractice cases are going to say spell check, grammar check, and Claude check your brief before you file it. And if you haven't done that final second set of eyes on your brief before you filed it, you fell with it below the standard of care in terms of litigators. I think that day's coming. Not today, maybe next year.

John Stobart: That's the it's a two-way sword and that's very interesting because I feel the same way. I think it is going just like it's tangentially now raised the standard on 128.7 where everybody has to verify cites. That wasn't in there before it just you had to, you know. I think it will raise the standard of care because if you do use a baseline like we started off in this podcast, if you use it to just review your documents before you file, right? Don't even have it make edits. Make it suggestions. You can make the edits yourself. And in that case, you wouldn't be using generative at all, right? You would just be those levels of errors should disappear. They should be, they should no longer happen, right? Party names, all the things that it can find out. So I do think that at a certain level, it is going to raise benchmark that, you know, remember research attorneys are putting these into AI at the state levels. That that came out a while ago in California. They're going to put your brief into AI, and their AI is going to tell them right away if those errors are there. Right. So remember that too. And your opponent's going to put that your brief into AI and they're going to find all your errors. And you know, there's a certain baseline of errors that even just a cursory, like you said, started with spell check, then you got like a grammar check now. There will be like an AI level check, which goes one step deeper, it goes into context. It also I have mine set up for Bluebook and for California Style Manual. It whips citations into I mean it perfectly and you know, case names and abbreviations and everything.

Tim Kowal: So is that part of your like your skill bag, your bag of tricks? Do you have kind of a short list of recommendations you think every attorney should have in their skill bag, like you know, like an AI brief proofreader to make sure that you're not doing anything that's going to make you look stupid. You know, just basic things like you know, it's just one step beyond the spell check and the grammar check. It's making sure that, you know, when I paste it in, you know, a piece of my previous brief, it didn't have an old party name. In it and things like that.

John Stobart: It'll find that right away. It will not let you get away with that. It'll also catch dates. So many times you put discovery documents in there and people try to put the in the in the definitions, you know, the like the class period or the claims period, it catches those every time. It's a this number's outside of range. You can have that date in there 90, you know, 35 times. You don't catch one of them. It catches it. So yes, and that's actually come preloaded with Claude, it's check doc. So if you open up your word and you have the Claude side panel and you go forward slash check doc, it'll do all that. That's already there. I'll go ahead.

Tim Kowal: Is that just the plug-in, the Claude plug-in right in Word?

Jeffrey Lewis: The and the most amazing thing is it'll track changes if you don't trust computers, if you're worried about how [inaudible], the changes will show up on the margin as Claude instead of Tim or John. Yeah, it's magical.

John Stobart: You guys are going to you guys are going to flip right now because I'm we're pushing back this brief back between me and my coworker and my Claude his Claude is putting comments and my Claude is answering his Claude's comments. I'm not kidding you.

Jeffrey Lewis: Claude comments. I've never had Claude generate comments before. I'm going to have to try that.

John Stobart: Yeah, I wouldn't I wouldn't recommend starting with that, but you can get there.

Tim Kowal: Yeah. Cause I will use I'll use Claude the way you suggested earlier, you know, ask it to red team this brief, you know, pretend you're the other side, tear it apart. Because when I was working with Claude, it said it was great, you know, it was definitely going to win. And then you put it, you just prompt it differently. I used to use Claude, you know, and then I would have ChatGPT red team it, thinking that two different models made a difference. But I don't think it's having two different models. You just put it in a new chat thread. And prompt it differently, like now be antagonistic, be adversarial and pick apart my brief, and it'll do it. You don't have to put it into a different model.

John Stobart: Yeah, just be careful because if you have in your instructions or in your about me file and it knows that you're an attorney for a certain side, it will check that and it'll be like, I know what you're trying to do. Seriously, because it because it's so I actually clean those all out. I don't have any of those instructions. I make sure the instructions I want it to get are loaded with each when I want it to use it. Does it always need to know that I'm a plaintiff-side, you know, class action attorney? It doesn't always need to know that, especially if I'm trying to do that, what you're doing.

Jeffrey Lewis: It knows, John. It knows. Have you heard about Hugging Face? It knows. And I'm surprised you say it, not he or she. Your AI knows. They have ways of getting information. So

John Stobart: I guess if you're using multiple you'd say you'd say they, right? They know. My all of my AIs know.

Tim Kowal: Yeah. They is more yeah, more frightening, more ominous.

John Stobart: Yeah.

Jeffrey Lewis: So John, remind us, what is the website if people want to, if they are afraid of TikTok and but they want to watch their videos, what is the website where people can find your advice about Claude?

John Stobart: Yeah, so it's stobart.ai my last name dot ai. Almost all of my good TikTok videos have made it on there. And you can watch them from your phone even. ChatGPT hosts that, so that's not a Claude product, that's a chat product. It went everything, it made the entire site and even hosts it for me. And if I want to make any changes, I literally say, Hey, add this and it does it. But that is where I have news prompts, I have some videos up there, all the case banks. I just added two cases to it recently. One case where it was a firm with AI policy and everything and still got filed a hallucination because one person didn't follow the policy. So Harvey policy, all the things, and one person just didn't do that last verification step and got

Jeffrey Lewis: Yeah, only as strong as the weakest link in the chain.

John Stobart: But yeah, stobart.ai and then it's @AI4Lawyers on TikTok.

Jeffrey Lewis: Okay. All right. And would you follow me on TikTok, please? Would you give me a follow, please? I think I have five followers on TikTok. Give me a follow when you have a chance.

John Stobart: Okay. It's going to ruin my ratio, but I'm going to do that, Jeff.

Jeffrey Lewis: Okay. All right. Thank you so much. You've been so generous with your time. And Tim, thank you for yielding the microphone to me. I know you usually do the heavy lifting. I think.

Tim Kowal: The iron grip on it, yeah.

Jeffrey Lewis: I think that wraps it up. Unless there's any other questions you have, Tim.

Tim Kowal: No, that's it. Thank you, John, for joining us. AI was used nonspecifically in the production of this podcast.

John Stobart: Yes.

Jeffrey Lewis: Nicely done.

Tim Kowal: All right, we'll see you next time.