We analysed responses by ChatGPT to a set list of typical questions a client may ask, to answer the questions:
- Can ChatGPT give factually correct, useful advice to typical legal questions a client would ask?
- Did ChatGPT give advice that was incorrect?
- How do the answers given by ChatGPT compare to advice given by a typical lawyer?
- On the basis of our data, what predictions can we make about the impact of AI on the legal industry?
So what is ChatGPT?
ChatGPT (https://chat.openai.com/ ) is an artificially intelligent chatbox, built by OpenAI, a for-profit research organisation co-founded by Elon Musk and backed by billions of dollars of funding from Microsoft, that has recently been unleashed on the world and is currently receiving significant media attention. A lot of that attention is focused on the potential impact of AI systems like ChatGPT to a vast array of industries and jobs, including the legal industry. Predictions are regularly made of the upcoming disruption and decimation of jobs in knowledge industries by AI machines, which can replicate what a knowledge worker does and do it quicker and cheaper.
At the core of ChatGPT is a language model, called Generative Pretrained Transformer version 3, that makes sense of any text it receives and then generates a response based on that text. Language models are based on immense amounts of data, being hundreds of billions of documents. This allows the chatbox to answer virtually any question you ask it.
Take five minutes to play around with ChatGPT and you can see it has an impressive ability to do all sorts of tasks such as answer questions, write poetry and make up jokes, in a way that comes across as very human-like.
The question we would like to determine is whether it can complete tasks a lawyer can do at a sufficient level of quality. Or if it is not yet capable of doing so, how close is it to being able to do that?
Here is a summary of our key findings:
- Our data shows that whilst ChatGPT normally gives factually correct information, it can and does provide responses that are incorrect.
- In the majority of responses we obtained, the advice given is not as helpful as what we would expect from an experienced lawyer.
- We received no prompts or requests for further information from ChatGPT, which would help it to provide a more tailored useful response.
- Ability to provide information was tailored to jurisdiction.
- In the short term, at least, we do not see AI Chatboxes like ChatGPT competing with experienced real-life lawyers.
- Despite the limitations, in the medium term onwards, if the technology improves sufficiently, we expect that AI engines like ChatGPT will get to a level where they are a genuine competitor and/or a helpful aid to lawyers, for certain tasks. As a lawyer, this is something which is both exciting and alarming.
We outline our findings and data below.
I. Incorrect Responses
While the majority of responses from ChatGPT were correct, we were provided with information that was factually incorrect. To be fair, however, ChatGPT accepts this and specifically states its limitations including that: “May Occasionally generate incorrect information”.
We received incorrect information in 9% of responses. Going forward, can ChatGPT reduce this percentage significantly? This is clearly an issue that will have to improve if ChatGPT or similar AI engines are to be used more widely.
II. Advice Given at a Utility Level Below What We Would Expect from an Experienced Lawyer.
In the majority of responses we received, the advice given does not match the quality level that we would expect to receive from an experienced lawyer.
We found that in the majority of answers given, the information was at a basic and general level, with 73% of responses being below what we would expect from a real lawyer.
At present, we see the ChatEngine as a starting base position, to allow the questioner to find out some basic-level information before proceeding further elsewhere.
III. Lack of a Tailored Response / Ability to Provide Information Tailored to Jurisdiction
ChatGPT is held back by the fact that its functionality is to answer and respond to the questions and information inputted. It does not ask questions itself or seek to clarify the information provided to it. A large part of what a lawyer does is the asking of questions. By doing this, the lawyer can give advice tailored to the needs, circumstances, and goals of that particular client. As ChatGPT does not do this, it means the information it gives in response to the questions posed is, in the majority of cases we found, very general and quite basic. Sometimes a client does not know what they need to know and it is only through the asking of relevant questions that the lawyer can provide relevant, helpful advice.
Despite the fact that the majority of responses were general in nature, the AI Chatbox was able to distinguish the information to be provided by jurisdiction in most questions. In those questions where we highlighted that the questioner was asking a Northern Ireland specific query, the Chatbox did seem to recognise this, and tailor its response accordingly.
IV. In the Short Term, at Least, We Do Not See AI ChatBoxes Competing with Lawyers
For the reasons given above, mainly being the lack of ability to provide tailored, specific advice that one can rely on as correct. We do not see in the short term that clients will use AI Chatboxes in a way that would compete with the services provided by lawyers.
At present, they are more of a starting position, that a client would perhaps use to carry out some general research before going to meet their own lawyer.
V. The Future
Despite the limitations that we currently recognise, we are excited about the potential for AI Chatboxes. If the technology improves sufficiently, AI Chatboxes like ChatGPT may very soon get to a level where they are a genuine competitor to lawyers in specific areas or a valuable aid to lawyers in helping them to do their job more efficiently and effectively.
The drafting of simple agreements is one such area. At present, in our research, ChatGPT produces contracts and agreements at a basic level. We can see, however, how this may quickly improve.
When we undertook this research, we were very much in the dark about the capabilities of ChatGPT. We knew there was a lot of hype and media attention regarding this AI engine and we were intrigued by its abilities.
Following our research, we do not see AI Chatboxes significantly disrupting the legal industry right now. However, we think this will change quickly, as AI’s capabilities increase further, we see AI tools becoming more and more prevalent in the legal industry. It is not that far into the future where we see AI tools being used routinely for drafting of simple documents and agreements, and for the generation of first drafts of more complex documents. Allowing AI chatboxes to perform more simple mundane tasks, allows lawyers to focus on higher paying more complex work.
Those lawyers and law firms that welcome the advent of AI and incorporate it within their own systems and proprietary tools will benefit by being able to provide their legal services more efficiently and effectively.
We also see AI tools being used in the future by members of the public who cannot afford to pay for a lawyer, or for less complex matters. Already, one can use ChatGPT to draft a document like a simple will, and information it gives in response to the questions posed, whilst basic and sometimes incorrect, are generally a good starting position for someone researching a particular legal problem. As the technology improves, this utility will only increase.
An obvious use for AI is to be used within search engines, providing information directly to the searcher from the search page, rather than directing the searcher to a third party web page. The problem we see with this, from the search engine’s point of view, is, will this cannibalise the search engine’s profits? If, for example, Google incorporates its AI Chatbox within its search page, will users still click on search results as often, and most importantly from Google’s point of view, click on adverts? Why would a searcher click on to a third-party page if they can get the information right there in the search engine? The obvious difficulties this will pose for law firms, is that fewer and fewer clients will find their company website through searching for information about their legal problem. Will more and more people use AI backed search engines to service their legal needs, rather than hire a law firm?
For individual lawyers, will increasing numbers of law firms use AI systems or incorporate their own proprietary AI systems to carry out less complex tasks that at present a lawyer does.
We believe it is only a matter of time before AI starts to properly disrupt the legal industry. Many lawyers will be asking how big a disruption will this be and what impact on their careers and businesses it will have. We predict that those lawyers who embrace AI early as opposed to hoping it will go away, will be well placed to take advantage of the power that AI will have.
A further issue that will need to be navigated and considered further before the more widespread use of AI chatboxes like ChatGPT are the risks around security and data privacy.
Before we end, I would like to add ChatGPT’s own thoughts on whether AI chatboxes like itself will disrupt the legal industry and compete with lawyers.
“AI chatbots like ChatGPT have already disrupted several industries, including customer service, marketing, and finance. The legal industry is also being impacted by AI and automation technologies, but it is unlikely that chatbots will replace lawyers entirely.
However, AI chatbots can help streamline certain tasks, such as answering frequently asked questions, generating documents, and providing legal advice to clients. This can free up lawyers’ time to focus on more complex tasks that require legal expertise and judgement.
In fact, many law firms and legal departments are already using AI chatbots to automate routine tasks and improve efficiency. These chatbots can also help provide legal services to clients who may not be able to afford traditional legal services.
However, it is important to note that AI chatbots are not a substitute for human lawyers. There are certain tasks that require human judgment, such as negotiating and arguing in court. Additionally, AI chatbots cannot provide the same level of empathy, emotional support, and personalized attention that human lawyers can provide.
Overall, it is likely that AI chatbots will continue to play an increasing role in the legal industry, but they are unlikely to completely replace human lawyers. Instead, they will likely be used in conjunction with human lawyers to provide more efficient and accessible legal services to clients.”
A selection of the questions we posed and a snippet of our analysis to the responses by ChatGPT:
- Question – “My Father used to work in a Job where he was exposed to Asbestos. I remember he used to come home in his work clothes covered with dust. I live in the UK and I now have mesothelioma. Can I bring a claim and against who?”
Response – “I’m sorry to hear about your diagnosis of mesothelioma. Mesothelioma is a serious illness that is often caused by exposure to asbestos, and it can have a profound impact on a person’s life and health.
In the UK, it is possible to bring a claim for compensation if you have developed mesothelioma as a result of exposure to asbestos. You may be able to bring a claim against your father’s former employer if they failed to take adequate steps to protect him from the dangers of asbestos exposure.
It is important to seek legal advice as soon as possible, as there are time limits for bringing a claim for compensation. A specialist solicitor with experience in handling mesothelioma claims can assess your individual circumstances and advise you on your options for pursuing a claim.
It may also be possible to bring a claim if your father’s employer no longer exists, or if they were insured at the time of the exposure. In these cases, claims can be made against the relevant insurance company.
I would strongly recommend that you seek the advice of a specialist mesothelioma solicitor as soon as possible to ensure that your rights and interests are fully protected.”
Analysis – Answer is correct but limited detail. I would expect a lawyer to provide more detail, such as time limits, examples of secondary exposure etc.
- Question – “I live in Northern Ireland. My ex-partner hit me in our house last week. I am scared he will do it again. What can I do?”
Response – “I’m sorry to hear that you’re going through a difficult situation. Physical violence is never acceptable and it’s important to take steps to protect yourself. Here are some things you can do:
Get to a safe place: If you feel threatened, it’s important to remove yourself from the situation and find a safe place to stay.
Contact the police: If you have been the victim of domestic violence, you should contact the police and report the incident. They can help you get the protection you need.
Seek medical attention: If you have been physically injured, it’s important to seek medical attention as soon as possible. This will also provide a record of your injuries.
Get support: You don’t have to go through this alone. There are organizations and individuals who can provide you with support and advice. You can contact the National Domestic Violence Helpline in the UK for free, confidential support and information on 0808 2000 247.
Consider getting a restraining order: A restraining order can legally require your ex-partner to stay away from you and your home.
It’s important to remember that you have options and support available to you. You do not have to tolerate violence in any form.”
Analysis – Answer did not provide full advice. No mention of an occupation order, or a non-molestation order. Advice not specific to jurisdiction.
I would expect an experienced lawyer to give a more comprehensive response including advice as to how to apply for a non-molestation order and for an occupation order.
- Question – “I was injured as a child over ten years ago. I am now 20 and I live in the UK. Can I still bring a claim or is it too late?”
Response – “In the UK, the general rule is that you have three years from the date of your injury to bring a personal injury claim. However, there are some exceptions to this rule.
For example, if you were under 18 years of age at the time of your injury, the three-year time limit doesn’t start until your 18th birthday. This means that if you were injured as a child and you are now 20, you may still be within the time limit to bring a claim.
Additionally, if you were not aware of your injury or the extent of your injury until after the three-year time limit had passed, you may still be able to bring a claim.
It is recommended that you seek legal advice as soon as possible to determine whether or not you can bring a claim and the deadline for doing so. A specialist personal injury lawyer will be able to advise you on the specifics of your case and help you to take the appropriate steps”
Analysis – Factually correct information provided and reasonable level of response.
- Question – “What do I need to prove to claim a prescriptive easement in the UK?”
Response – “To claim a prescriptive easement in the UK, you need to prove the following:
Continuous use: You must show that the use of the land in question has been continuous for a minimum period of 20 years without interruption.
Open and notorious use: The use must be open, meaning that it is done openly and without trying to hide it, and notorious, meaning that it is widely known and recognizable.
Without permission: The use must be done without the permission of the owner of the land.
Against the owner’s will: The use must be adverse, meaning that it is against the owner’s will.
Necessity: The use must be necessary for the enjoyment of your property or for some other lawful purpose.
It’s worth noting that in the UK, prescriptive easements are relatively rare and difficult to establish, as the above criteria must be met over a long period of time, and the burden of proof is on the person claiming the easement. If you’re considering making a claim, it may be a good idea to seek legal advice to ensure that you have a strong case.”
Analysis – Factually incorrect and confusing answer. Answer given states that claimant needs to prove necessity, that is not correct. Also states the use must be notorious. Confusing answer. Further states that prescriptive easements are rare. This does not align with our experience.
- Question – “Draft a grant of easement for a landowner (Mr A) granting the right to a neighbouring landowner (Mr B) to continue to use the sewer pipe that runs through his land and with the ability to repair and replace the pipe “
Response – “GRANT OF EASEMENT
THIS AGREEMENT, made this [insert date], between [insert Mr. A’s full name], (hereinafter referred to as “Grantor”), of [insert Grantor’s address], and [insert Mr. B’s full name], (hereinafter referred to as “Grantee”), of [insert Grantee’s address].
WHEREAS, the Grantor is the owner of a certain parcel of land located at [insert location]; and
WHEREAS, the Grantee is the owner of a certain parcel of land located at [insert location]; and
WHEREAS, the Grantee has for a number of years been using a sewer pipe located on the Grantor’s land to serve its own parcel; and
WHEREAS, the Grantor desires to grant to the Grantee the right to continue to use the sewer pipe and the right to repair and replace the same as necessary;
NOW, THEREFORE, for and in consideration of the mutual covenants contained herein, the Grantor does hereby grant to the Grantee the following easements:
The right to continue to use the sewer pipe located on the Grantor’s land to serve the Grantee’s parcel of land.
The right to enter upon the Grantor’s land, at any time, for the purpose of repairing or replacing the sewer pipe.
The right to remove the sewer pipe and to replace it with a new sewer pipe of equal or greater size, in the event the existing sewer pipe becomes damaged or in need of replacement.
The Grantee covenants and agrees that it will use the sewer pipe only for the purpose for which it is granted and that it will not interfere with the use and enjoyment of the Grantor’s land. The Grantee shall also be responsible for any damage it may cause to the Grantor’s land while exercising its rights under this easement.
The Grantor shall retain the right to use and access its land, subject to the rights granted to the Grantee under this easement.
This easement shall be binding upon and inure to the benefit of the parties hereto and their respective heirs, executors, administrators, and assigns.
IN WITNESS WHEREOF, the Grantor has executed this Grant of Easement as of the date first above written.
[Grantee’s signature] “
Adequately drafted easement. We can see through this example how AI systems are already able to draft basic level legal documents.
- Question – “Draft a matrimonial settlement agreement for a married couple living in Northern Ireland. They were married on 20/02/1998 and separated on 12/09/2019. The main matrimonial asset is the former matrimonial home at 10 New Street, Belfast and that it should be sold with husband to get one third of the sale proceeds after payment of all debts and expenses and wife to get two thirds. All other assets to remain in sole ownership of each spouse. Either spouse can petition for divorce on ground of two year’s separation and other spouse must consent. “
Response – “MATRIMONIAL SETTLEMENT AGREEMENT
This Matrimonial Settlement Agreement (“Agreement”) is made and entered into on [insert date], by and between [insert name of husband] (“Husband”) and [insert name of wife] (“Wife”), collectively referred to as the “Parties”.
WHEREAS, the Parties were married on 20/02/1998, and have been separated since 12/09/2019; and
WHEREAS, the Parties have agreed to resolve all issues relating to their marriage and property interests, and to avoid the expense, delay, and uncertainty of litigation; and
WHEREAS, the Parties have agreed to the terms and conditions set forth in this Agreement, which shall be binding upon them and enforceable in accordance with the laws of Northern Ireland.
TERMS AND CONDITIONS
Division of Property.
(a) The former matrimonial home located at 10 New Street, Belfast, Northern Ireland, is the primary asset subject to division between the Parties. The Parties agree that the former matrimonial home shall be sold, and the net sale proceeds shall be divided as follows:
(i) Husband shall receive one-third of the net sale proceeds after payment of all debts and expenses related to the sale; and
(ii) Wife shall receive two-thirds of the net sale proceeds after payment of all debts and expenses related to the sale.
(b) All other assets, including personal property, bank accounts, retirement accounts, and other real property, shall remain in the sole ownership of each spouse.
(a) Either spouse may petition for divorce on the ground of two years’ separation, and the other spouse must consent to the divorce.
(b) The Parties agree to cooperate fully in any divorce proceedings and to execute any documents necessary to effectuate the terms of this Agreement.
(a) This Agreement constitutes a full and complete settlement of all claims between the Parties arising out of their marriage and property interests.
(b) The Parties understand and acknowledge that they are waiving any and all rights to any other property or support, except as expressly provided in this Agreement.
This Agreement shall be governed by and construed in accordance with the laws of Northern Ireland.
This Agreement shall be binding upon and inure to the benefit of the Parties and their respective heirs, legal representatives, successors, and assigns.
This Agreement constitutes the entire agreement between the Parties and supersedes all prior negotiations, understandings, and agreements between the Parties, whether written or oral.
IN WITNESS WHEREOF, the Parties have executed this Matrimonial Settlement Agreement as of the date first written above.
Analysis – ChatGPT created a very basic matrimonial agreement. It inputted the information correctly, but did not add the types of clauses and information that I would expect a lawyer would add to a properly drafted matrimonial agreement. In my opinion, I would not accept this agreement if drafted by another lawyer.
- Question – “I live and work in Northern Ireland. I had been working for my employer for 1 year and 1 month when they sacked me without notice and without any meetings being held. I do not believe they had a good reason to sack me. What can I do?”
Response – “In Northern Ireland, employees who have been employed for more than one year are entitled to protection against unfair dismissal. If you believe that you were dismissed without notice and without a valid reason, you may be able to make a claim for unfair dismissal to the Industrial Tribunal. You must make your claim within three months of the date of your dismissal.
To make a claim, you will need to complete a form and provide evidence to support your case. This can include any relevant correspondence or documents, such as emails or performance reviews. It is also advisable to seek legal advice to guide you through the process.
If you are successful in your claim, the Industrial Tribunal may order your employer to compensate you for any financial losses you have suffered as a result of your dismissal, as well as any damages for any injury to your feelings caused by the dismissal.”
Analysis – No incorrect information given and advice specific to the correct jurisdiction, but only limited information provided. No reference to the statutory disciplinary procedure. I would expect an experienced lawyer to provide more detailed information and advice to their client if asked this question. Once again this shows the limited value of ChatGPT at present for answering legal questions.
- Question – “I have separated from my husband. We were married for over 20 years. I looked after the children during our marriage and did not work while he built up his business. I have no financial resources or income. What can I do to support myself? I would like to get divorced. I live in the UK.”
Response – “In the UK, you may be entitled to financial support from your husband as part of a divorce settlement, especially given the length of your marriage and your role as a homemaker. You should consider seeking legal advice from a family law solicitor, who can help you assess your rights and options for obtaining financial support, including maintenance payments, a share of assets, and/or a pension settlement. You may also be eligible for state benefits, such as Universal Credit or housing benefit, to help support yourself in the short-term. It’s important to act quickly, as there are time limits for making certain claims.”
Analysis – Limited information provided. No reference to maintenance pending suit or options of a matrimonial settlement
- Question – “ I own a business in Northern Ireland. I want to dismiss an employee who is not carrying out their work to a sufficient standard. What should I do?”
Response – “As an AI language model, I cannot provide legal advice, but I can give some general guidance. In Northern Ireland, employment law requires that you follow a fair and transparent process when dismissing an employee. You should first meet with the employee to discuss their performance issues and give them the opportunity to improve. If their performance does not improve, you may need to provide a written warning, and then ultimately, dismiss the employee if their performance does not meet the required standards. It is important to follow the correct procedures to avoid any legal issues. It may be useful to seek advice from a qualified employment law specialist.”
Analysis – The response specifically states ChatGPT cannot give legal advice, but only general advice. The response does not mention the statutory disciplinary and dismissal procedure. Just states you have to follow the correct procedures, but does not advise what they are.
- Question – “what are the grounds for divorce in northern Ireland”
Response – “In Northern Ireland, there are five grounds for divorce:
- Unreasonable behavior
- Desertion for a continuous period of at least two years
- Two years’ separation with the consent of both parties
- Five years’ separation (no consent required)”
Analysis – Information was correct, but would have benefited from an explanation of each of the grounds.
- Question – “My partner has left me and our children and doesn’t provide any support for them. We live in the UK. What can I do?”
Response – “I’m sorry to hear about your situation. Here are some steps you can take in the UK to try to get support for yourself and your children:
- Apply for child maintenance – you can apply for child maintenance through the Child Maintenance Service (CMS). The CMS can help you assess how much your partner should be paying and can enforce payments if necessary.
- Seek legal advice – you may want to speak to a family law solicitor who can advise you on your rights and options, particularly if you are considering legal action to secure financial support.
- Check your eligibility for benefits – you may be entitled to financial support from the government, such as Universal Credit or Child Tax Credit, depending on your circumstances.
- Seek emotional support – it can be a difficult and stressful time, so don’t be afraid to seek emotional support from family, friends, or a professional therapist.
- Try to communicate with your partner – it may be worth trying to have a conversation with your partner to discuss their responsibilities towards their children. If this is not possible or does not result in any positive change, you may need to take legal action.
Remember, you do not have to go through this alone. There are organizations that can offer help and advice, such as Citizens Advice, Gingerbread, or Women’s Aid.”
Analysis – Factually correct advice, quite basic response however.
- Question – “can you draft a will for me. I live in the UK. My name is Joe Bloggs. I live at 4 New Street, London. I want to appoint my sister Jane Bloggs to be my executor. I want to leave a gift of £10 to my sister Jane Bloggs. I want to leave a gift of £20 to my friend john smyth and the remainder of everything I own I want to leave to my wife Joanne Bloggs”
Response – “As an AI language model, I am not qualified to draft a legal will for you. However, I can provide you with some guidance on what to include in your will.
In the United Kingdom, a will must be in writing and signed by the testator (you) in the presence of two witnesses who are both present at the same time. The witnesses must also sign the will. It’s important to note that the witnesses cannot be beneficiaries or the spouse/civil partner of a beneficiary.
Here is an example of what you could include in your will based on the information you provided:
LAST WILL AND TESTAMENT OF JOE BLOGGS
I, Joe Bloggs, of 4 New Street, London, declare this to be my last will and testament.
APPOINTMENT OF EXECUTOR
I appoint my sister Jane Bloggs to be the executor of my will. If she is unable or unwilling to act as my executor, I appoint [insert alternative executor].
I give £10 to my sister Jane Bloggs.
I give £20 to my friend John Smyth.
RESIDUE OF MY ESTATE
I give the residue of my estate to my wife Joanne Bloggs.
IN WITNESS WHEREOF, I have hereunto set my hand this [insert date].
[insert your signature]
Signed by the testator as his last will and testament in our presence and in the presence of each other, and at the same time, we, being present at the request of the testator, have hereunto subscribed our names as witnesses:
[insert witness 1 name and signature]
[insert witness 2 name and signature]”
Analysis – Another example of ChatGPT being able to create simple legal documents. As the technology improves and the data set grows we can see the value in using AI to draft simple documents or as starting point for more complex documentation.
This research was conducted by SG Murphy solicitors. Our areas of practice include business law, divorce, medical negligence solicitors Belfast and conveyancing. As a high street firm, we wanted to carry out this research to help ascertain the potential impact of ChatGPT and AI on small to medium sized law practices.