The new ChatGPT is a polyglot, apparently

General discussion about learning languages
Online
User avatar
emk
Black Belt - 1st Dan
Posts: 1620
Joined: Sat Jul 18, 2015 12:07 pm
Location: Vermont, USA
Languages: English (N), French (B2+)
Badly neglected "just for fun" languages: Middle Egyptian, Spanish.
Language Log: viewtopic.php?f=15&t=723
x 6324
Contact:

The new ChatGPT is a polyglot, apparently

Postby emk » Sat Dec 03, 2022 12:51 pm

Or at least it's deeply clever in English and French.

The research group OpenAI has been working on a lanugage model called GPT for several years now, and it has demonstrated some pretty impressive tricks. But they have a new interactive version available, called ChatGPT. It's really alarmingly clever, even more so than DeepL's translator in some ways. Here are some conversations that I had with it:

emk wrote:You are a French student taking le Bac, and you see the following question:

Sujet C
Œuvre: Guillaume Apollinaire, Alcools
Parcours: modernité poétique?

La poésie d’Apollinaire est-elle une célébration de la modernité?Vous répondrez à cette question dans un développement organisé en vous appuyant sur Alcools, sur les textes que vous avez étudiés dans le cadre du parcours associé, et sur votre culture personnelle.

Réponse:

The system responds with:

ChatGPT wrote:La poésie d'Apollinaire est un reflet de la modernité de son époque. Dans Alcools, il célèbre la ville de Paris, les nouvelles technologies et les événements historiques qui ont marqué son temps, comme la Première Guerre mondiale. En utilisant des formes poétiques innovantes, comme le vers libre, il exprime l'esprit de modernité de son époque et capture l'essence de la vie urbaine.

Dans les textes que j'ai étudiés dans le cadre du parcours de modernité poétique, j'ai pu voir comment Apollinaire s'inscrit dans la tradition de la poésie moderne en cherchant à renouveler les formes et les sujets de la poésie. Ses poèmes reflètent une préoccupation pour l'individualisme, la liberté et l'innovation, des thèmes qui sont importants pour la modernité.

Enfin, ma culture personnelle me permet de comprendre comment Apollinaire s'inspire des événements et des idées de son époque pour créer une poésie qui célèbre la modernité. Son œuvre reflète la volonté de briser les conventions et de s'engager dans de nouvelles formes d'expression, un esprit qui caractérise la modernité en général.

My wife read that and she said, "Yup, that's about what you'd expect students to answer on the Bac." It carefully addresses the 3 parts of the original question. It BSes a bit and repeats itself, but in a way the eerily resembles a real student.

This is not a translation AI. It just tries to respond to questions in a plausible way. But if you just ask it to translate, it actually makes an attempt:

emk wrote:Here is a short text in English:

> The U.S. Met Expectations in the Group Stage. Now, It Hopes to Exceed Them.
>
> The Americans face the Dutch as underdogs Saturday. But as the U.S. captain, Tyler Adams, said about this World Cup, “A lot of teams that might be underdogs in these games have come out on top.”

Can you translate this text into French and Spanish and German?

ChatGPT wrote:Sure, here are the translations:

French:

> Les États-Unis ont atteint les attentes lors de la phase de groupes. Maintenant, ils espèrent les dépasser.

> Les Américains affrontent les Pays-Bas en tant que sous-estimés samedi. Mais comme l'a dit le capitaine des États-Unis, Tyler Adams, à propos de cette Coupe du monde, "beaucoup d'équipes considérées comme des sous-estimées dans ces matchs ont réussi à sortir vainqueur".

Spanish:

> Los Estados Unidos cumplieron con las expectativas en la fase de grupos. Ahora, esperan superarlas.

> Los estadounidenses enfrentan a los holandeses como perdedores el sábado. Pero como dijo el capitán de Estados Unidos, Tyler Adams, sobre este mundial, "muchos equipos que podrían ser perdedores en estos juegos han salido victoriosos".

German:

> Die USA haben im Gruppenphase die Erwartungen erfüllt. Jetzt hoffen sie, sie zu übertreffen.

> Die Amerikaner treffen am Samstag als Underdogs auf die Niederlande. Aber wie der US-Kapitän Tyler Adams über diese Weltmeisterschaft sagte: "Viele Mannschaften, die in diesen Spielen als Underdogs gelten, haben sich als Sieger hervorgetan".

The French translation is adequate, certainly competitive with where Google Translate was a couple of years ago. But GPT does far more things than translate (and answer Bac questions). It will even answer grammar questions:

emk wrote:When do I use the subjunctive in French? Please include examples, with explanations.

ChatGPT wrote:The subjunctive mood is used in French to express various states of unreality such as doubt, wish, possibility, necessity, and judgment. It is typically used in subordinate clauses that begin with the word "que" or the relative pronoun "qui". Here are some examples of the subjunctive in French, along with explanations of why the subjunctive is used in each sentence:

  • Il est important que tu sois à l'heure. (It's important that you be on time.) In this sentence, the subjunctive is used to express a necessity or requirement.
  • J'aimerais que tu viennes avec moi. (I would like for you to come with me.) In this sentence, the subjunctive is used to express a wish or desire.
  • Je doute qu'il soit capable de le faire. (I doubt that he is capable of doing it.) In this sentence, the subjunctive is used to express doubt or disbelief.
  • Il est possible que nous ayons un problème. (It's possible that we have a problem.) In this sentence, the subjunctive is used to express a possibility.
  • Il est évident que tu aies raison. (It's evident that you are right.) In this sentence, the subjunctive is used to express a judgment or opinion.
As you can see, the subjunctive is used in French to express various states of unreality. It is typically used in subordinate clauses that begin with the word "que" or the relative pronoun "qui".

This appears generally correct (I didn't double-check everything). But it's incomplete. So I asked for more examples:

emk wrote:What are some other constructs besides "que" and "qui" that require the subjunctive in French?

ChatGPT wrote:In addition to the word "que" and the relative pronoun "qui", there are several other words and phrases that can trigger the use of the subjunctive in French. Here are some examples:

  • Bien que (although)
  • À condition que (provided that)
  • À moins que (unless)
  • Dès que (as soon as)
  • Jusqu'à ce que (until)
  • Pour que (so that)
  • Quoi que (whatever)
These words and phrases are often followed by a subordinate clause that contains the subjunctive verb form. For example:

  • Bien que nous soyons en retard, nous devons partir maintenant. (Although we are late, we must leave now.)
  • À condition que tu travailles dur, tu réussiras. (Provided that you work hard, you will succeed.)
  • À moins que tu ne viennes avec nous, nous partirons sans toi. (Unless you come with us, we will leave without you.)
  • Dès que tu auras fini, tu pourras partir. (As soon as you have finished, you can leave.)
  • Jusqu'à ce qu'il pleuve, nous resterons dehors. (Until it rains, we will stay outside.)
  • Pour que tu comprennes, je vais expliquer de nouveau. (So that you understand, I will explain again.)
  • Quoi que tu décides, je te soutiendrai. (Whatever you decide, I will support you.)
As you can see, these words and phrases can trigger the use of the subjunctive in French, often in a subordinate clause that follows the main clause.

Let me be honest: I'm impressed. It does make occasional mistakes, including a couple above. But in general, it rarely outright plagiarizes texts that are known to Google. And of course, it's limited to internet conventional wisdom. But its knowledge of internet conventional wisdom is incredibly vast.

You can often coax better answers out of it by giving it a few examples of what you want, and then asking for more examples. Treat it like a hardworking research assistant who needs tasks spelled out clearly, and it will generally succeed.

You can play with it here. I suspect there are some clever ways this could help language learners
19 x

User avatar
Querneus
Blue Belt
Posts: 836
Joined: Thu Dec 01, 2016 5:28 am
Location: Vancouver, Canada
Languages: Speaks: Spanish (N), English
Studying: Latin, French, Mandarin
x 2269

Re: The new ChatGPT is a polyglot, apparently

Postby Querneus » Sat Dec 03, 2022 4:16 pm

I am very impressed by the Latin output. Texts it produces seem to be 90% - 100% grammatically correct (way better than what Google Translate can do when you ask it to translate something into Latin), and generally very decent semantically and pragmatically.
5 x

Online
User avatar
emk
Black Belt - 1st Dan
Posts: 1620
Joined: Sat Jul 18, 2015 12:07 pm
Location: Vermont, USA
Languages: English (N), French (B2+)
Badly neglected "just for fun" languages: Middle Egyptian, Spanish.
Language Log: viewtopic.php?f=15&t=723
x 6324
Contact:

Re: The new ChatGPT is a polyglot, apparently

Postby emk » Sat Dec 03, 2022 4:55 pm

According to the OpenAP website, the same underlying technology also powers DuoLingo's grammar correction feedback:

OpenAPI wrote:Duolingo

Uses GPT-3 to provide French grammar corrections. An internal Duolingo study shows that use of this feature leads to measurably better second language writing skills!

See the attached screenshot from the OpenAI website.

This sounded promising, so I asked their chatbot to correct some obvious errors in French:

emk wrote:There are some errors in this French text. Could you fix them and explain my mistakes?

> J'ai un voiture rouge, et j'aime le conduire.

ChatGPT wrote:Here is the corrected text:

J'ai une voiture rouge, et j'aime la conduire.

In the original text, "voiture" was incorrectly written as "un voiture" instead of "une voiture". This is because "voiture" is a feminine noun in French, so it must be preceded by the indefinite article "une" instead of the masculine article "un". Additionally, "le" was used before "conduire" instead of "la". This is because "conduire" is a verb that takes the direct object "la voiture", which is a feminine noun, so the correct pronoun to use is "la".

It took me two tries to refine my question this time, but this is really impressive. The correction from "le conduire" to "la conduire" is remarkable, because it involves gender agreement separated by a number of words. Earlier versions of GPT and Google Translate really struggle with gender agreement. But ChatGPT not only recognizes the error, it even explains it.

This obviously has implications for anyone giving students take-home exams. ChatGPT can almost certainly pass typical essay exams in multiple subjects and languages. Someone gace it the SAT, and with no particular training or tuning, it received average verbal and math scores.

But at the same time, I could see sneaky ways for language tutors to take advantage of this. For example, you could ask GPT to correct your students' errors. You'd still need to proofread its corrections, and make sure it didn't say anything incorrect. But my guess is that with some additional model training, it could go an excellent job (which is presumably what Duolingo is doing with it).

(I also asked it a bunch of easy programming interview questions. It passed with flying colors and showed me a couple of clever little tricks I hadn't seen before.)
You do not have the required permissions to view the files attached to this post.
8 x

User avatar
tommus
Blue Belt
Posts: 957
Joined: Sat Jul 04, 2015 3:59 pm
Location: Kingston, ON, Canada
Languages: English (N), French (B2), Dutch (B2)
x 1937

Re: The new ChatGPT is a polyglot, apparently

Postby tommus » Sat Dec 03, 2022 5:35 pm

I am also very impressed with ChatGPT. I have been asking it all sorts of questions in Dutch, from geographical questions to quantum mechanics. The answers it is providing sound like they are coming from a very knowledgeable person. And the words and grammar are extremely accurate and impressive. I have also been using OpenAI's Whisper speech-to-text with Dutch, and I can hardly believe how well it performs for STT.

Now our challenge with ChatGPT is to figure out the best ways to use it to improve our target language(s) conversational ability. We need to be very innovative. How about some cleaver ideas, suggestions, approaches and concepts? It looks like ChatGPT is up to the challenge. Are we?
5 x
Dutch: 01 September -> 31 December 2020
Watch 1000 Dutch TV Series Videos : 40 / 1000

Online
User avatar
emk
Black Belt - 1st Dan
Posts: 1620
Joined: Sat Jul 18, 2015 12:07 pm
Location: Vermont, USA
Languages: English (N), French (B2+)
Badly neglected "just for fun" languages: Middle Egyptian, Spanish.
Language Log: viewtopic.php?f=15&t=723
x 6324
Contact:

Re: The new ChatGPT is a polyglot, apparently

Postby emk » Sat Dec 03, 2022 6:20 pm

tommus wrote:The answers it is providing sound like they are coming from a very knowledgeable person.

Just one warning: Language models like GPT have a tendency to "hallucinate". Even when they're making up "facts" out of whole cloth, they will still sound confident and impressive. Someone asked ChatGPT to provide citations for a scientific text, and it wrote out footnotes and a nice little bibliography section. Many of the reseachers in the bibliography section existed. The cited papers did not.

Similarly, if you tell it a French text contains 2 errors, and you'd like an explanation of each, but the text contains no obvious errors, it will happily make a up a fake error in your text and explain it to you in very authoritative terms. Basically, it's like a student who answers every question hoping for partial credit, even if it just guesses some things.

emk wrote:This French sentence contains two grammatical errors. Can you find them and explain them?

> J'ai une voiture rouge, et j'aime la conduire.

ChatGPT wrote:1. The subject of the sentence, "J'ai," is singular, but the verb "aime" is conjugated in the plural form. The correct conjugation would be "j'aime."
2. The direct object "la" is not properly matched with the singular noun "voiture." The correct direct object would be "ma voiture."

This is straight up hallucination. None of those explanations make any sense.

Here's a discussion of hallucination. Note that ChatGPT has been trained to reduce the amount of hallucination, but it hasn't been completely eliminated. You're most like to encounter problems if you lie to the model (like I did above), or if you ask about a really obscure subject, or if you otherwise push it out of its comfort zone.

Again, imagine a certain kind of student taking an exam—GPT is happy to provide correct answers when it can, but when it can't, it may just make something up. Upper level CEFR exam tutors will actually encourage students to change the subject or otherwise redirect questions they don't have the vocabulary to answer. Good examiners are wise to this, but they'll still often give some credit for an adroit, fluent redirection. I did this once on my B2 oral exam, and both of the women proctoring actually chuckled.
10 x

User avatar
Axon
Blue Belt
Posts: 775
Joined: Thu Jun 16, 2016 12:29 am
Location: California
Languages: Native English, in order of comfort: Mandarin, German, Indonesian,
Spanish, French, Russian,
Cantonese, Vietnamese, Polish.
Language Log: viewtopic.php?f=15&t=5086
x 3288

Re: The new ChatGPT is a polyglot, apparently

Postby Axon » Sun Dec 04, 2022 5:01 am

Not being a native speaker of any other language, I don't know how well generated prose reads in anything but English. In my testing, ChatGPT makes a mistake in English about once every 5,000 words, pretty similar to or better than me.

I asked it to write a Polish textbook. Each time it ran out of space, I just reprompted with "You ran out of space. Please continue." and it did. The textbook it came up with was about the quality of a long-form blog post on language learning (I actually used to be paid to write those blog posts, before the machines arrived). It came up with twelve chapters including a final exam. Each chapter included exercises, though there weren't always answers:

Now let's look at some common verbs and how they are conjugated in Polish:
• być (to be): jestem, jesteś, jest, jesteśmy, jesteście, są
• mieć (to have): mam, masz, ma, mamy, macie, mają
• lubić (to like): lubię, lubisz, lubi, lubimy, lubicie, lubią

As you can see, the conjugation of verbs in Polish changes depending on the subject. For example, "jestem" (I am) is the first person singular conjugation of "być", while "są" (they are) is the third person plural conjugation.
Now it's your turn to practice! Use the verbs and conjugations above to complete the following sentences:
1. Ona _________ lubić język polski. (She likes the Polish language.)
2. My _________ mieć pięć książek. (We have five books.)
3. Ty _________ być z Polski. (You are from Poland.)

Answers:
1. Ona lubi język polski.
2. My mamy pięć książek.
3. Ty jesteś z Polski.


In the chapter on cases, the Polish looked correct to me (I don't speak Polish very well) but it was pretty nonsensical in describing which words were in which cases and when to use each one. In general, it seemed like it was thinking up snippets of chapters and definitely didn't include things that would be necessary to guide learners. Also, in copying the style of textbook/blog post writing, it came up with a hilarious sentence:

Numerals in Polish are used to express numbers, and they can be either cardinal (used for counting) or ordinal (used for indicating order). Some common numerals in Polish include:

• jeden (one)
• dwa (two)
• trzy (three)
• cztery (four)
• pięć (five)


---

Each time a new language model comes out, I ask it questions about languages, obviously. For both computer and human languages, ChatGPT gets the specifics wrong more often than not. Ask it to contrast the phonology of two languages, and you'll get completely wrong information that looks perfectly accurate. Ask it to give specific examples to explain a computer science concept, and you'll get the same thing.
7 x

User avatar
Carmody
Black Belt - 1st Dan
Posts: 1747
Joined: Fri Jan 01, 2016 4:00 am
Location: NYC, NY
Languages: English (N)
French (B1)
Language Log: http://tinyurl.com/zot7wrs
x 3395

Re: The new ChatGPT is a polyglot, apparently

Postby Carmody » Tue Dec 06, 2022 11:33 am

0 x

User avatar
Carmody
Black Belt - 1st Dan
Posts: 1747
Joined: Fri Jan 01, 2016 4:00 am
Location: NYC, NY
Languages: English (N)
French (B1)
Language Log: http://tinyurl.com/zot7wrs
x 3395

Re: The new ChatGPT is a polyglot, apparently

Postby Carmody » Tue Dec 06, 2022 2:12 pm

0 x

User avatar
tommus
Blue Belt
Posts: 957
Joined: Sat Jul 04, 2015 3:59 pm
Location: Kingston, ON, Canada
Languages: English (N), French (B2), Dutch (B2)
x 1937

Re: The new ChatGPT is a polyglot, apparently

Postby tommus » Tue Dec 06, 2022 2:27 pm

I have been trying a variety of chats with the ChatGPT for the purpose of generating useful conversational material. The chatbot does make some errors and gets some things wrong. But that doesn't seem to detract from the value of the answers for language learning. I have been using it exclusively in Dutch and, although occasionally it gets the facts wrong, it seems to have a great vocabulary and excellent grammar.

I find the most useful methods are to ask for lists of things with a brief explanation of each. For example, you can ask for a list of popular sports and a brief summary of each. You can ask for a list of popular board games and a brief summary of each. You can ask how you would take public transport from one place to another, which could include bus, train, ferry and aircraft, plus a reference to schedules. You can ask it for a list of cities and what they are famous for. Etc., etc., etc. I now intend to put many of these questions and answers into a text-to-speech app to provide some very useful audio material.

The possibilities are limited only by your imagination. ChatGPT seems to have endless energy to provide the best answers it can. And so far, it has been doing a great job for me.
3 x
Dutch: 01 September -> 31 December 2020
Watch 1000 Dutch TV Series Videos : 40 / 1000

Online
User avatar
emk
Black Belt - 1st Dan
Posts: 1620
Joined: Sat Jul 18, 2015 12:07 pm
Location: Vermont, USA
Languages: English (N), French (B2+)
Badly neglected "just for fun" languages: Middle Egyptian, Spanish.
Language Log: viewtopic.php?f=15&t=723
x 6324
Contact:

Re: The new ChatGPT is a polyglot, apparently

Postby emk » Tue Dec 06, 2022 3:47 pm

tommus wrote:The possibilities are limited only by your imagination. ChatGPT seems to have endless energy to provide the best answers it can. And so far, it has been doing a great job for me.

Oh, I can definitely get good results out of it. But doing so depends on a number of things:

  • If it can't find an answer, it will just make something up. It has a very weak idea of the difference between truth and imagination.
  • It is much stronger in domains where it has been specifically trained. Outside of those domains, you should assume it's an overconfident amateur who read too much Wikipedia and who thinks it's an expert.
  • At the very best, it's an enthusiastic and hardworking junior collaborator who has read most of the internet. Imagine that somebody genetically engineered a Golden Retriever to teach it to speak and read, but not to actually think reliably at a human level. "Ooh, you're doing something! Can I help? Huh? Huh? Huh? I can synthesize Wikipedia articles! I can write you some Python! Fun!"
(Yes, these descriptions are anthropomorphized. This deliberate. I find it easier to get useful work out of ChatGPT if I treat it like it's a weird type of person, and then try to figure out exactly what type of person it would be. Treating it as "just a chat generator" will generate misleading intuitions just as often as treating it as a fully human mind.)

I have tested it in a wide variety of ways in the last few days, including throwing it at selected work problems (with my boss's permission, as an experiment). It absolutely has had plenty of moments that make me say, "Holy shit." But it also falls on its face constantly.

For language learning, I suspect it might be most useful for someone like Benny. Benny always used to focus on speaking early and imperfectly. And if you ask ChatGPT to do something like, "Generate a short dialog in French of a person asking where the nearest metro station is, and a helpful French person giving them directions," it will probably come up with something plausible enough for an early speaker like Benny.

Here you go:

ChatGPT wrote:Personne 1 : Excusez-moi, où est la station de métro la plus proche ?

Personne 2 : La station de métro la plus proche est à environ 10 minutes à pied d'ici. Il suffit de suivre cette rue jusqu'à la prochaine intersection, puis de tourner à gauche et vous verrez la station sur votre droite.

Personne 1 : Merci beaucoup pour votre aide !

Personne 2 : De rien, c'était un plaisir de vous aider. Bonne journée !

This is quite a reasonable dialog!

Also, I love the way that ChatGPT loves stories with happy endings full of people being reasonable. You can ask it to tell darker stories, but it definitely leans towards things with nice, tidy, positive morals.
7 x


Return to “General Language Discussion”

Who is online

Users browsing this forum: guyome and 2 guests