2 weeks to build a language app (a music-style language trainer)

All about language programs, courses, websites and other learning resources
User avatar
Adrianslont
Blue Belt
Posts: 827
Joined: Sun Aug 16, 2015 10:39 am
Location: Australia
Languages: English (N), Learning Indonesian and French
x 1936

Re: 2 weeks to build a language app (a music-style language trainer)

Postby Adrianslont » Thu Sep 03, 2020 9:28 pm

I’m sorry if I’ve been a bit inconsistent talking about typical length of utterances.

The example of “If only | I’d | known” was inherited from that explanation of backchaining I found and the ordering lunch example was my own. Ideally, the app would handle both kinds of scenario (single clauses and multiple clauses) - but as someone mentioned earlier speech runs together and it’s not always so easy to split where you want to.

I’ve done some manual splitting speech in audacity and readjusting subs in subtitleedit and using Speater - and looking at wave forms while doing so, and trying to find better places to split or merge an utterance. and it’s variable. Sometimes it’s easy and sometimes not really possible to split where you want because the speech is so run together. It just sounds rubbish. I am probably not telling you anything you don’t know.

Also, as far as automatic splitting goes, have you had a look at Speater? It has defaults which are pretty good and you can adjust them. There might be some development ideas for you there - or you might already be on top of that.
2 x

User avatar
ryanheise
Green Belt
Posts: 459
Joined: Tue Jun 04, 2019 3:13 pm
Location: Australia
Languages: English (N), Japanese (beginner)
x 1681
Contact:

Re: 2 weeks to build a language app (a music-style language trainer)

Postby ryanheise » Fri Sep 04, 2020 5:23 pm

Day 6: I spent today thinking about how reviews should work, so 1 more day behind :-) My thoughts regarding treating the whole audio file like a binary tree have changed slightly. Rather than breaking down the file into branches and sub-branches, looking from the top down, I instead want to make everything revolve around the review unit. The user should not enter the app looking at the whole file, they should be looking at a sliding window of units. (As alternative terminology to "unit" I could also call it a "section"). The other thing I spent today thinking about was how to manage multiple training routines in the app, because I imagined myself wanting to have a memorisation routine and a listening routine that I want to switch between.

(By the way, I really appreciate the feedback above. I didn't get a chance to reply today as I was busy coding and trying to meet my self imposed deadline. But perhaps tomorrow I will have some time to take a break from the coding and reply to these.)
2 x

User avatar
ryanheise
Green Belt
Posts: 459
Joined: Tue Jun 04, 2019 3:13 pm
Location: Australia
Languages: English (N), Japanese (beginner)
x 1681
Contact:

Re: 2 weeks to build a language app (a music-style language trainer)

Postby ryanheise » Sat Sep 05, 2020 11:07 am

Adrianslont wrote:I’ve done some manual splitting speech in audacity and readjusting subs in subtitleedit and using Speater - and looking at wave forms while doing so, and trying to find better places to split or merge an utterance. and it’s variable. Sometimes it’s easy and sometimes not really possible to split where you want because the speech is so run together. It just sounds rubbish. I am probably not telling you anything you don’t know.


I know exactly what you mean. For example, if there's a diphthong followed by a fricative, and you want to split it so that you hear the complete diphthong in the first segment but you don't want to hear any of the fricative that starts the next segment. However, because speech runs together, the fricative may come in early making it difficult to capture a pure sound for the diphthong.

Also, as far as automatic splitting goes, have you had a look at Speater? It has defaults which are pretty good and you can adjust them. There might be some development ideas for you there - or you might already be on top of that.


I haven't used it, and I think to some extent I would prefer not to look at any other apps so that I can start with a fresh perspective on what I think would be useful to me. But of course in terms of the splitting defaults, it could be interesting to know what adjustable options there are.

I spent some time in the past trying to research better ways of splitting audio and I think I came up with a pretty good algorithm, although it is expensive to run and therefore in this app I'll be doing something rather simple: analysing the energy of the samples and splitting where the energy drops, guided by the user's preferred segment length.

(By the way, I originally wrote this reply using the word "clip" everywhere, but I am slowly learning to say "segment" instead ;-) )
2 x

User avatar
ryanheise
Green Belt
Posts: 459
Joined: Tue Jun 04, 2019 3:13 pm
Location: Australia
Languages: English (N), Japanese (beginner)
x 1681
Contact:

Re: 2 weeks to build a language app (a music-style language trainer)

Postby ryanheise » Sat Sep 05, 2020 5:43 pm

Day 7 update: The Google Play Store link has now been activated which means that people who have PM'd me for the invite can now finally download the app! Prepare to be underwhelmed :-) The version that's published right now was my progress a couple of days after starting the project. I haven't been publishing new releases since then because I knew the store link wasn't active anyway, but now that I know it's active, I plan to publish a new release just as soon as I get the code back in working order.

Today I was supposed to be working on the audio analyser, but instead visited family and ended up doing very little coding. Although I did add a language selection screen. What I've decided on in terms of the database is that each audio file will have one associated language. You could argue that some audio files have mixed languages, but that's a complication I don't want to deal with. In addition to the audio language, I will allow each audio file to have more than one transcript. So for Japanese, you could have a transcript all written in romaji, and another transcript written in proper Japanese. I guess for Chinese you might have simplified and traditional options, and maybe there are other languages that can be written in multiple scripts.
2 x

User avatar
ryanheise
Green Belt
Posts: 459
Joined: Tue Jun 04, 2019 3:13 pm
Location: Australia
Languages: English (N), Japanese (beginner)
x 1681
Contact:

Re: 2 weeks to build a language app (a music-style language trainer)

Postby ryanheise » Sun Sep 06, 2020 5:29 pm

Day 8 screenshot (beauty is on the inside)

Image

What's going on here is that I have an algorithm that takes a list of segments and constructs a unit tree, which is then being fed to the UI. The final piece of the puzzle is to generate a proper list of segments by analysing the audio. So we're almost there, despite appearances. (3,000 lines of code.)
2 x

User avatar
ryanheise
Green Belt
Posts: 459
Joined: Tue Jun 04, 2019 3:13 pm
Location: Australia
Languages: English (N), Japanese (beginner)
x 1681
Contact:

Re: 2 weeks to build a language app (a music-style language trainer)

Postby ryanheise » Mon Sep 07, 2020 2:41 pm

Day 9 screenshots

Image

If you click on "Prepare next section" it automatically identifies the next section of audio for you to study and splits it into segments, shown in the right screen. The intention is to allow you to check and optionally edit the splits (although I don't have controls for that yet). Note that I'm now using the terminology "section" instead of "unit".

Once you've prepared the next section, it gets added to your training set, and you then click on "Begin training session", plug your headphones in, and go on your walk (or whatever else) while the app plays the segments in some programmed sequence as per your preferences. I'll probably add some better visualisations to make this whole process more intuitive.

One way of using the app is that each day, you could study one new section of audio, but you will also be reviewing sections from past days according to an algorithm (which I think I have discussed previously). Depending on how many repetitions you have configured in your training routine, and how much time you have available to study, you may choose to learn more than one new section each day and that is also fine. Each section will have its own maturity and the algorithm should cope with any number of sections maturing on the same day.
2 x

Flickserve
Orange Belt
Posts: 155
Joined: Thu May 17, 2018 10:08 pm
Languages: *
x 198

Re: 2 weeks to build a language app (a music-style language trainer)

Postby Flickserve » Mon Sep 07, 2020 11:57 pm

ryanheise wrote:OK, a question. What import options would people find most useful?

- Open file on device
- Open file from phone's media library
- Transfer file from computer
- Download file from webpage

If I were to implement ONE of these, which one would be the most useful?



At the moment, I prefer computer due to the large screen. I find the phone screen too small to work comfortably. A tablet would be easier I guess.

For example, I create anki cards with audio from subtitles and mp3 on my PC and then transfer the finished cards to my phone which is easy to review when I am out and about.

I use workaudiobook app on PC (not on phone) to easily subtitle clips of audio, then use the srt file it generates to split the audio into individual sentences which get turned into cards.

The weakness in workaudiobook and anki is it doesn't let you link up a few sentences for context. Workaudiobook also doesn't easily let you add a second subtitle for translation.
0 x

User avatar
ryanheise
Green Belt
Posts: 459
Joined: Tue Jun 04, 2019 3:13 pm
Location: Australia
Languages: English (N), Japanese (beginner)
x 1681
Contact:

Re: 2 weeks to build a language app (a music-style language trainer)

Postby ryanheise » Tue Sep 08, 2020 5:40 am

Flickserve wrote:
ryanheise wrote:OK, a question. What import options would people find most useful?

- Open file on device
- Open file from phone's media library
- Transfer file from computer
- Download file from webpage

If I were to implement ONE of these, which one would be the most useful?



At the moment, I prefer computer due to the large screen. I find the phone screen too small to work comfortably. A tablet would be easier I guess.

For example, I create anki cards with audio from subtitles and mp3 on my PC and then transfer the finished cards to my phone which is easy to review when I am out and about.


Thanks for that feedback, that's also my preference. Although I have started with a file import and cloud import feature first, so this will end up being something that comes later.

One thing about transferring from a computer is that there is not a consistent way to do this for iOS and Android. Let's see if the existing import options I provide are good enough for iPhone users, and if so, I'll just implement an import-from-computer options specifically for Android.

I use workaudiobook app on PC (not on phone) to easily subtitle clips of audio, then use the srt file it generates to split the audio into individual sentences which get turned into cards.

The weakness in workaudiobook and anki is it doesn't let you link up a few sentences for context. Workaudiobook also doesn't easily let you add a second subtitle for translation.


I'm glad you said that because those are exactly the two features I'm also interested in! Of course it's not necessarily desirable to build a single app without any weaknesses. The app I'm in the process of building will have its own concept with its own purpose in mind, and I'll avoid adding in anything to make it work more like other apps. That way, it will be possible to make the app not only more limited, but also more focused / useful / intuitive / convenient for its intended purpose.

P.S. If you are on Android, feel free to PM me to try it out. Bits of functionality are starting to work, so I'll try to do another alpha release soonish.
1 x

User avatar
ryanheise
Green Belt
Posts: 459
Joined: Tue Jun 04, 2019 3:13 pm
Location: Australia
Languages: English (N), Japanese (beginner)
x 1681
Contact:

Re: 2 weeks to build a language app (a music-style language trainer)

Postby ryanheise » Wed Sep 09, 2020 5:44 pm

Day 10 and 11: Unfortunately I've been feeling sick these past couple of days and haven't had the energy to do any programming. But I plan to pick things up again tomorrow, and I'm still trying to aim for the 2 week goal. This means I have only 3 more days to create something that I can actually use. So here is my new adjusted schedule:

Day 12: Get the "Begin training session" button working with a simple screen design.
Day 13: Automatically cut audio into segments
Day 14: Segment editor

I will try to do an alpha release on Day 12, 13, and 14 (for Android) and then after Day 14, I will do the audio analyser for iOS, and then try to get it approved by Apple for testing on iOS.

On an unrelated note, the file format will be similar to what Cainntear proposed, but with room for multiple translations and transcripts.

Code: Select all

{
   "version": 1,
   "title": "Peppa Pig",
   "duration": 192873421, // Us
   "language": "ja", // ISO 639-*
   "scripts": ["Jpan"], // ISO 15924
   "translationLanguages": ["en"], // ISO 639-*
   "segment": {
      "start": 129837, // Us
      "end": 274723, // Us
      "transcripts": ["..."],
      "translations": ["..."],
      "segments": [
         ...
      ]
   }
}


start/end/transcripts/translations can be omitted from parent segments and they will be automatically inferred from the child segments.
1 x

Cainntear
Black Belt - 3rd Dan
Posts: 3468
Joined: Thu Jul 30, 2015 11:04 am
Location: Scotland
Languages: English(N)
Advanced: French,Spanish, Scottish Gaelic
Intermediate: Italian, Catalan, Corsican
Basic: Welsh
Dabbling: Polish, Russian etc
x 8657
Contact:

Re: 2 weeks to build a language app (a music-style language trainer)

Postby Cainntear » Thu Sep 10, 2020 1:21 pm

ryanheise wrote:On an unrelated note, the file format will be similar to what Cainntear proposed, but with room for multiple translations and transcripts.

Code: Select all

{
   "translationLanguages": ["en"], // ISO 639-*
   "segment": {
      "translations": ["..."],
      ]
   }
}

Personally I'd favour using a dictionary for the translations rather than parallel arrays. That would save having a whole load of null values floating around while a translation is work in progress. It also avoids having to add in all the nulls as soon as a new translation is started, if the format is used as a live datatype rather than just the archival format. (For example, if someone built a Vue app to work off the same data or to browse it online, JSON would obviously be the native format.)
That said, you might have solid architectural reasons for your choice.

start/end/transcripts/translations can be omitted from parent segments and they will be automatically inferred from the child segments.

"Can be" as in "optional", or "can be" as in "will be...?"

I'm thinking about materials specifically designed for learners with a pause for response. In certain use cases, the gaps may be considered part of the data, whereas in others they might not, and in certain situations that will correspond to the levels of grouping. Or maybe that can be obviated by automatic silence insertion in your app.
0 x


Return to “Language Programs and Resources”

Who is online

Users browsing this forum: No registered users and 2 guests