2 weeks to build a language app (a music-style language trainer)

All about language programs, courses, websites and other learning resources
User avatar
ryanheise
Green Belt
Posts: 304
Joined: Tue Jun 04, 2019 3:13 pm
Location: Australia
Languages: English (N), Japanese (beginner)
x 953
Contact:

Re: 2 weeks to build a language app (a music-style language trainer)

Postby ryanheise » Tue Oct 13, 2020 4:38 pm



Progress:

- The sound wave changes colour from left to right tracking the current position
- Added a screen to trim the intro and outro

Another thought going on in the back of my mind is that the JSON import format discussed earlier will probably not fit exactly with how the app will work. I think what's needed instead is just markers to identify semantic boundaries, like sections, sentences, and optionally phrases and words, and then let the app use its smarts to internally arrange that into a tree (or trees). This will let the app intelligently rebalance the trees according to semantic information and the user's preferences. For the non-programmer, the tree analogy is that at the base of the tree you have one trunk. That trunk splits into maybe two branches, and then each of those branches may split into two or more branches, and each of those may again split into smaller branches until eventually reaching the leaves. This is exactly what's happening in the app, except that the user experiences things in reverse, starting at the outer leaves and seeing them join together into larger and larger units rather than split apart.

What's left:

- Let the user customise the routine (e.g. number of reps, playback algorithm)
- Probably some more things which I won't realise until I start using/testing it.
4 x

User avatar
ryanheise
Green Belt
Posts: 304
Joined: Tue Jun 04, 2019 3:13 pm
Location: Australia
Languages: English (N), Japanese (beginner)
x 953
Contact:

Re: 2 weeks to build a language app (a music-style language trainer)

Postby ryanheise » Wed Oct 21, 2020 4:34 pm

BIG MILESTONE: I've stopped fixing bugs and adding new features, and actually started using it for language learning for the first time today! Here's a video of it in operation, and below the video I'll explain what's happening and outline my plans from here.



EXPLANATION

First, I'm preparing the next section of audio to add to my training set. This involves splitting the audio into bite-sized chunks that my mind can cope with. As I adjust the start/end points of each segment, the section is rebalanced. The algorithm tries to identify 4 clumps, each with 1-3 segments in them, and this is based on the gaps between the clumps. As you can see in the above video, I was able to influence the algorithm by creating a larger gap between two of the segments, and it then realised that that would be an ideal point to split the clumps. That's what I was going for since I prefer to have 2 segments per clump (although the algorithm can still work with 1-3).

Second, I open the settings and select a playback algorithm suitable for listening practice, and set it to repeat each atomic segment 3 times, then each clump 2 times, and left all the others at their default values.

Third, I began the training session. Because of the algorithm I selected, it first goes through each atomic segment one by one (3 reps each). Then it merges adjacent pairs of segments (i.e. into clumps) and goes back for a second pass (2 reps each). Then it merges those and does a 3rd pass (2 reps each). Then it merges those and does a 4th pass (2 reps each).

After the session completes, the intention is that that's all you do for the day (kind of like Glossika, 10 sentences per day, although you can double it as desired). The next day, you prepare another section (or 2) and begin another training session, but then the past sections will also come back for review using a review algorithm (not shown above). The review algorithm is itself quite interesting. Each section has a "maturity" which increments after each training session. Based on that maturity, the starting merge level of that section is different. So the next level of maturity would be that the section wouldn't start from the atom level, it would skip that and start from the level where the first merge has already taken place, and on the upper end, an extra merge level will be added (i.e. two sections would be merged together). There's a maximum maturity level which will eventually be configurable in the settings. This is basically what's going on in that mysterious tree diagram I posted a few pages back.

PLANS FROM HERE

First of all, I want to do an alpha release of this in its current state because I'd like to see whether what I've done is actually useful to other people and get your feedback on what I could improve. I'll say it was a lot easier when I originally created this algorithm just for myself because I didn't need to worry about making it look pretty, I think this time around since I'm building an actual app, and it's got to be intuitive and stable, I've realised it's a lot more work. But at the end of the day, I'm glad I did it because what my old script couldn't do before was fit in my pocket while I go out for my walks. I'll be interested to hear if anyone has a similar use case. I'll just randomly share another use I've discovered for this app: I bought some waterproof shower speakers a while ago and used them to maximise my immersion opportunities. This morning, I streamed the audio from my app through the shower speakers and that turned out to be great. I had previously been setting my episode of Peppa Pig to play on repeat but I'd always miss a word at the same point and would have to wait 4 minutes for it to come back. This way, I can hear a phrase again immediately before progressing to the longer segments.

Now for the bigger plans: When I first started building this, I had already written some algorithms for myself for different purposes which are outlined in my log. This current app was intended to be something that I hadn't already built, specifically designed to help me with my memorisation experiment. I think pretty soon after I announced my intention to build the app in this thread, the idea started morphing and including ALL of the other ideas that I had. I am not sure at all if this is a good idea because I have struggled to make something coherent. Actually a lot of the delays up front were a result of trying to come up with a data model that addressed all of these different use cases.

I can actually think of multiple apps here, and these don't need to all be built as part of a single app. This is where I'd also like some feedback, although I think I first want to release the alpha for what I've created so far so you can get to "experience" what the app listening experience is like for your own language material, and then you might be able to have a more informed opinion on what i should do next. One thing in particular that might not be suitable for this Language Trainer app is the review system. I love the review system, but for my own use case for this app, I think what I really wanted was the ability to select an area of the file that I wanted to practice and just jump straight into a training session for that area, rather than the review system tell me what area I should study. The review system I've built is instead tied to that concept of "pipelines" that I talked about in an earlier post where you continuously feed new content into the pipeline. Sections increase in maturity and eventually get retired, and replaced by ever-fresh content. I think it has a lot of benefits, but maybe just not for this app.

A SECOND APP?

The above video comes from a podcast called JLPT Stories. I had previously written an algorithm that analyses podcasts to automatically rank and sort their language difficulty. Here's a graph which I produced from that analysis showing the computed relative difficulty of individual episodes from the JLPT podcast:

Image

Peppa Pig is probably the exception to the rule, but I do get most of my listening immersion from podcasts, and so a lot of the personal scripts that I've written tend to revolve around them. The JLPT Stories podcast actually comes with transcripts (as do a number of other Japanese podcasts I listen to), but the cool thing is that I can probably write an algorithm that analyses the audio and automatically figures out which sentences of the transcript go next to which parts of the audio, and could integrate that automatically into the app.

So imagine a sort of podcast player where it automatically divides up the audio into segments, but then it also automatically gives you the transcript, and an integrated dictionary, and then the playback algorithm above could interleave the text-to-speech translation of this transcript with the audio from the podcast. Continuous podcast feeds also solve the problem of needing a constant stream of new content to satisfy the review algorithm, and I think it also makes it easier for the average person to get audio content into the app, because it can be searched, browsed and downloaded or subscribed to.

There are also technical reasons why this might be a good idea. With the Language Trainer app, I'm running into a limit of how much analysis can be done on the device. The sorts of analysis I'm doing on the audio is quite computationally intensive, and to import a 1 hour episode of a podcast may actually take 1 hour of processing time! The beauty of podcasts is that they're public and they have permanent URLs, so I can index them, analyse them once for their timestamps and cache those timestamps on the server. Ideally the moment a new episode comes out, the server will already be pre-processing it and then by the time you want to download it in the app, the timestamps and all other metadata will be available instantly.

There was another app I also wanted to build for my language learning experiments related to language exchange. However, seeing how much time I'm spending on this current app, I think I really need to judiciously choose where I spend my time. It's not easy for me to participate in language exchanges at the moment either, due to the pandemic.

Anyway, in my next update I'll hopefully be able to get the alpha through the Apple store. I expect it could take quite a while for them to approve it on the first submission, though. The Android version will hopefully be approved sooner.
4 x

User avatar
MorkTheFiddle
Brown Belt
Posts: 1161
Joined: Sat Jul 18, 2015 8:59 pm
Location: usa
Languages: English (N). Read (only) French and Spanish. Studying Ancient Greek, aiming for mastery by 2424. Studying a bit of Latin and Japanese. Once studied Old Norse. Dabbled in Catalan, Provençal and Italian.
Language Log: https://forum.language-learners.org/vie ... 11#p133911
x 2047

Re: 2 weeks to build a language app (a music-style language trainer)

Postby MorkTheFiddle » Wed Oct 21, 2020 5:51 pm

A fellow programmer once said, "If it's not pretty, it's not finished." So, yes, go for pretty. You've done a lot of work on this.

Lately I've been gathering together Spanish and French poetry readings from the Internet, which I would like to try out on your app, once you get the beta ready.
1 x
Tu sabes cuando sales pero no sabes cuando regresas.

User avatar
tangleweeds
Green Belt
Posts: 366
Joined: Sat Jul 18, 2015 7:09 pm
Location: Portland, Oregon, USA
Languages: English (N)
beginner: Irish, Norwegian
clearing cobwebs: Japanese
on the shelf: French, Latin
wanderlust: Vietnamese
Language Log: viewtopic.php?t=705
x 886

Re: 2 weeks to build a language app (a music-style language trainer)

Postby tangleweeds » Wed Oct 21, 2020 7:43 pm

That's wonderful, I want it already! (I'm on iOS, so I'll probably have to wait though, right? I've done software testing for pay though, so if you need iOS testing done...)

The only suggestion I might make is that while watching, I really wanted to jump up a level of the tree immediately after having learned a pair/triad of shorter segments, so I could hear the conjoined segments a couple of times while they were fresh in my mind--then hop back down the tree in order move on to the next set of segments. But I have no idea how this might interface with how you've structured movement within your tree.

I would be less likely to use the review algorithm, because I can't even manage steady reviews on Anki. Not a feature request here, just an explanation: I tweak review algorithms so I can overlearn at the beginning then I don't need to things again for a long time, so it's more like decks of flashcards that I can pick up when convenient, but stay sorted by how well I know them.
2 x
Irish & Japanese are on vacation for the 11/2020 6WC
Wanderlust: still Vietnamese

User avatar
ryanheise
Green Belt
Posts: 304
Joined: Tue Jun 04, 2019 3:13 pm
Location: Australia
Languages: English (N), Japanese (beginner)
x 953
Contact:

Re: 2 weeks to build a language app (a music-style language trainer)

Postby ryanheise » Thu Oct 22, 2020 2:10 am

tangleweeds wrote:I would be less likely to use the review algorithm, because I can't even manage steady reviews on Anki. Not a feature request here, just an explanation: I tweak review algorithms so I can overlearn at the beginning then I don't need to things again for a long time, so it's more like decks of flashcards that I can pick up when convenient, but stay sorted by how well I know them.


Listening to the way you like to tweak review algorithms, dare I say that what I had in mind *might* just be the perfect review algorithm for the type of learner that you are :-) As someone myself who generally doesn't like Anki and the obligation to review every day lest your reviews stack up and put you in "review hell", I designed this algorithm for someone (i.e. me) would would rather do overlearning up front and then be done with the reviews forever. Let me explain with a bit of context:

Where I would see myself using this algorithm is learning through stories. My driving motivation is always to get through the story by battling through the new vocabulary and grammar in the next section each day, and through overlearning, have repetitions concentrated within -- and tapering off -- over the first week. Glossika is the main inspiration for this which introduces 10 sentences per day and does a lot of repetitions of those sentences in the first day, then those sentences are reviewed on day 2, but with fewer repetitions, then again on day 3 with fewer repetitions still, then on day 4 and day 5 also. After day 5, you're done learning and you'll never review those sentences again. This opens up your schedule to always take in new material so you are not a slave to your old reviews.

For what I had in mind, though, the difference with Glossika is that rather than learn 10 isolated sentences each day, the sentences actually link together to form a story (which itself gives you more anchor points to help your memory), and then over the course of those 5 days of overlearning, those different parts of the story get "merged" together, and you get to ENJOY the story in its full glory by being able to comfortably listen to long sections of it without your brain struggling to keep up with the unfamiliar vocabulary. After 5 days, there is no need to spend your time reviewing those old parts of the story. You get to keep moving forward hearing new parts of the story and new vocabulary. You will primarily get your reviews over the long term through the natural recurrence of old word and old grammar in NEW contexts and new sentences.

Hopefully that helps to give a clearer picture of what I had in mind, but maybe the only way to really make it clear would be to build it. Fortunately I've done all of the hard work building the algorithm (the app in its current state is therefore closer to this new concept rather than the old one), so what I'll do from this point is make a copy of the code for the other app, inherit all of that review algorithm code, add the remaining features I need to realise this idea to its full potential, and then remove the review code from the original app replacing it by the ability to freely navigate a single file.

But first, I plan to release the current app *as is*, even though it still has the review code in it, just because I want to get it out there. Just keep in mind that when I do delete the review code, you may need to reinstall the app because the internals of the data model will be completely changed.

I look forward to being able to share the alpha with you all.
3 x

User avatar
ryanheise
Green Belt
Posts: 304
Joined: Tue Jun 04, 2019 3:13 pm
Location: Australia
Languages: English (N), Japanese (beginner)
x 953
Contact:

Re: 2 weeks to build a language app (a music-style language trainer)

Postby ryanheise » Thu Oct 22, 2020 2:54 pm

It turns out the name "Language Trainer" is already taken on the App Store, so I've renamed it to Wordwalker.

And it's available! I'm happy with this, because one of the original motivations was to be able to use it while going out on my daily walks.

Now I just need to design a new logo...
2 x

User avatar
MorkTheFiddle
Brown Belt
Posts: 1161
Joined: Sat Jul 18, 2015 8:59 pm
Location: usa
Languages: English (N). Read (only) French and Spanish. Studying Ancient Greek, aiming for mastery by 2424. Studying a bit of Latin and Japanese. Once studied Old Norse. Dabbled in Catalan, Provençal and Italian.
Language Log: https://forum.language-learners.org/vie ... 11#p133911
x 2047

Re: 2 weeks to build a language app (a music-style language trainer)

Postby MorkTheFiddle » Thu Oct 22, 2020 5:44 pm

Wordwalker sounds better to me than Language Trainer, FWIW.
But I am in trouble here, because I have only an iOS gadget (iPod). :cry:
I have a Google Pixel 2 phone, but I discontinued the account and scraped it clean.
1 x
Tu sabes cuando sales pero no sabes cuando regresas.

User avatar
tangleweeds
Green Belt
Posts: 366
Joined: Sat Jul 18, 2015 7:09 pm
Location: Portland, Oregon, USA
Languages: English (N)
beginner: Irish, Norwegian
clearing cobwebs: Japanese
on the shelf: French, Latin
wanderlust: Vietnamese
Language Log: viewtopic.php?t=705
x 886

Re: 2 weeks to build a language app (a music-style language trainer)

Postby tangleweeds » Thu Oct 22, 2020 5:59 pm

MorkTheFiddle wrote: I have only an iOS gadget (iPod). :cry:
See, more iOS demand! :mrgreen: All I have is an iPad & an iPhone, unless you count the Kindle fire, which requires downloading apps from the stupid Amazon app store. I use the iPad for all my home study, and the iPhone only when I'm on the go--it's pretty old, plus I consider it perpetually COVID contaminated so only touch it when I'm already in away-from-home :o !!everything-is-unclean!! :o mode.
2 x
Irish & Japanese are on vacation for the 11/2020 6WC
Wanderlust: still Vietnamese

User avatar
ryanheise
Green Belt
Posts: 304
Joined: Tue Jun 04, 2019 3:13 pm
Location: Australia
Languages: English (N), Japanese (beginner)
x 953
Contact:

Re: 2 weeks to build a language app (a music-style language trainer)

Postby ryanheise » Fri Oct 23, 2020 3:24 am

MorkTheFiddle wrote:Wordwalker sounds better to me than Language Trainer, FWIW.


Thanks! The history of this name actually predates the conception of the app. I think it was 2 years ago, one of my students was building an app called "baby walker", so I had that name as inspiration. Then one day as I was going for a walk through the Sydney Botanic Gardens while listening to Japanese material, the name "word walker" occurred to me as a cool name for a future project. I'm definitely not this creative usually (as is evidenced by the name "language trainer"), but I like that this name encapsulates language and walking, and that it even has a nice alliteration of the W's.

But I am in trouble here, because I have only an iOS gadget (iPod). :cry:
I have a Google Pixel 2 phone, but I discontinued the account and scraped it clean.


My situation is worse, I actually don't have an iPhone or an iPod. I may end up needing to buy one to be able to fully test if it works (I currently use an iPhone simulator), but hopefully everyone will tell me if it doesn't work and what doesn't work. I think that apps for iPhone should also work on those newer iPods that have the screen. But are you saying you don't currently have any sort of usable smartphone? That is quite rare.

In other news, I've spent some time last night and this morning playing around with a new logo design:

Image

It's a bit abstract, but those circles could represent the imprint of a foot (representing walking) or they could also represent ever-growing segments as they get merged together.
2 x

User avatar
MorkTheFiddle
Brown Belt
Posts: 1161
Joined: Sat Jul 18, 2015 8:59 pm
Location: usa
Languages: English (N). Read (only) French and Spanish. Studying Ancient Greek, aiming for mastery by 2424. Studying a bit of Latin and Japanese. Once studied Old Norse. Dabbled in Catalan, Provençal and Italian.
Language Log: https://forum.language-learners.org/vie ... 11#p133911
x 2047

Re: 2 weeks to build a language app (a music-style language trainer)

Postby MorkTheFiddle » Fri Oct 23, 2020 4:58 pm

ryanheise wrote:
MorkTheFiddle wrote:But are you saying you don't currently have any sort of usable smartphone? That is quite rare.
:D Yup, that's what I'm saying. Most iOS apps work on my iPod, but because it's missing its own connection to the internet, it needs a wifi source. It has a lot of apps, but the weather app and the music player are what I use most. I don't do social media, so, bottom line, I don't really need a smartphone.
1 x
Tu sabes cuando sales pero no sabes cuando regresas.


Return to “Language Programs and Resources”

Who is online

Users browsing this forum: No registered users and 1 guest