substudy: Make Anki cards and other resources from video & bilingual subtitles (command-line)

All about language programs, courses, websites and other learning resources
User avatar
emk
Black Belt - 1st Dan
Posts: 1620
Joined: Sat Jul 18, 2015 12:07 pm
Location: Vermont, USA
Languages: English (N), French (B2+)
Badly neglected "just for fun" languages: Middle Egyptian, Spanish.
Language Log: viewtopic.php?f=15&t=723
x 6319
Contact:

Re: substudy: A tool for making bilingual subtitles (MacOS X or Linux, command-line)

Postby emk » Sun Feb 12, 2017 3:33 pm

One more tiny piece is working! Using the sample sub/idx files so generously provided by Stefan:

Stefan wrote:I believe I've got .sub/.idx for 10 movies if it would be of interest?

…I was able to write a working sub/idx image decoder library in Rust. Sample output:

example.0000.png

This was harder than I expected, because the 'sub' files are technically MPEG 2 Program Streams containing Packetized Elementary Stream packets containing split up pieces of subtitle information packets, which in turn contain run-length-encoded image data and a series of control sequences. Decoding all of this proved rather fiddly. ;)

If you're a Rust programmer, you can find documentation and example code at docs.rs.

This is the first step towards improved subtitle OCR: I can now parse the images into a standard format. The next challenge is breaking the image up into individual letters. This is tricky because:

  1. The "r" and the "i" are attached by the black shadow, but should be separate letters.
  2. The stem and the dot of the "i" are attached by the black shadow but should be treated as a single letter. And indeed, they might not always be attached.
  3. Short spaces indicate letter breaks, long spaces indicate word breaks.
  4. Subtitles may extend across more than one line.
So I'll be working on that. And yes, this is a terrible use of my language learning time, because in the time it takes me to figure all this out, I could just go learn a lot more Spanish. But it's an interesting technical challenge.
You do not have the required permissions to view the files attached to this post.
4 x

User avatar
emk
Black Belt - 1st Dan
Posts: 1620
Joined: Sat Jul 18, 2015 12:07 pm
Location: Vermont, USA
Languages: English (N), French (B2+)
Badly neglected "just for fun" languages: Middle Egyptian, Spanish.
Language Log: viewtopic.php?f=15&t=723
x 6319
Contact:

Re: substudy: A tool for making bilingual subtitles (MacOS X or Linux, command-line)

Postby emk » Mon Feb 13, 2017 1:52 pm

I've just released a "vobsub2png" tool that converts sub/idx files into PNG files with JSON metadata, for people who want to experiment with image-based subtitles.

For more information on this, and on other lower-level subtitle utilities, please see the thread Rust subtitle utilities project. That's where I'll announce low-level tools that may eventually become part of a future version of substudy.
1 x

kaegi
White Belt
Posts: 13
Joined: Tue Dec 27, 2016 7:43 pm
Languages: German (N), English (C2), Japanese (beginner)
x 12

Re: substudy: A tool for making bilingual subtitles (MacOS X or Linux, command-line)

Postby kaegi » Fri Feb 17, 2017 7:29 pm

I'm now done with aligner, which is the tool I promised. The API is very easy and can hopefully be integrated in substudy without any problems.

One thing though: I strongly prefer copylefted licenses for my personal projects, so this project is published under the AGPL-3.0 and will stay that way. I'm sorry for that inconvenience.

You seem to like technical challenges, so the "Algorihm" section might be interesting for you (dynamic programming).
1 x

User avatar
emk
Black Belt - 1st Dan
Posts: 1620
Joined: Sat Jul 18, 2015 12:07 pm
Location: Vermont, USA
Languages: English (N), French (B2+)
Badly neglected "just for fun" languages: Middle Egyptian, Spanish.
Language Log: viewtopic.php?f=15&t=723
x 6319
Contact:

Re: substudy: A tool for making bilingual subtitles (MacOS X or Linux, command-line)

Postby emk » Fri Feb 17, 2017 10:17 pm

(Attention conservation notice: Way too many words about software licensing arcana. You can safely skip this.)

kaegi wrote:I'm now done with aligner, which is the tool I promised. The API is very easy and can hopefully be integrated in substudy without any problems.

Thank you for building this tool and releasing it as free software!

kaegi wrote:One thing though: I strongly prefer copylefted licenses for my personal projects, so this project is published under the AGPL-3.0 and will stay that way. I'm sorry for that inconvenience.

I'm quite sympathetic to the goals of the free software movement. I've been a paying member of the Free Software Foundation on and off over the years, I've personally released over a hundred libraries and programs under free licenses, and I've helped previous employers release literally millions of dollars of code under the GPL and LGPL. I've even helped one company who was accidentally in violation of the GPL (thanks to a third-party vendor) negotiate with the FSF to fix the problem and come into compliance.

So I strongly support your right to license your code as you wish, and I thank you for choosing a free license. The world needs more free software!

But unfortunately, I'm not going to incorporate AGPLed code in substudy. Let me explain why. The AGPL raises particular issues that the GPL and LGPL do not, thanks to this key clause:

Notwithstanding any other provision of this License, if you modify the Program, your modified version must prominently offer all users interacting with it remotely through a computer network (if your version supports such interaction) an opportunity to receive the Corresponding Source of your version by providing access to the Corresponding Source from a network server at no charge, through some standard or customary means of facilitating copying of software.

The big question here is what does "interacting" mean in this context? Obviously if somebody were to create a web application which directly uses your library to align subtitles, then using that application through a web browser would qualify as "interacting", and they would also need to license their application under the AGPL. But what if a web application calls substudy as a command-line tool? What if I package up substudy as a REST API and a web application calls that? In the case of the GPL and LGPL, there are well-understood answers to these questions, based on court precedents, industry practice and the definition of "derivative work" in copyright law. But the AGPL is more of an unknown quantity. It's used by only two major, popular projects that I'm aware of (MongoDB and iText), and one of those projects (MongoDB) was forced to include an explicit clarification:

To make the above practical, we promise that your client application which uses the database is a separate work. To facilitate this, the mongodb.org supported drivers (the part you link with your application) are released under Apache license, which is copyleft free.

If I chose to use your library in substudy, I would need to figure out whether or not I needed a similar license clarification from you. But there's also the issue of relicensing all of my own code. Let's look at the license text (emphasis added):

You may convey a work based on the Program, or the modifications to produce it from the Program, in the form of source code under the terms of section 4, provided that you also meet all of these conditions:

[list]
[*]...
[*]c) You must license the entire work, as a whole, under this License to anyone who comes into possession of a copy. This License will therefore apply, along with any applicable section 7 additional terms, to the whole of the work, and all its parts, regardless of how they are packaged. This License gives no permission to license the work in any other way, but it does not invalidate such permission if you have separately received it.

So if I were to use your AGPLed library in substudy, I would need to relicense all of my own substudy code under the AGPL, as well as any future GUI apps or websites that I build to make substudy easier to use.

Personally, this is all too much work for me. I certainly could have chosen to use the GPL for substudy, which is clearer and more popular than the AGPL. But honestly, if somebody chooses to use substudy code in a proprietary software product, I'm not actually going to call up the copyright lawyer I usually work with and file a copyright suit. (For something bigger than substudy I might feel differently. If I actually use the GPL for my code, I have every intention of enforcing it.) But for substudy, I simply chose to eliminate all of these legal complications and dedicated my code to the public domain, with a fallback license for countries like Germany that don't allow public domain dedications. So substudy is as legally unrestricted as I can possibly make it. Honestly, the intent here is pretty much the same as that behind the hilarious WTFPL.

So anyway, that's why AGPLed licensed code is off-limits for inclusion in substudy. Sorry about that, but thank you for releasing your program!
1 x

kaegi
White Belt
Posts: 13
Joined: Tue Dec 27, 2016 7:43 pm
Languages: German (N), English (C2), Japanese (beginner)
x 12

Re: substudy: A tool for making bilingual subtitles (MacOS X or Linux, command-line)

Postby kaegi » Fri Feb 17, 2017 10:49 pm

Well, I knew this would hurt :(

I'm quite aware of the consequences the AGPL-3.0 has. I've spent a great deal of 2,5 months creating this piece of software in my free time so that any person who wants the same thing as me (correcting subtitles to learn or just enjoy a show in a different language) can freely do so. But if anyone would go and make money out of it (or use it in proprietary programs which I try to avoid), it would feel like "stealing" and taking advantage of all that time I invested.

Circumventing the limitations of the GPL is very easy due to the small surface to the outside world (be that the API or the command line arguments): one can create a "server" which simply calls the command line program and then the "client" is not required to implement the GPL.

I will try to sort my thoughts, and maybe I will release it under a non AGPL-license (someday)... but I'm not making any promises.
0 x

User avatar
emk
Black Belt - 1st Dan
Posts: 1620
Joined: Sat Jul 18, 2015 12:07 pm
Location: Vermont, USA
Languages: English (N), French (B2+)
Badly neglected "just for fun" languages: Middle Egyptian, Spanish.
Language Log: viewtopic.php?f=15&t=723
x 6319
Contact:

Re: substudy: A tool for making bilingual subtitles (MacOS X or Linux, command-line)

Postby emk » Fri Feb 17, 2017 11:59 pm

kaegi wrote:Well, I knew this would hurt :(

Yeah, sorry about that, and my apologies. :-( I do strongly encourage you to license your program under whatever license works best for you, and I won't ask you to do otherwise. Personally, I find it deeply annoying when people badger free software authors to change their licenses.

But the flip side is this is that I'm not going to make a major change to substudy's license, either (as using an AGPLed library would require).

kaegi wrote:But if anyone would go and make money out of it (or use it in proprietary programs which I try to avoid), it would feel like "stealing" and taking advantage of all that time I invested.

That is a perfectly reasonable way to feel! But it's worth keeping in mind that none of the copyleft licenses (LGPL, GPL or AGL) prohibit other people from making money off of your software. These licenses only require that if other people do distribute your software, that they're forced to do so on the same terms that you distributed the original: they must allow other people to copy and modify it. But they're perfectly welcome to charge money for distribution or support or whatever, as the FSF explains here (you've probably already seen this).

kaegi wrote:Circumventing the limitations of the GPL is very easy due to the small surface to the outside world (be that the API or the command line arguments): one can create a "server" which simply calls the command line program and then the "client" is not required to implement the GPL.

Honestly, I wouldn't count on the AGPL providing very much protection in this case. The fundamental problem is that the AGPL is a copyright license, not an EULA, and as such, it can only reliably affect derivative works. And under US copyright law, if process A communicates with process B via a "small surface", then process A is unlikely to be considered a derivative work of process B. You can't make your license automatically extend to other people's code unless a enough mixing occurs to legally qualify as a derivative work, basically.

So if somebody wants to be a real jerk, they can just go ahead and do exactly what you're describing, and there's a small (but non-zero!) chance that a court will force them to share the source code for process A. They're more likely to be forced to share the source code for process B because of the AGPL, depending on the facts of the case. But on the other hand, somebody (like me) who doesn't want to be a jerk, will instead be hyper-cautious and stay well out of the grey area. So in this case, you might only protect yourself against people who already respect your wishes as an author. Still, that might be sufficient in your case.

kaegi wrote:I will try to sort my thoughts, and maybe I will release it under a non AGPL-license (someday)... but I'm not making any promises.

No problems! If you ever release a version under the LGPL or GPL, I'd definitely be willing to consider making it an optional compile-time feature in substudy. But that's totally up to you, and there's no rush.

Also, do feel free to check out my vobsub crate for Rust. It can extract timing information and PNGs from just the *.sub file. With a little bit more work, it could be modified to extract subtitle timing from VOB and Matroska video files. This might allow you extract alignment data directly from the video files without requiring users to extract sub/idx files manually.
0 x

kaegi
White Belt
Posts: 13
Joined: Tue Dec 27, 2016 7:43 pm
Languages: German (N), English (C2), Japanese (beginner)
x 12

Re: substudy: A tool for making bilingual subtitles (MacOS X or Linux, command-line)

Postby kaegi » Sat Feb 18, 2017 8:26 am

And under US copyright law, if process A communicates with process B via a "small surface", then process A is unlikely to be considered a derivative work of process B. You can't make your license automatically extend to other people's code unless a enough mixing occurs to legally qualify as a derivative work, basically.

So if somebody wants to be a real jerk, they can just go ahead and do exactly what you're describing, and there's a small (but non-zero!) chance that a court will force them to share the source code for process A.


That is a pretty convincing argument. I've though about the licensing issue and have re-licensed aligner under the GPL-3.0. I think the gains for the users of substudy are worth the cost (somebody who tries to be a jerk and even might gets through with it). The GPL-3.0 gets my intention across just as well.

Also, do feel free to check out my vobsub crate for Rust. It can extract timing information and PNGs from just the *.sub file. With a little bit more work, it could be modified to extract subtitle timing from VOB and Matroska video files. This might allow you extract alignment data directly from the video files without requiring users to extract sub/idx files manually.


This sounds very interesting (both the .sub and .mkv part). I will see when I get to implement that. I also had the idea to use ffmpeg to automatically extract the subtitles but decided against such a heavyweight dependency (the binary is supposed to be the smallest usable wrapper around the library). But if there ever is a small standalone Rust library that does exactly that, it would be very helpful!
1 x

User avatar
emk
Black Belt - 1st Dan
Posts: 1620
Joined: Sat Jul 18, 2015 12:07 pm
Location: Vermont, USA
Languages: English (N), French (B2+)
Badly neglected "just for fun" languages: Middle Egyptian, Spanish.
Language Log: viewtopic.php?f=15&t=723
x 6319
Contact:

substudy 0.4.1 is out!

Postby emk » Sat Mar 18, 2017 8:48 pm

I've released a new version of substudy! Substudy can:

  1. Help you convert movies or TV shows into Anki decks!
  2. Extract all the dialog from a movie and turn it into an MP3 playlist.
  3. Convert a movie into a web page that allows you to quickly browse through all the dialog in a movie.
  4. Create bilingual subtitle files.
For screenshots and instructions, see the substudy home page. For installation instructions, see the GitHub page.

What's new in this version:

  • Substudy should now build more easily on modern systems, especially Macs.
  • It's possible (though not especially easy) to get substudy to work on Windows.
  • Substudy now has more extensive tests which verify that it works on all supported platforms. :-)
Note that there are no new features in this release. If substudy already works great for you, you don't need to upgrade. This release is basically a "placeholder" release while I work on more advanced subtitle tools.
5 x

User avatar
emk
Black Belt - 1st Dan
Posts: 1620
Joined: Sat Jul 18, 2015 12:07 pm
Location: Vermont, USA
Languages: English (N), French (B2+)
Badly neglected "just for fun" languages: Middle Egyptian, Spanish.
Language Log: viewtopic.php?f=15&t=723
x 6319
Contact:

substudy 0.4.4

Postby emk » Thu Nov 23, 2017 7:50 pm

I just spent my Thanksgiving morning updating substudy to the latest libraries and cleaning up some old code. The biggest user-visible improvement in this release is much better error messages.

I'm strongly considering making some major improvements to substudy. Possibilities include, in no particular order, and with no promises that any of these will happen:

  • Pre-built binaries for Mac, Windows and Linux.
  • A basic GUI so that you don't need to use the command-line.
  • A GUI for watching videos and capturing only specific cards.
  • Tools for making cloze cards easily.
  • An Anki plugin that makes it much easier to import the cards generated by substudy.
If any of this interests you, please let me know.

(Also, I've revived my old Spanish deck, after a very long period of neglect, and it's actually going very nicely, especially since many of these cards are years past their due dates...)
3 x

User avatar
emk
Black Belt - 1st Dan
Posts: 1620
Joined: Sat Jul 18, 2015 12:07 pm
Location: Vermont, USA
Languages: English (N), French (B2+)
Badly neglected "just for fun" languages: Middle Egyptian, Spanish.
Language Log: viewtopic.php?f=15&t=723
x 6319
Contact:

Help needed to test binary releases!

Postby emk » Fri Nov 24, 2017 1:03 pm

I'm working hard to make substudy more user-friendly, but it's a gradual process. But if you use Mac, Windows or Linux, and you're reasonably familiar with the command-line, I could use your help!

I've just made the first official binary releases for substudy. You can find everything you need here:

You'll need to install ffmpeg first (as described in the instructions), and put all the binaries somewhere on your path. If you can get this to work, I'd love to hear about it. And if it fails terribly, I'd love to hear about that, too.

Also, in order to make cross-platform binary versions, I had to replace the encoding detector (uchardet) and the language detector (cld2) with pure Rust versions. So if you see any issues with encoding detection or language detection, please let me know immediately. Thank you!

As always, a huge thank you to people for trying out substudy, and for letting me know what works and what doesn't. I couldn't do this without you!
0 x


Return to “Language Programs and Resources”

Who is online

Users browsing this forum: No registered users and 2 guests