Problem downloading DLI, checking links (size).

[astromule]

Hi! I've had a problem downloading all courses from https://www.livelingua.com/dli-language-courses.php, using "DownThemAll!", as for some languages the addon interprets the links as repetitions and don't download them. The problem is that sometimes seem are to be identical copies and sometimes only the name is the same. I've downloaded all languages, but I believe some of them were interpreted as "identical" when they actually weren't, for the following languages: arabic, polish, russian, portuguese, thai.

With DownThemAll you can export your links to a simple text file. Is there some way to check the links with the downloaded content? I'll need to check both size and name. The download managers that I've tried (jdownloader, mipony, freedownloadmanager, downthemall) only offer the option to check by name only and they ask for each individual file each time. That means that if you're downloading 800 files they're going to ask you the same question 800 times.

Thanks in advance.

[rdearman]

You could use wget to mirror the entire website down to your harddrive. I think that is what you want, basically download everything regardless and don't prompt.

wget -mkEpnp http://example.org

Because this will hammer the crap out of the website you might want to look at the various wget options for limiting the bandwidth used.

[astromule]

Thanks! I didn't know about wget, so I tried to use it using this guide https://builtvisible.com/download-your- ... with-wget/ I couldn't make it work.
In the past to download full websites I've used HTTrack Website Copier and WebSuction.
But isn't a way to check just for the missing files? DownThemAll, for example, compares what has been downloaded and if it finds it there, it doesn't download it. As I said before, the problem with this is that several files from DLI have the same name but different content. I'd like if possible to not have to download everything again, but as I already have the links, perhaps that would be the easiest option, just to go language by language and create another folder when the files in the DTA list appear in red. That's what I did for Thai, when I detected the problem.

You could use wget to mirror the entire website down to your harddrive. I think that is what you want, basically download everything regardless and don't prompt.

wget -mkEpnp http://example.org

Because this will hammer the crap out of the website you might want to look at the various wget options for limiting the bandwidth used.

[daegga]

If you can easily export all those links that weren't downloaded, then save them in a file one URL per line. Then invoke "wget -x -i <file>". This will make sure that they get downloaded in the same directory structure as on the server, so you can distinguish easily between the languages even if they aren't mentioned in the filename.