A language learners’ forum

To repeat my question - Are you comfortable with XPath to parse HTML from a Web page?

Your more specific requirement narrows it down a lot. I haven't looked at that particular dictionary, but this is how you would parse a regular WordReference dictionary entry to identify definitions of the headword:


where "term" is the word being searched on, and "meanings" is a variable that collects the definitions.

The HTML in your case is very simple and I don't think you even need to worry about tables. If you right-click on a page element in your browser and select Inspect element (I'm using Firefox at the moment), you can see the HTML for what you are looking at (or ctrl-U to see the whole Web page). The information you want is all contained in pairs of span tags, with and used to distinguish the spans for the original phrase and the translation.

Thanks. I've googled Xpath and I don't even understand the basic description of it - so I wouldn't know where to start! I managed to download every .html file from https://www.collinsdictionary.com/dicti ... an-english, and I've managed to convert the .azw dictionary to a single html file (although it either crashes or the formatting breaks down when I try to open it in Firefox - perhaps the file is to big?). I just don't know where to start, what software to use, etc. Any pointers at preliminary steps would be massively appreciated.

OK. XPath might actually be overkill for this. What browser are you using, and can you use JavaScript, or Python?
With your "sphere" example page loaded in Firefox, if you go to Tools/Web Developer/Scratchpad, paste in the following code, and click "Run," what do you see?

Thank you, that's definitely a step in the right direction - the phrases neatly pop up in a little dialogue box. Unfortunately it loses the formatting, so I'd have to manually separate each Italian phrase from the English translation (my plan was to use a 'split by bold' command in Excel to separate them all at once). So how would I go about doing it as a batch? I have 80,000 html files that I've pulled from the Collins site. I have minimal knowledge of Python or Jave, but if I know which aspects to focus on, I'm more than willing to take the time to learn, because if could figure out a way of getting them in bulk, I'd have my Anki decks sorted for the next 5 years, haha.

Yes, that was meant as just a step or hint - I wasn't planning to write a whole script. You'd need another variable to hold the elements with the class "example translation," so you'd have both languages extracted, and you would probably want to append the pairs to a file instead of using an alert() function. Manually separating them should not be an issue. Beyond that, your script would need to loop through each HTML page at a time, extracting and saving the data before loading the next page. That could be done in either Python or JavaScript (probably Node.js would be best). For Javascript, see Node.js's fs (file system) module for reading local files and directories, and DOMParser() for loading an HTML document into something easy to parse. For Python, see the discussion at http://www.mzan.com/article/19385837-py ... data.shtml, which looks like a similar scraping problem to yours.

I would keep WordReference's copyright notice in mind, though, and their page about licensing this dictionary data. Maybe they would even sell it to you in the format you want.