Investigating a problem, help appreciated

Discuss technical problems and features here
User avatar
emk
Black Belt - 1st Dan
Posts: 1744
Joined: Sat Jul 18, 2015 12:07 pm
Location: Vermont, USA
Languages: English (N), French (B2+)
Badly neglected "just for fun" languages: Middle Egyptian, Spanish.
Language Log: viewtopic.php?f=15&t=723
x 6971

Investigating a problem, help appreciated

Postby emk » Thu Nov 14, 2024 11:45 am

It looks like we might have lost a bunch of data from the last several months. I have no idea what's going on here, and I haven't been touching anything at all lately. :?

We do have automatic backups back to October 30th, plus a manual backup from May. I have made sure the backups from the 30th and several days after don't expire.

There is some discussion here.

If anyone knows when this problem occurred, or if they can describe the extent of what has been lost a bit more precisely, I can look at our more recent backups and see whether we can recover anything. But I could really use some help with this. Thank you to anyone who can provide more information! And my apologies for whatever has happened.
1 x

jeffers
Blue Belt
Posts: 978
Joined: Sat Aug 22, 2015 4:12 pm
Location: UK
Languages: Speaks: English (N), Hindi (A2-B1)

Learning: The above, plus French (A2-B1), German (A1), Ancient Greek (?), Sanskrit (beginner)
Language Log: https://forum.language-learners.org/vie ... 15&t=19785
x 3291
Contact:

Re: Investigating a problem, help appreciated

Postby jeffers » Thu Nov 14, 2024 1:00 pm

I'm pretty sure someone else had the same problem a few weeks ago, but I can't remember who. Also, I remember rdearman replying to that one, but I can't say that for sure. I've had a look through recent posts but haven't managed to find it.

EDIT: I've looked through rdearman's recent posts and can't find it. Either I'm misremembering or the thread has deleted.
0 x
Le mieux est l'ennemi du bien (roughly, the perfect is the enemy of the good)

French SC Books: 0 / 5000 (0/5000 pp)
French SC Films: 0 / 9000 (0/9000 mins)

User avatar
emk
Black Belt - 1st Dan
Posts: 1744
Joined: Sat Jul 18, 2015 12:07 pm
Location: Vermont, USA
Languages: English (N), French (B2+)
Badly neglected "just for fun" languages: Middle Egyptian, Spanish.
Language Log: viewtopic.php?f=15&t=723
x 6971

Re: Investigating a problem, help appreciated

Postby emk » Thu Nov 14, 2024 1:26 pm

Is this just affecting certain accounts? Or have we lost all data from everyone during some window of time?

If someone can help me figure out what we're missing, I can look to see if we can restore it. But without knowing what's gone I can't do much.

Also, all the mods should now have my direct personal contact info.
0 x

User avatar
Iversen
Black Belt - 5th Dan
Posts: 5050
Joined: Sun Jul 19, 2015 7:36 pm
Location: Denmark
Languages: Monolingual travels in Danish, English, German, Dutch, Swedish, French, Portuguese, Spanish, Catalan, Italian, Romanian and (part time) Esperanto
Ahem, not yet: Norwegian, Afrikaans, Platt, Scots, Russian, Serbian, Bulgarian, Albanian, Greek, Latin, Irish, Indonesian and a few more...
Language Log: viewtopic.php?f=15&t=1027
x 15991

Re: Investigating a problem, help appreciated

Postby Iversen » Thu Nov 14, 2024 2:03 pm

It's not all accounts within a certain time frame. I have checked the entries in my own log back to 2018 (using some lists I keep for that thread), and it seems that only the posts marked as deleted have disappeared I could see them when I was logged in as a moderator, but not if I I logged out - and now I can't see them at all. But if there was a hole covering for instance a week then I would have been able to se it. My guess is that only some random accounts were hit, but those accounts were obliterated in their entirety.
1 x

User avatar
emk
Black Belt - 1st Dan
Posts: 1744
Joined: Sat Jul 18, 2015 12:07 pm
Location: Vermont, USA
Languages: English (N), French (B2+)
Badly neglected "just for fun" languages: Middle Egyptian, Spanish.
Language Log: viewtopic.php?f=15&t=723
x 6971

Re: Investigating a problem, help appreciated

Postby emk » Thu Nov 14, 2024 2:36 pm

OK, so the feasible fixes that I could try are:

  1. Restore either the May 1 or the Oct 30 backup to a new database server.
  2. Easyish: Manually dump post data for certain users through the backup date.
  3. Much harder: Try to restore the deleted accounts as if nothing happened.
I am really sorry for this situation, and I wish I had known shortly after the problem occurred. I have given the mods personal contact info for future problems, but that doesn't help with this one.

But unfortunately I am time-limited through the end of the year, which affects how deep I can dig into doing a complex, custom restauration. Just getting the data out is easier.

The other thing that would help a lot is a list of accounts affected, so I knew what to extract.
2 x

kundalini
Orange Belt
Posts: 139
Joined: Sun Jan 24, 2021 8:17 pm
Languages: English (C), Greek (low intermediate)
x 468

Re: Investigating a problem, help appreciated

Postby kundalini » Thu Nov 14, 2024 5:14 pm

emk wrote:The other thing that would help a lot is a list of accounts affected, so I knew what to extract.


This is the earliest mention of a deleted account that I recall seeing:

https://forum.language-learners.org/viewtopic.php?f=15&t=21081
2 x
Iliad: 14 / 24
French SC Books: 2709 / 5000
French SC Films: 6700 / 9000
Spanish SC Books: 340 / 2500
Spanish SC Films: 300 / 4500

User avatar
emk
Black Belt - 1st Dan
Posts: 1744
Joined: Sat Jul 18, 2015 12:07 pm
Location: Vermont, USA
Languages: English (N), French (B2+)
Badly neglected "just for fun" languages: Middle Egyptian, Spanish.
Language Log: viewtopic.php?f=15&t=723
x 6971

Re: Investigating a problem, help appreciated

Postby emk » Thu Nov 14, 2024 6:44 pm

kundalini wrote:This is the earliest mention of a deleted account that I recall seeing:

https://forum.language-learners.org/viewtopic.php?f=15&t=21081

Thank you so much! Unfortunately, it looks like October 8th, which is well after the May 1 backup, but before the oldest remaining daily backup from October 30th. Likely the only way to recover anything in that time window is the Google cache, the Wayback Machine, or maybe a Bing cache if there is one.
0 x

User avatar
handa
Posts: 3
Joined: Sun Oct 13, 2024 12:38 pm
Languages: Swedish (N), English, French, Japanese
x 13

Re: Investigating a problem, help appreciated

Postby handa » Thu Nov 14, 2024 11:45 pm

My original account (same username) was among those that were deleted. I made a post to my log thread at the beginning of September and discovered that I couldn't log into my account or reset the password the first week of October.
1 x

User avatar
Iversen
Black Belt - 5th Dan
Posts: 5050
Joined: Sun Jul 19, 2015 7:36 pm
Location: Denmark
Languages: Monolingual travels in Danish, English, German, Dutch, Swedish, French, Portuguese, Spanish, Catalan, Italian, Romanian and (part time) Esperanto
Ahem, not yet: Norwegian, Afrikaans, Platt, Scots, Russian, Serbian, Bulgarian, Albanian, Greek, Latin, Irish, Indonesian and a few more...
Language Log: viewtopic.php?f=15&t=1027
x 15991

Re: Investigating a problem, help appreciated

Postby Iversen » Fri Nov 15, 2024 7:52 am

handa wrote:My original account (same username) was among those that were deleted. I made a post to my log thread at the beginning of September and discovered that I couldn't log into my account or reset the password the first week of October.


This observation is important, since it pinpoints that the problem occurred in September or during the very first days of October 2024. So there is a gap between an old backup from May and the current database.

The data up to May could possibly be retreived with SQL select where you first make a list of all the accounts in May and and one valid now and run them together (they probably contain hidden number codes). Then those posts in the result that have missing fields from today are the one that have disappeared. Accounts that were made after May, but before October can't be reconstructed with this method. The posts may actually still be there, but just without any member to refer to. In principle posts from rescued member IDs could potentially still be found, but you need to know the structure of the database to see whether that is possible. Actually, we do know that it mainly is the member list that is in trouble - even if all post of certain members had disappeared you would still see an empty member ID if it just was the list of posts that had been hit. The bad news is of course that the accident that hit the database may have hit both tables because of a command that connected them - and then the summer posts would be gone too.

With a list of lost accounts you can make a select on the table of posts and possibly others too (there may be index tables and things like that - plus tables tied to the PM system etc., but they are probably not important. The result should be one table with lost members and their data plus another with posts up to May for those members (or possibly up till now). And then those tables could be inserted into a copy (NOT the running system, please!!) of the current database to see what happens ... maybe the result is a database where the data can be accessed, but there may be additional tables that have something to do with security that prevents users from seeing them. Images may be lost: since they aren't accessible unless you are logged (a major itch BTW seen from my perspective since new users may think the whole system isn't working properly) they must be located somewhere outside the main database, and heaven knows what state those data are in or how they are connected to the posts that reference them.
1 x

User avatar
neofight78
Blue Belt
Posts: 545
Joined: Wed Jul 22, 2015 8:02 pm
Location: Nottingham, UK
Languages: English (N), Russian (B2), Dabbled in Spanish, Serbian, Norwegian
Language Log: viewtopic.php?t=833
x 1245

Re: Investigating a problem, help appreciated

Postby neofight78 » Fri Nov 15, 2024 9:40 am

I don't know about timing, but I noticed my avatar got deleted and I had to reupload it. Don't know if it's connected to this issue or not, but maybe a useful datapoint?
1 x


Return to “Technical Support and Feature Requests”

Who is online

Users browsing this forum: No registered users and 1 guest