# All my posts today have been deleted



## Vlad

I just woke up to a note from our server techs. Last night we suffered a massive database corruption in an InnoDB table that caused a cascading effect and the host needed to roll back to a snapshot from the day before. This is the reason why your posts and PMs all disappeared.

Sorry for the inconvenience, this is as far as I remember the first time we needed to do this in the last 17 years.


----------



## lill_canele

I noticed that all my posts and emoticons that I did today haven’t been deleted.  
Edit: Sorry, no intention on any specific action of deleting. I meant more like disappeared.

Anyone have the same issue?


----------



## jblended

Yep, all my posts and PMs from the last 24 hours have gone.


----------



## jellyv

Yes, me too, but I think it's just an indexing problem. The site's been in maintenance this evening. Let's see how it shakes out.


----------



## Swanky

I promise no one has deleted all of anyone posts or PMs today! I know @Vlad had to work on some unexpected tech stuff.


----------



## bigverne28

lill_canele said:


> I noticed that all my posts and emoticons that I did today haven’t been deleted.
> Edit: Sorry, no intention on any specific action of deleting. I meant more like disappeared.
> 
> Anyone have the same issue?


It happened about an hour ago. I couldn’t post as the server went down for around 20 mins. When I went back in a whole day of posts from everyone on the thread were missing. It’s also across more than one thread so I suspect it’s a wider issue.


----------



## Julie_de

Yes, my private messages disappeared too and the post


----------



## Gwinneth

I thought my computer got hacked or smthing. I guess everyone had same problems


----------



## whateve

Curious to see if they reappear.


----------



## limom

My posts have been deleted too.
What is the deal?
What is the point of having a forum if opinions and posts within the guidelines are deleted?
Yes, it is a privately owned forum but this is very frustrating. 
And confusing as well.


----------



## bigverne28

limom said:


> My posts have been deleted too.
> What is the deal?
> What is the point of having a forum if opinions and posts within the guidelines are deleted?
> Yes, it is a privately owned forum but this is very frustrating.
> And confusing as well.


It's seems like it's an IT error rather than "deleted" posts. All posts for the last 24hrs seem to have disappeared from threads, so I think everyone is affected. Hopefully everything is backed up and can be restored at some point.


----------



## limom

bigverne28 said:


> It's seems like it's an IT error rather than "deleted" posts. All posts for the last 24hrs seem to have disappeared from threads, so I think everyone is affected. Hopefully everything is backed up and can be restored at some point.


Thanks for the input.
Is this because pictures, gifs and the likes take too much space?


----------



## bigverne28

limom said:


> Thanks for the input.
> Is this because pictures, gifs and the likes take too much space?


I don't know the cause but the site was down last night for about 20mins and again for the same period of time at around 4.30am (GMT), this morning.


----------



## snibor

I wouldn’t take anything personally. The site was acting weird and I was having issues. I assumed there was tech stuff going on which happens from time to time. No biggie


----------



## papertiger

As @Swanky mentioned not a mod thing.

I haven't checked on my own posts today, but interestingly, posts had been moved yesterday after reported OT, and then mysteriously they all moved back to their original thread this morning and so were reported again.

I think it's just a glitch.


----------



## Nahreen

I got an error this morning (5 am West European time) trying to access the site (database error). I think some new threads might still be missing. I know Notorios pink had a new thread about the Christies auction that I saw late last night and I can´t find it. I remember it because I had planned to comment in the morning.


----------



## Vlad

I just woke up to a note from our server techs. Last night we suffered a massive database corruption in an InnoDB table that caused a cascading effect and the host needed to roll back to a snapshot from the day before. This is the reason why your posts and PMs all disappeared.

Sorry for the inconvenience, this is as far as I remember the first time we needed to do this in the last 17 years.


----------



## 880

I posted several posts which are no where to be found. I rewrote some which were replies.


----------



## jblended

bigverne28 said:


> It's seems like it's an IT error rather than "deleted" posts. All posts for the last 24hrs seem to have disappeared from threads, so I think everyone is affected. Hopefully everything is backed up and can be restored at some point.


Yes this. Sorry I wasn't clear in my o.p. I didn't mean that mods were deleting anything (they do explain in notifications if they have to remove a post). This is clearly a technical glitch that erased the past 24 hours for everyone.
I wish I'd posted that I robbed a bank because then the evidence would have been erased!


----------



## limom

880 said:


> I posted several posts which are no where to be found. I rewrote some which were replies.


Wow, impressive that you recall your posts


jblended said:


> Yes this. Sorry I wasn't clear in my o.p. I didn't mean that mods were deleting anything (they do explain in notifications if they have to remove a post). This is clearly a technical glitch that erased the past 24 hours for everyone.
> I wish I'd posted that I robbed a bank because then the evidence would have been erased!


The internet never forget!
The IT dudes can find stuff, I can’t even recall at all 
A hot mess


----------



## jblended

limom said:


> The internet never forget!
> The IT dudes can find stuff, I can’t even recall at all


Oh no! My masterplan has been foiled!


----------



## momasaurus

Everything from yesterday (Wednesday) has disappeared.


----------



## Notorious Pink

Nahreen said:


> I got an error this morning (5 am West European time) trying to access the site (database error). I think some new threads might still be missing. I know Notorios pink had a new thread about the Christies auction that I saw late last night and I can´t find it. I remember it because I had planned to comment in the morning.


Thanks for the heads up - I will recreate the thread.


----------



## south-of-france

Thank you Vlad and everyone for doing so much work!


----------



## etoupebirkin

Thanks for letting us know!!!


----------



## leechiyong

Would also like to say thank you to everyone.  May your IT team get some sleep or at the least, massive quantities of coffee.


----------



## Vlad

leechiyong said:


> Would also like to say thank you to everyone.  May your IT team get some sleep or at the least, massive quantities of coffee.



Well, I slept fine through it all (aside from my toddler being up for two hours, concerned over some random _word_ that needed to be shared at 3am). Our hosting company techs did a banging job to get the site back up and running!


----------



## amelrod1

Vlad said:


> I just woke up to a note from our server techs. Last night we suffered a massive database corruption in an InnoDB table that caused a cascading effect and the host needed to roll back to a snapshot from the day before. This is the reason why your posts and PMs all disappeared.
> 
> Sorry for the inconvenience, this is as far as I remember the first time we needed to do this in the last 17 years.


Yup!  I noticed an error message last night.  As an IT person myself who has had to deal with disaster recovery issues, I feel your pain.  Kudos to the Prod Support/Operations team who got it sorted out.


----------



## muchstuff

Thanks @Vlad for letting us know and to the IT team for their hard work!


----------



## LKBNOLA

Vlad said:


> I just woke up to a note from our server techs. Last night we suffered a massive database corruption in an InnoDB table that caused a cascading effect and the host needed to roll back to a snapshot from the day before. This is the reason why your posts and PMs all disappeared.
> 
> Sorry for the inconvenience, this is as far as I remember the first time we needed to do this in the last 17 years.


Thank you for everything you are doing to fix it! Much appreciated. These things happen but you and your team seem to have it sorted out going forward.


----------



## lill_canele

Thank you @Vlad and the team for your updates and your hard work!   

It's okay, much better than when one of my folders at work with an entire year's worth of work was deleted.


----------



## Vlad

I just woke up to a note from our server techs. Last night we suffered a massive database corruption in an InnoDB table that caused a cascading effect and the host needed to roll back to a snapshot from the day before. This is the reason why your posts and PMs all disappeared.

Sorry for the inconvenience, this is as far as I remember the first time we needed to do this in the last 17 years.


----------



## Chanbal

No worries, we will survive. Thanks for letting us know!


----------



## Lellabelle

Good to know, thanks.  Other than data lost to the roll back, are there any other impacts, or is there any need to update passwords or the like?

Thanks for the quick work getting everything back online!


----------



## Vlad

lill_canele said:


> It's okay, much better than when one of my folders at work with an entire year's worth of work was deleted.



That scenario makes me queasy just thinking about it. I backup everything regularly, digital tragedy strikes typically whenever you need it least.

The irony is that the corruption happened 23h:30min after the last 24h-interval database backup snapshot, so we almost maximized the data loss. Nicely done...


----------



## Megs

So sorry everyone! This didn't affect passwords or anything of the sort, just anything you shared on Wednesday pretty much was wiped and couldn't be saved. It happened on PurseBlog.com as well - no comments saved or editorial that we wrote on the backend yesterday saved. It isn't great, but could have been worse!


----------



## Yoshi1296

Yeah I remember yesterday I was unable to access the forum and purseblog around the evening I think? 

Thank you for getting it back up and running! Sorry this happened to yall. But im glad it wasnt worse!


----------



## paula3boys

I wondered what happened yesterday. On and off, several times, it kept telling me that the site was insecure and my browser wouldn't let me proceed. Then last night it just showed a white screen with an error message. Glad it is all fixed now. Thank you to those who work hard keeping it going and those who fix it


----------



## missie1

I figured something was funny because I couldn’t log in and then it kept jumping in and out. Be thankful it was just one day.


----------



## 880

Thank you so much for all the hard work that goes into TPF. Your members appreciate everything you do!


----------



## rose60610

So in the grand scheme of things, it was just a minor glitchy error with no harm occurring other than lost posts/messages?  Could this particular glitch repeat itself?


----------



## Penelopepursula

I can't log on using my app in Google, had to use another browser.


----------



## Vlad

rose60610 said:


> So in the grand scheme of things, it was just a minor glitchy error with no harm occurring other than lost posts/messages?  Could this particular glitch repeat itself?




We're doing a post mortem a the moment to figure out the root cause of the corruption to prevent it going forward. Regardless of its outcome, we did decrease the interval of backup snapshots to 8h to prevent more data loss in case this should happen again.


----------



## rose60610

Thanks!


----------



## orangefeast

I am curious if the thread I posted at 11:45pm is publicly viewable.  I can see it myself but wonder if it having been posted at 11:45pm last night means it is not publicly viewable.


----------



## Hermes Nuttynut

Will we get the Ukrainian flag colors back on the Purse Forum name at the top?  I noticed they were gone around midday yesterday.


----------



## Swanky

orangefeast said:


> I am curious if the thread I posted at 11:45pm is publicly viewable.  I can see it myself but wonder if it having been posted at 11:45pm last night means it is not publicly viewable.


It’s there!


----------



## Vlad

I just woke up to a note from our server techs. Last night we suffered a massive database corruption in an InnoDB table that caused a cascading effect and the host needed to roll back to a snapshot from the day before. This is the reason why your posts and PMs all disappeared.

Sorry for the inconvenience, this is as far as I remember the first time we needed to do this in the last 17 years.


----------



## cap4life

Explains the 404 error that was thrown around the afternoon PDT yesterday. Whatever database this messaging board sits on should have all the data stored. Maybe there’s a problem between the database and the front end interface. In any case, I appreciate all the hard work that’s gone into debugging!


----------



## Luxlover4ever

Ugh !!! we lost our post


----------



## BeenBurned

Was this message related to whatever happened?


----------



## Chagall

Hermes Nuttynut said:


> Will we get the Ukrainian flag colors back on the Purse Forum name at the top?  I noticed they were gone around midday yesterday.


I noticed the Ukrainian colours were missing. I like seeing them there.


----------



## whateve

Vlad said:


> We're doing a post mortem a the moment to figure out the root cause of the corruption to prevent it going forward. Regardless of its outcome, we did decrease the interval of backup snapshots to 8h to prevent more data loss in case this should happen again.


I was having trouble posting earlier in the day yesterday. Things kept timing out.


----------



## Addicted to bags

Thank you for proactively letting us know. I first thought it was overly heavy handed moderation when I went into my food thread and saw the last 2 pages missing! I was going to write an annoyed email this morning. Glad I didn't do it.


----------



## Yoshi1296

BeenBurned said:


> Was this message related to whatever happened?
> View attachment 5379214



Yeah this is what I was getting. I'm assuming it is related. Since the site was compromised, chrome assumes it is unsafe to access I think.


----------



## Lake Effect

@Vlad @Megs  Isn’t it nice to see what a difference you have made in peoples lives?? Can you imagine if you needed the blog/forum to go dark for updates, server, platform etc for a day or two?? Line up the crisis support!
Side note, too bad your  toddler can’t message me and hang out at 3 am.


----------



## Vlad

BeenBurned said:


> Was this message related to whatever happened?
> View attachment 5379214





Yoshi1296 said:


> Yeah this is what I was getting. I'm assuming it is related. Since the site was compromised, chrome assumes it is unsafe to access I think.



No that was our 1yr HTTPS certificate expiring yesterday, unrelated to this issue.


----------



## Gardria

I still can't go some thread since last night.
It says " An unexpected database error occurred. Please try again later. "


----------



## DaisyDD

Vlad said:


> I just woke up to a note from our server techs. Last night we suffered a massive database corruption in an InnoDB table that caused a cascading effect and the host needed to roll back to a snapshot from the day before. This is the reason why your posts and PMs all disappeared.
> 
> Sorry for the inconvenience, this is as far as I remember the first time we needed to do this in the last 17 years.


 I was online yesterday when this was happening. The only way I could get in the forum was through notifications via my email, I could then get in but once in pages would load and then freeze and I would end up back in my email. Going through the web I would get system error. I hope not too much was lost.


----------



## ConnieinSeattle

I'm glad the forum is back!  thank you for the mostly recovery.


----------



## Vlad

Gardria said:


> I still can't go some thread since last night.
> It says " An unexpected database error occurred. Please try again later. "



Do you have an example?


----------



## etoile de mer

Slightly different than @DaisyDD 's experience, but I also must have been online while it was happening. I could (briefly) access the posts that had disappeared by clicking through via the Alerts dropdown links. But then, soon after, the alerts related to the disappeared posts also vanished.

Thanks so much for letting us know what happened! Was certainly odd so see the system acting so glitchy!


----------



## Gardria

Vlad said:


> Do you have an example?


----------



## Vlad

I just woke up to a note from our server techs. Last night we suffered a massive database corruption in an InnoDB table that caused a cascading effect and the host needed to roll back to a snapshot from the day before. This is the reason why your posts and PMs all disappeared.

Sorry for the inconvenience, this is as far as I remember the first time we needed to do this in the last 17 years.


----------



## lovlouisvuitton

I'm getting nowhere trying to view the ATLV Thread, even when I'm not logged on. I get a blank page with an error message asking me to come back later. No problems with other Threads that I tested (time issue). The only way around it is, I need to log in and if I click on old ATLV notifications which takes me to the ATLV Thread. I'm on my desktop, so don't know if it's issue's with corruption or my computer. It started yesterday and still going on this morning, it's Good Friday here as I live in the future! 

*EDIT:* It happens when I actually try to Click on the *Louis Vuitton Shopping* Threads to get to the ATLV Thread. I get this message: 

White blank page with:  An unexpected database error occurred. Please try again later.


----------



## vickenator

As an IT professional, I winced when I saw the stickied message. I've been there and it's no fun. Thanks for getting things back up and running so quickly and for reducing the snapshot time to 8h as that will help if it ever happens again. I'd say one day of data loss in 17 years is a pretty good run, but the postmortem will help you guys out as well. Best of luck and thanks for keeping things going. This explains why the site was acting a little weird yesterday so that's a relief to know as well.


----------



## Vlad

Gardria said:


> View attachment 5379391








						Chanel Shopping
					

Sub-Forum for Chanel shopping-related threads




					forum.purseblog.com
				




The Chanel Shopping sub works fine, please try again.


----------



## katev

I am still having difficulty accessing the entire site. I have to search for other members and then try to "back-door enter" limited threads via their posts. When I try to access the site directly I still get this message:

"An unexpected database error occurred. Please try again later." 

I have already reset my computer and I am afraid to log out to try and log back in because I might lose the limited access that I have; any suggestions? Thanks!


----------



## Penelopepursula

Try a different browser. I couldn't get through yesterday on Chrome but surprisingly Safari worked.


----------



## Vlad

lovlouisvuitton said:


> I'm getting nowhere trying to view the ATLV Thread, even when I'm not logged on. I get a blank page with an error message asking me to come back later. No problems with other Threads that I tested (time issue). The only way around it is, I need to log in and if I click on old ATLV notifications which takes me to the ATLV Thread. I'm on my desktop, so don't know if it's issue's with corruption or my computer. It started yesterday and still going on this morning, it's Good Friday here as I live in the future!
> 
> *EDIT:* It happens when I actually try to Click on the *Louis Vuitton Shopping* Threads to get to the ATLV Thread. I get this message:
> 
> White blank page with:  An unexpected database error occurred. Please try again later.





katev said:


> I am still having difficulty accessing the entire site. I have to search for other members and then try to "back-door enter" limited threads via their posts. When I try to access the site directly I still get this message:
> 
> "An unexpected database error occurred. Please try again later."
> 
> I have already reset my computer and I am afraid to log out to try and log back in because I might lose the limited access that I have; any suggestions? Thanks!



I am going to dig into the logs to see what is happening here. On my end I can navigate all the various parts of the forums without issue.


----------



## redney

@Vlad, same thing is happening to me on my phone but not on my laptop when I try to access my watched threads at https://forum.purseblog.com/watched/threads. For both, I use Chrome.

When on my laptop, I can get to the watched threads page, but when on my phone (Android Samsung) I get the error message "An unexpected database error occurred. Please try again later."


----------



## Gardria

Vlad said:


> Chanel Shopping
> 
> 
> Sub-Forum for Chanel shopping-related threads
> 
> 
> 
> 
> forum.purseblog.com
> 
> 
> 
> 
> 
> The Chanel Shopping sub works fine, please try again.



Still not working 
I'm using chrome. But yeah now I just tried use other browser it worked.

Oh. I just deleted all my history and cookies from chrome. and it works now


----------



## katev

katev said:


> I am still having difficulty accessing the entire site. I have to search for other members and then try to "back-door enter" limited threads via their posts. When I try to access the site directly I still get this message:
> 
> "An unexpected database error occurred. Please try again later."
> 
> I have already reset my computer and I am afraid to log out to try and log back in because I might lose the limited access that I have; any suggestions? Thanks!





Vlad said:


> I am going to dig into the logs to see what is happening here. On my end I can navigate all the various parts of the forums without issue.





Gardria said:


> Still not working
> I'm using chrome. But yeah now I just tried use other browser it worked.
> 
> Oh. I just deleted all my history and cookies from chrome. and it works now



I've just updated and relaunched Chrome and now I can access the site again, thanks!


----------



## Jem131

Vlad said:


> I just woke up to a note from our server techs. Last night we suffered a massive database corruption in an InnoDB table that caused a cascading effect and the host needed to roll back to a snapshot from the day before. This is the reason why your posts and PMs all disappeared.
> 
> Sorry for the inconvenience, this is as far as I remember the first time we needed to do this in the last 17 years.



Sadly, nothing surprises me anymore.


----------



## 501st

limom said:


> My posts have been deleted too.
> What is the deal?
> What is the point of having a forum if opinions and posts within the guidelines are deleted?
> Yes, it is a privately owned forum but this is very frustrating.
> And confusing as well.



Welcome to the real life of system administration where this is bound to happen at some point, at any scale. Remember the Facebook DNS issue taking down all their services for hours last year or so?


----------



## vickenator

Was there another server hiccup recently? One of the threads I follow (the Involuntary Coach Ban Support Group) seems to have lost all posts from Thursday 5/26 through yesterday, 5/28.


----------



## Vlad

vickenator said:


> Was there another server hiccup recently? One of the threads I follow (the Involuntary Coach Ban Support Group) seems to have lost all posts from Thursday 5/26 through yesterday, 5/28.



Not that I am aware of, but I can ask our server technicians.


----------

