Tweet
Ok Folks I for some reason I cannot explain have been having difficulty linking from Sony's Forums so i decided to cut and paste this item...
From...http://eqforums.station.sony.com/eq/...id=261....this is taken...
Hello everyone,
We have now completed our last group of character restores in regards to the update issue that occurred on the morning of January 18th. Before I go into the details let me start by giving our apologies to all of those that were unknowingly affected by the correction of our error. We did feel that we needed to be aggressive in dealing with this issue. I also hope that everyone can understand that we needed to be vague on the details while our actions were being taken. However, now that it is complete let me take a moment and explain our solutions.
The summary is that we rolled back any character that had traded with the Beta merchants from our Prophecy of Ro testing server. We also rolled back characters that had traded with those ground zero characters as well as any subsequent traders along down the line.
Taking a step back I will start with the morning of the update. During the beta testing of an expansion we will have a number of NPCs available to facilitate the testing of new tradeskill recipes. Since the point is to actually test recipes and combines the items are free or extremely cheap. Unfortunately, due to a data error, these 25 NPCs with 1,530 items ended up on the live servers. As soon as we realized the issue we removed the NPCs from the Plane of Knowledge. We decided not to take down the servers until we fully understood the situation and its ramifications. This was the approach that we continued to take throughout the day – slow, steady, and methodical. After researching and tracking both the players and items we crafted a solution that would not require server downtime and would only affect a small percentage of our overall population.
We began with the characters that purchased items directly from the Beta merchants taking the time to exclude those people that only purchased harmless items such as water. We also excluded anyone who only sold to those merchants. After that we expanded to those that traded with those characters and iterated down the chain to include all trades. We felt that we had to widen the web in this way to prevent a number of secondary methods of dealing with these ill-gotten items such as trading the results of tradeskill combines. At this point we understand that we are directly affecting players that might have had no knowledge of the source of these items or results of these items. However, we believe it was necessary to take this aggressive step to maintain the integrity of the economy.
Soon thereafter we expanded our investigation to the other sources such as guild bank, shared bank, bazaar transactions, and other more obscure possibilities. This afternoon we ended with a final sweep of all areas to ensure that we conducted as thorough of a clean up as possible. We did this in a step by step manner so that we could keep an eye on the number of characters affected, maintain a high level of thoroughness, and minimize any chance of errors.
I have also seen a number of people suggest that we should have taken down the servers and rolled back every active character on every server. That was my first thought as well but once you get into the details you can see that this would not have been the best course of action. Primarily, every character would have lost their play time instead of the small percentage that was involved. Also, keep in mind that any rollback or restore has a small chance of an error and rolling back that number of characters would have magnified those exceptions to an unacceptable degree. After going through this process over the last 24 hours and seeing the results I am confident that we took the best course of action under the circumstances.
The result is that the integrity of the economy has been maintained while touching less than 1% of the character population of EverQuest. The majority of the EverQuest population was able to enjoy the game normally with zero server downtime and no loss of play time. We do understand that this might not matter to those of you that were rolled back and for that we apologize.
In conclusion, there are two known issues with the rollbacks regarding guild membership status and missing corpse issues that may have affected some customers. Customer Service will be assisting with these specific known issues, but will not be assisting with requests for reimbursement of items, coin, experience, AA’s, or flags that were removed due to the character rollbacks. If your character is having problems joining or being removed from a guild or if your character appears to be missing a corpse, please submit a petition under the appropriate category and they will assist you as quickly as possible.
Thank you for your patience,
Chris Lena
Producer
EverQuest
Kytherea ~~wielding a giant hammer & a hairless War Hamster~~
"I'm late, I'm late...."
...Ahem please forgive this method, but I also want you to excuse this whine...45 MINUTES is NOT a terrible loss of playtime Chris Lena and would not have cost one of our Guild mates Epics or another's 1.5 epic loot among other things...What should have been done is to drop all servers imediately and then rollback all online and offline to the point before the servers were put back up and go from there THIS has happened before, and no adverse problems nor were innocents subjected in any way other than a few minutes loss...
Bad Call Sony...Just my humble opinion.
From...http://eqforums.station.sony.com/eq/...id=261....this is taken...
Hello everyone,
We have now completed our last group of character restores in regards to the update issue that occurred on the morning of January 18th. Before I go into the details let me start by giving our apologies to all of those that were unknowingly affected by the correction of our error. We did feel that we needed to be aggressive in dealing with this issue. I also hope that everyone can understand that we needed to be vague on the details while our actions were being taken. However, now that it is complete let me take a moment and explain our solutions.
The summary is that we rolled back any character that had traded with the Beta merchants from our Prophecy of Ro testing server. We also rolled back characters that had traded with those ground zero characters as well as any subsequent traders along down the line.
Taking a step back I will start with the morning of the update. During the beta testing of an expansion we will have a number of NPCs available to facilitate the testing of new tradeskill recipes. Since the point is to actually test recipes and combines the items are free or extremely cheap. Unfortunately, due to a data error, these 25 NPCs with 1,530 items ended up on the live servers. As soon as we realized the issue we removed the NPCs from the Plane of Knowledge. We decided not to take down the servers until we fully understood the situation and its ramifications. This was the approach that we continued to take throughout the day – slow, steady, and methodical. After researching and tracking both the players and items we crafted a solution that would not require server downtime and would only affect a small percentage of our overall population.
We began with the characters that purchased items directly from the Beta merchants taking the time to exclude those people that only purchased harmless items such as water. We also excluded anyone who only sold to those merchants. After that we expanded to those that traded with those characters and iterated down the chain to include all trades. We felt that we had to widen the web in this way to prevent a number of secondary methods of dealing with these ill-gotten items such as trading the results of tradeskill combines. At this point we understand that we are directly affecting players that might have had no knowledge of the source of these items or results of these items. However, we believe it was necessary to take this aggressive step to maintain the integrity of the economy.
Soon thereafter we expanded our investigation to the other sources such as guild bank, shared bank, bazaar transactions, and other more obscure possibilities. This afternoon we ended with a final sweep of all areas to ensure that we conducted as thorough of a clean up as possible. We did this in a step by step manner so that we could keep an eye on the number of characters affected, maintain a high level of thoroughness, and minimize any chance of errors.
I have also seen a number of people suggest that we should have taken down the servers and rolled back every active character on every server. That was my first thought as well but once you get into the details you can see that this would not have been the best course of action. Primarily, every character would have lost their play time instead of the small percentage that was involved. Also, keep in mind that any rollback or restore has a small chance of an error and rolling back that number of characters would have magnified those exceptions to an unacceptable degree. After going through this process over the last 24 hours and seeing the results I am confident that we took the best course of action under the circumstances.
The result is that the integrity of the economy has been maintained while touching less than 1% of the character population of EverQuest. The majority of the EverQuest population was able to enjoy the game normally with zero server downtime and no loss of play time. We do understand that this might not matter to those of you that were rolled back and for that we apologize.
In conclusion, there are two known issues with the rollbacks regarding guild membership status and missing corpse issues that may have affected some customers. Customer Service will be assisting with these specific known issues, but will not be assisting with requests for reimbursement of items, coin, experience, AA’s, or flags that were removed due to the character rollbacks. If your character is having problems joining or being removed from a guild or if your character appears to be missing a corpse, please submit a petition under the appropriate category and they will assist you as quickly as possible.
Thank you for your patience,
Chris Lena
Producer
EverQuest
Kytherea ~~wielding a giant hammer & a hairless War Hamster~~
"I'm late, I'm late...."
...Ahem please forgive this method, but I also want you to excuse this whine...45 MINUTES is NOT a terrible loss of playtime Chris Lena and would not have cost one of our Guild mates Epics or another's 1.5 epic loot among other things...What should have been done is to drop all servers imediately and then rollback all online and offline to the point before the servers were put back up and go from there THIS has happened before, and no adverse problems nor were innocents subjected in any way other than a few minutes loss...
Bad Call Sony...Just my humble opinion.
GM Brewing 12/18/04
Smithing 110
Pottery 175
Comment