12 ->
^^
vv
List results:
Search options:
Use \ before commas in usernames
    history of metroid 2002 speed demos archive
  • part 1 (june 2003 - 11 november 2003)
  • part 2 (11 november 2003 - 30 june 2004)
  • part 3 (30 june 2004 - 15 july 2004)


part 3

prologue

the gruesome nature of deskjockey's real life schizophrenia had been very difficult for me to bear, and so i confided in rachel. i also thought that it would be better for her to find out about "marie" from me rather than to wonder who this girl was i was constantly talking to online.

on the evening of 30 june, rachel slipped while talking to marie, and it became clear to marie that i had told rachel their secret. as root, they typed "rm -rf /" - the command to erase the entire computer's filesystem recursively without asking for confirmation.

i recall being sick to my stomach, my hands shaking, but otherwise calm enough to think through what had just happened and to prioritize my actions for the next few minutes. i think that living with this individual for months had gone a long way toward preparing me for a situation like this. for example, a few months prior, in a moment of rage at me for doing something or other, they had actually deleted the entire irc server they had created, letting it run from memory alone for a few weeks until they cooled off and rebuilt it on my urging. almost no one suspected anything, because the irc server had continued to run virtually problem-free, even though it no longer existed on the server computer's hard drive. i wondered if the other servers would be the same way now.

because i had just downloaded metroid 2002 in its entirety to my pc a few days before, it seemed likely that at least the core of the site would narrowly escape annihilation, but because the forum data was stored elsewhere on the server's hard drive, i would stand to lose at least the last few months of posts if i didn't squeeze a backup out now before the data was erased (the most recent forum backup i had was several months old). i navigated to the forum backup screen and requested a backup. i received a 'file not found' error, so i reloaded the forum to try again. now the entire forum was gone.

i calmly instructed everyone on irc to meet on espernet and waited for the end to come. radix was still connected to the server (marie had terminated all of my connections) and wanted to know if he could do anything to help, so i gave him the root password in a final hope for salvaging the forum. but marie had deleted the root user, as well.

i knew that the rest of my summer break was probably spoken for now, but as i thought about how brilliant this person had been, i realized that i would never be safe online again. suddenly it seemed almost futile to rebuild the server with a genius of that magnitude out there bent on extracting revenge on me and my community.

but i knew that i owed it to these people who had given me so much. i had to try.



act 1: everything that's wonderful

sitting alone in my parents' basement, i considered my next move.

the first priority was to disconnect my desktop mac from the outside, because it was where the only remaining copy of metroid 2002 existed. if marie's intention was to completely erase the site, then that was where it would go next.

i reached around behind the machine and plucked the ethernet cable from the network port. then i went into the airport menu and disabled the wireless interface.

with the machine completely disconnected from the outside world, i moved to my ibook and navigated to the ev1servers members' area in a browser. i put in a "priority restore" request, so that the hard drive on the server would be wiped and a new install of linux put back on within two hours' time. the cost would be $74.

unfortunately, marie knew what would start out as the root password on this new installation of linux. there was no way at that time to automatically change this in the ev1servers system, so it would be a race against it to try to take over the new server the moment the machine became visible on the internet.

i got on espernet, where i could talk to radix. whether he would make a good co-admin was irrelevant - i didn't feel as though i was capable of facing the threat of marie on my own, and he was the best man for the job. i began a direct chat with him, so that our conversation would not be routed through espernet servers and, hopefully, not anywhere close enough for marie to sniff out. generating new passwords, i gave him instructions on how to become root on the server and set the root password. i thought that with two people hammering away at the machine, it would more likely be us than it who would get in first.

radix and i made small talk and i tried to relax for the next four hours until finally, at 3:38 am, reloading the restore request at the ev1 site showed that someone was about to fill it. i was pleased to see that the restore had been downgraded to a "standard" restore, for which i would only have to pay $30. it was only fair, what with them taking twice as long as they were supposed to to do the restore.

at 4:34 am, while radix was taking a bathroom break, i successfully logged into the server and changed the admin and root passwords. i checked for other connections and saw nothing, but i knew that my worries about marie were far from over.

i was pleased to see that the old, wiped disk had been attached to the server as a secondary drive, while the new installation i had logged into was actually on a new primary disk. i figured that this turn of events could only make my life easier, as i wanted to attempt to recover some of the data that had been on the disk, especially in the case of the forum. my head was filled with memories of great posts, both social and constructive, that had been made over the last two months since the last backup had been made.

i discussed the situation with radix:

Quote:
04:43:25 AM: Radix: so what's your plan now
04:46:35 AM: Nate: trying to find data recovery
04:46:39 AM: Nate: the problem is that it's ext3
04:47:35 AM: Radix: dont know anything about the file system
04:47:46 AM: Radix: but unless it's encrypted, the files are there at some point on the disk
04:47:57 AM: Radix: so just a raw read would be find parts of them
04:50:07 AM: Radix: but if things were fragmented, actually restoring the files will be about impossible
04:50:08 AM: Nate: yeah...it's conceivable we could actually write it ourselves
04:50:16 AM: Nate: well yeah except for that, fuck


at 5:00 am, i entered an ev1servers support chatroom with "cindy" and asked how to go about recovering data from a drive that had been rm -rf /'d. she was reluctant to provide me with information, because ev1 did not support actually taking the actions she was prescribing, but i was able to glean one crucial piece of information that would come to define our entire recovery operation:

Quote:
[05:19:56] Cindy R : dd if=/dev/hdc1 of=oldfilesystem.fs
[05:20:06] Cindy R :  you can work in chunks, too - you could limit the size of the oldfilesystem.fs file by doing something like
[05:21:20] Cindy R :  but ***NOTE*** this is not supported activity and we do not recommend running these commands


it was in this way that i learned about the dd or raw read command. even though all of the files on the old drive had been deleted, the data that had made up the files still existed on the drive, as radix had said earlier. the dd command was the missing link i needed to begin reading from the old drive.

i began by reading a tiny block from the start of both the primary and secondary drives to files and sending those files to myself for examination in virtualdub's hex editor. sure enough:

Quote:
05:55:05 AM: Radix: can you give the first few hex digits
05:55:13 AM: Radix: one of them ought to be 55
05:55:28 AM: Nate: it spells LILO
05:55:43 AM: Radix: heh
05:55:45 AM: Nate: linux loader...
05:55:51 AM: Nate: but you probably know
05:55:58 AM: Nate: checking the other one
05:56:07 AM: Nate: it's almost the same
05:56:10 AM: Nate: a handful of bytes are different
05:56:16 AM: Radix: yeah all right
05:56:21 AM: Radix: so we know we're reading the hard drives ;-)
05:56:24 AM: Nate: cool
05:56:58 AM: Radix: how big was the old disk?
05:57:21 AM: Nate: supposedly 60 gig


by now the loss we had just suffered was finally beginning to sink in:

Quote:
06:24:13 AM: Radix: i keep fighting my instant to check m2k2 forum
06:24:19 AM: Radix: instinct
06:24:20 AM: Nate: yeah... me too... -_-
06:24:31 AM: Radix: i'll check the nes videos instead (kill me)
06:24:37 AM: Nate: i can't stop thinking about that w.dawgs post
06:24:42 AM: Nate: where he edited like six times today
06:24:49 AM: Radix: i only saw 4 edits
06:24:56 AM: Nate: so beautiful
06:24:58 AM: Radix: smoething about ice missiles in reverse?
06:25:07 AM: Nate: yeah i guess minor s.break
06:25:12 AM: Nate: he ended up saying like 2-3 sec faster iirc
06:25:18 AM: Nate: but dunno how he calced that


downloading the 60 gig file representing all of the erased drive's data on my parents' slow connection would be no small feat:

Quote:
06:40:10 AM: Radix: how long will it take you to download 60gb
06:41:23 AM: Nate: um i dunno
06:41:30 AM: Nate: i think i get 3 megabaud if i'm really really lucky?
06:41:51 AM: Nate: 45 hours?
06:41:53 AM: Nate: that can't be right
06:42:22 AM: Nate: 45 hours
06:42:23 AM: Nate: shit


on top of that, having to use the new primary disk on the server to store the data from the old primary disk meant that i couldn't serve anything from the machine in the meantime. i changed the domain record for www.metroid2002.com to point to my .mac account, where i uploaded a brief, explanatory message.

radix said that he would write a program to cut chunks out of the 60 gig file representing all of the data on the old drive, which we would then compress using gzip for faster downloading. then he went to bed; it was 6:45 am, and he had to wake up for work a few hours later, being accustomed to going to bed around 2:00 every night.

meanwhile, i secured my internet connection to the best of my ability, moved back to my desktop mac and tried to figure out how to download a 60 gig (to be precise, 60,022,480,896 byte) file. apache, the web server, wouldn't do it: the maximum file size it could handle at that time was only 4 gig. i installed proftpd on the server to get around that problem and began the long download, not knowing how long it would take radix to complete his program. the download further ensured that the site would not go back up anytime soon, as using the same connection to upload the site as to download the erased drive data would cause both transfers to run unacceptably slowly.

at 8:49 am, i went to get a few hours of sleep.


act 2: your impossible dream

from work, radix logged into my desktop mac (where the data was coming down) and prepared his large file slicing program, cuthdb.c. unfortunately, i did not have any more time to work on the recovery before i left with my parents for the annual family reunion in iowa. i probably could have gotten out of it if i had really tried, but i reasoned that it would be several days before the old data finished downloading, and i would not be able to do anything radix could not do in that time, anyway.

while i was in iowa over independence day weekend, radix wrote his scanning program and fed it strings to help it identify blocks of data that were likely to contain the forum:

Quote:
10:23:05 PM: Radix: well i looked in the forum backup from may 1 for some "unique" words that I can put into my program i'll write to scan the old drive for portions of files
10:23:15 PM: Radix: i'll use these words to try to find the forum...
10:23:18 PM: Radix: chromodynamics
10:23:18 PM: Radix: tunneling
10:23:18 PM: Radix: compactivityness
10:23:18 PM: Radix: antiochforever
10:23:18 PM: Radix: REJOOVINATEDDD!
10:24:09 PM: Ajbolt89: hgahahaha
10:24:13 PM: Ajbolt89: hahahaha
10:24:28 PM: Ajbolt89: I saw compactivityness and thought it was just a random jumble of letters
10:24:30 PM: Ajbolt89: that was awesome
10:24:40 PM: Radix: i think it's safe to assume those words dont appear anywhere else in the site ;-)


then, on 4 july, his scanning program produced output for the first time. however, he was limited to the first 17 gig of the data, because my parents' unreliable connection had killed the download. ever helpful, radix logged into the server and used his splitting program to split the original 60 gig data file, enabling him to resume the transfer of the data to my mac. he then used his modified md5 program to make sure each segment of the data was downloading properly.

the next day, he reported detecting pieces the forum:

Quote:
05:13:20 PM: Radix: guess i'd better check on the transfer status
05:18:51 PM: Radix: 24/56 files sent
05:23:35 PM: Radix: i found evidence of the forum in several parts of the drive
05:23:39 PM: Radix: it's probably fragmented to hell
05:24:21 PM: Radix: Forum fragment compactivityness found at 26760962694
05:24:36 PM: Radix: Forum fragment antiochforever found at 26761900870
05:24:48 PM: Radix: Forum fragment REJOOVINATEDDD! found at 26763760528
05:25:08 PM: Radix: and then there's things at 298* and 321*
05:25:13 PM: Ajbolt89: hahahaa
05:25:23 PM: Ajbolt89: REJOOVINATEDDD!


i arrived back at my parents' house on 6 july at 4:51 pm and helped radix check the downloaded segments of the server data. radix had begun to send some segments to fileplanet (at that time, speed demos archive still used gamespy's fileplanet as its primary file storage location). he was able to upload to fileplanet an order of magnitude faster than to my mac on my parents' slow connection.

once all the data segments were downloaded and verified with md5 that evening, i attempted to upgrade the operating system on the server. the version installed by ev1servers was already two years old and had a multitude of security problems, and i felt that it was an unacceptable risk to leave the server in that condition before i started to upload the site.

unfortunately, rpm, the red hat package manager, failed to update all of the packages needed for the server to boot properly, and so the server never came back up when i rebooted it to complete the upgrade of the operating system:

Quote:
08:45:12 PM: Nate: 7/6/04 8:34:26 PM
08:45:12 PM: Nate: DataCenter
08:45:19 PM: Nate: server cannot connect to the network. it is missing essential dependencies files needed to start eth0. server will need to be restored to get back online


that meant forking over another $30 to ev1 for a another standard restore (i couldn't afford a priority restore by that point) and more waiting to restore the site. on top of that, without rpm, i had no idea how i was supposed to upgrade the operating system on the machine, leaving it wide open to marie.

needless to say, that was the last time i used rpm.

ev1 worked much more quickly this time, restoring the server less than an hour after i put in the restore request. by 10:24 pm, i was back in. i installed apache (the web server), proftpd (the ftp server) and mysql (for the forum) and began to upload the site in the early hours of 7 july:

Quote:
02:03:53 AM: Nate: uploading metroid 2002


the forum source code finished uploading about eleven hours later, and i restored the forum database backup from 1 may. then i located the forum data files mysql had created on the disk and sent samples of them to radix to feed into his scanning program.

it turned out that, out of the three files mysql created to represent each table in the forum's database, only one, the .MYD file, was vital to recover - another was constant (it could be created just as well with the 1 may backup), and the last could be regenerated from a valid .MYD file by running a repair command in mysql.

but we weren't out of the water yet. some of the forum's .MYD files, especially the one representing the posts text table, were many megabytes in size, and there was no doubt in radix's or my mind that such a large file would be horribly fragmented on the disk. that meant that we would have to join together perhaps thousands of tiny segments with single-byte accuracy to successfully restore what had happened on the forum after 1 may.

still, less than one week after marie had issued the rm -rf / command, metroid 2002 and its users' directories were back up for all the world to see. radix and i spent no time revelling in this small victory and began instead to concentrate on reconstructing the shattered forum.


act 3: impossible is nothing

at first, things seemed almost hopeless for the forum. radix found a large block of posts all in one place, but our luck quickly ran out as the data broke down into tiny 4 kilobyte blocks. he was reduced to manually scrolling through 4k blocks of raw data, trying to guess where the next tiny slice of the forum would start.

an analogy would be shredding a small novel into thousands of pieces, each about 4,000 characters long, blowing those pieces all over your lawn and then attempting to reconstruct the story with 100% accuracy. often you would be able to tell which piece came next, but sometimes there might be more than one piece that fit, engendering situations in which you have one piece left over that belongs somewhere in the already reconstructed mass of text you thought was complete.

to make matters worse, mysql was refusing to reconstruct the database files properly if there was even one tiny mistake in the data radix recovered, a mistake neither of us could see. going back to the analogy of the novel, this requirement of 100% accuracy meant that if even a single one of the pieces of the book were damaged in the slightest, its corner dog-eared or torn, you would fail the task, and it would be as if you had never even tried, the book lost forever.

as the gravity of the situation became apparent, morale among the administrators hit an all-time low. but i was still quite angry about what had happened to me, to my site and, most of all, to the people who loved it, and that it was my fault only infuriated me more. i was determined to win against marie, to reduce its impact on my community to nil, and so i tried to spur radix on as the situation grew more and more dim:

Quote:
02:33:40 AM: Radix: it was going really well, i found all of may and the start of june in a segment in '27'
02:33:42 AM: Radix: but then it vanished
02:33:48 AM: Radix: i found a few portions in '01'
02:33:53 AM: Radix: but now i'm at a post that i dont know what follows
02:34:07 AM: Radix: so i need to do another search of the entire thing to find it based on one samll idea
02:34:19 AM: Nate: can i see what the end looked like?
02:34:44 AM: Radix: 62e4fbd055 (null) R (null) [i
02:34:48 AM: Radix: it's the start of a new post
02:34:52 AM: Radix: the only text we have is [i
02:34:56 AM: Radix: which probably is [img
02:35:04 AM: Radix: but maybe it's just [i for italic
02:35:13 AM: Radix: so either it's [img:62e4fbd055
02:35:22 AM: Radix: or [i:62e4fbd055
02:35:46 AM: Nate: but it must be that
02:35:48 AM: Radix: actually i can do a fast seach on the block boundries for anything that starts with those
02:35:54 AM: Nate: yeah
02:35:56 AM: Nate: go go go
02:36:04 AM: Nate: i can't think of anything else it could be
02:36:34 AM: Nate: btw i don't know how your text wrangling skills are but i can easily find/replace to create a restore file to recreate the database file
02:36:49 AM: Nate: can't believe i didn't think of that before
02:36:53 AM: Nate: what we're doing is retarded
02:36:57 AM: Nate: the way you're not supposed to do it
02:36:59 AM: Nate: of course it fails
02:39:12 AM: Radix: i just dont like delaying the forum being back
02:39:16 AM: Radix: for any longer
02:39:18 AM: Nate: why?
02:39:21 AM: Radix: started seach
02:39:25 AM: Radix: because ppl dont have a place to bull shit
02:39:36 AM: Nate: why is that bad?
02:39:52 AM: Nate: i think it would be worse to come back without familiar threads like the political one or the homosexuality one
02:39:55 AM: Radix: if site's 1 forum goes down for too long and site 2's forum attract's there users, site 1 won't get them back
02:39:56 AM: Nate: people wouldn't feel at home
02:40:05 AM: Radix: 'their'
02:40:11 AM: Nate: but yeah hard to say when it goes from my fear to yours being the bigger one

[...]

02:50:41 AM: Radix: it was [img
02:50:53 AM: Nate: cool
02:51:03 AM: Radix: only a 1k fragment
02:51:14 AM: Nate: ouch
02:51:26 AM: Radix: ends in the middle of a quote tunnel post
02:51:55 AM: Nate: cool
02:51:56 AM: Radix: can you open 01 in vdub
02:52:12 AM: Radix: and search for [/quote:23088554d4
02:52:58 AM: Nate: searching
02:54:55 AM: Nate: 2F6BECE0
02:56:37 AM: Nate: i'll start on the script as soon as i wake up and run some tests
02:56:38 AM: Radix: i dont nkow ... i'm losing hope
02:56:43 AM: Nate: sleep
02:56:48 AM: Nate: you will feel more like doing it tomorrow
02:56:50 AM: Radix: i already lost some hope last night when i couldn't find the end of "I wouldn't cal"
02:57:03 AM: Radix: but then i found it and found so much more
02:57:20 AM: Radix: it moved from 24 to 25 to 26 to 27
02:57:26 AM: Radix: and then i found all of april and may in one area
02:57:36 AM: Radix: and i switched to a much faster way to do it
02:57:42 AM: Radix: but still :-\
02:57:52 AM: Nate: i think you're just getting tired of doing it
02:57:55 AM: Nate: we're almost done though
02:57:59 AM: Nate: actually we can stop at any time
02:58:05 AM: Nate: but at this point i see no reason to stop
03:00:34 AM: Nate: i need to go to bed
03:00:39 AM: Nate: pick it up tomorrow
03:00:47 AM: Radix: i'm just going to send myself the rest of 01
03:00:50 AM: Radix: and go to bed
03:00:55 AM: Nate: ok go ahead and send me what you have if you can
03:01:01 AM: Radix: it's on the server
03:01:05 AM: Radix: those 2 files
03:01:05 AM: Nate: if i can restore it maybe it will give you a reason to keep going


as the days dragged by, radix assembled hundreds upon hundreds of bits of the forum's remains into a mock database file, on which i ran mysql repair commands. we learned about the format of the database file, and we learned the meanings of the various errors mysql would spew at us when things were not 100% right. radix wrote a program to scan the file undergoing reconstruction for errors, and the work on eliminating errors sped up dramatically.

little by little, piece by piece, we struggled to pull the forum back from its grave:

Quote:
06:07:55 PM: Nate: a handful of errors
06:07:56 PM: Nate: mysql> REPAIR TABLE phpbb2_posts USE_FRM;
06:07:56 PM: Nate: +--------------------+--------+----------+----------------------------------------------------------------+
06:07:56 PM: Nate: | Table              | Op    | Msg_type | Msg_text                                                      |
06:07:56 PM: Nate: +--------------------+--------+----------+----------------------------------------------------------------+
06:07:56 PM: Nate: | phpbb.phpbb2_posts | repair | info    | Found link that points at 802112 (outside data file) at 33084  |
06:07:56 PM: Nate: | phpbb.phpbb2_posts | repair | info    | Found link that points at 802924 (outside data file) at 99752  |
06:07:56 PM: Nate: | phpbb.phpbb2_posts | repair | info    | Found link that points at 802980 (outside data file) at 144316 |
06:07:56 PM: Nate: | phpbb.phpbb2_posts | repair | info    | Key 1 - Found wrong stored record at 548856                    |
06:07:56 PM: Nate: | phpbb.phpbb2_posts | repair | info    | Found link that points at 771468 (outside data file) at 564272 |
06:07:56 PM: Nate: | phpbb.phpbb2_posts | repair | info    | Wrong bytesec: 172-90-0 at 743116; Skipped                    |
06:07:56 PM: Nate: | phpbb.phpbb2_posts | repair | info    | Found link that points at 803036 (outside data file) at 691592 |
06:07:56 PM: Nate: | phpbb.phpbb2_posts | repair | info    | Wrong bytesec: 91-0-13 at 754976; Skipped                      |
06:07:56 PM: Nate: | phpbb.phpbb2_posts | repair | info    | Wrong bytesec: 225-64-49 at 750636; Skipped                    |
06:07:56 PM: Nate: | phpbb.phpbb2_posts | repair | info    | Wrong bytesec: 64-24-254 at 750656; Skipped                    |
06:07:56 PM: Nate: | phpbb.phpbb2_posts | repair | info    | Wrong bytesec: 1-1-1 at 742376; Skipped                        |
06:07:56 PM: Nate: | phpbb.phpbb2_posts | repair | info    | Found wrong packed record at 737248                            |
06:07:56 PM: Nate: | phpbb.phpbb2_posts | repair | info    | Found link that points at 788520 (outside data file) at 739296 |
06:07:56 PM: Nate: | phpbb.phpbb2_posts | repair | info    | Found link that points at 789512 (outside data file) at 740288 |
06:07:56 PM: Nate: | phpbb.phpbb2_posts | repair | info    | Found link that points at 789748 (outside data file) at 740452 |
06:07:56 PM: Nate: | phpbb.phpbb2_posts | repair | info    | Found link that points at 792360 (outside data file) at 743100 |
06:07:56 PM: Nate: | phpbb.phpbb2_posts | repair | info    | Found link that points at 794888 (outside data file) at 743480 |
06:07:56 PM: Nate: | phpbb.phpbb2_posts | repair | info    | Found link that points at 793352 (outside data file) at 744056 |
06:07:56 PM: Nate: | phpbb.phpbb2_posts | repair | info    | Found link that points at 793876 (outside data file) at 744508 |
06:07:56 PM: Nate: | phpbb.phpbb2_posts | repair | info    | Found link that points at 803524 (outside data file) at 747916 |
06:07:56 PM: Nate: | phpbb.phpbb2_posts | repair | info    | Found link that points at 800560 (outside data file) at 749788 |
06:07:56 PM: Nate: | phpbb.phpbb2_posts | repair | info    | Found link that points at 801660 (outside data file) at 752364 |
06:07:56 PM: Nate: | phpbb.phpbb2_posts | repair | warning  | Number of rows changed from 0 to 20829                        |
06:07:56 PM: Nate: | phpbb.phpbb2_posts | repair | status  | OK                                                            |
06:07:56 PM: Nate: +--------------------+--------+----------+----------------------------------------------------------------+
06:07:56 PM: Nate: 24 rows in set (0.56 sec)
06:07:56 PM: Nate:
06:07:56 PM: Nate: mysql>
06:36:23 PM: Nate: got another page of fusion best times posts with that most recent repair
06:45:36 PM: Nate: had over 10k hits on sunday
06:45:51 PM: Nate: so the week of downtime doesn't seem to have affected the site's popularity
06:51:49 PM: Nate: a good sign for the forum imo


Quote:
09:06:30 PM: Radix: i found more missing pieces of posts
09:06:34 PM: Radix: i think i have the entire thing now
09:06:39 PM: Radix: the size is 810952
09:06:49 PM: Radix: which is greater than any of the numbers baout being 'outside data file'
09:28:26 PM: Radix: i now understand the format of these .myd files
09:28:32 PM: Radix: .MYD :-0
09:50:43 PM: Radix: seaching for users file
09:50:49 PM: Radix:    ^r
11:00:06 PM: Nate: cool
11:00:10 PM: Nate: how's it going?
11:00:53 PM: Radix: found some more of it in 25
11:00:55 PM: Radix: but nothing in 26/27
11:00:58 PM: Radix: so the rest is probably in 01
11:01:04 PM: Radix: but there's none of it in the portions i alrady have
11:01:12 PM: Radix: try a repair with the posts file i uploaded
11:01:20 PM: Nate: ok
11:01:57 PM: Radix: need to take garbage out
11:02:11 PM: Nate: mysql> REPAIR TABLE phpbb2_posts USE_FRM;
11:02:11 PM: Nate: +--------------------+--------+----------+-------------------------------------------------------------+
11:02:11 PM: Nate: | Table              | Op    | Msg_type | Msg_text                                                    |
11:02:11 PM: Nate: +--------------------+--------+----------+-------------------------------------------------------------+
11:02:11 PM: Nate: | phpbb.phpbb2_posts | repair | info    | Delete link points outside datafile at 802112              |
11:02:11 PM: Nate: | phpbb.phpbb2_posts | repair | info    | Wrong bytesec: 1-1-0 at 802924; Skipped                    |
11:02:11 PM: Nate: | phpbb.phpbb2_posts | repair | info    | Wrong bytesec: 1-0-161 at 802980; Skipped                  |
11:02:11 PM: Nate: | phpbb.phpbb2_posts | repair | info    | Key 1 - Found wrong stored record at 548856                |
11:02:11 PM: Nate: | phpbb.phpbb2_posts | repair | info    | Wrong bytesec: 1-1-0 at 771468; Skipped                    |
11:02:11 PM: Nate: | phpbb.phpbb2_posts | repair | info    | Wrong bytesec: 50-56-48 at 754976; Skipped                  |
11:02:11 PM: Nate: | phpbb.phpbb2_posts | repair | info    | Wrong bytesec: 207-64-48 at 750636; Skipped                |
11:02:11 PM: Nate: | phpbb.phpbb2_posts | repair | info    | Wrong bytesec: 64-28-254 at 750656; Skipped                |
11:02:11 PM: Nate: | phpbb.phpbb2_posts | repair | info    | Wrong bytesec: 217-64-49 at 775404; Skipped                |
11:02:11 PM: Nate: | phpbb.phpbb2_posts | repair | info    | Wrong bytesec: 85-0-216 at 745476; Skipped                  |
11:02:11 PM: Nate: | phpbb.phpbb2_posts | repair | info    | Wrong bytesec: 217-64-49 at 775404; Skipped                |
11:02:11 PM: Nate: | phpbb.phpbb2_posts | repair | info    | Wrong bytesec: 100-57-0 at 763364; Skipped                  |
11:02:11 PM: Nate: | phpbb.phpbb2_posts | repair | info    | Wrong bytesec: 85-0-246 at 747672; Skipped                  |
11:02:11 PM: Nate: | phpbb.phpbb2_posts | repair | info    | Wrong bytesec: 64-24-254 at 768240; Skipped                |
11:02:11 PM: Nate: | phpbb.phpbb2_posts | repair | info    | Wrong bytesec: 64-28-254 at 783504; Skipped                |
11:02:11 PM: Nate: | phpbb.phpbb2_posts | repair | info    | Wrong bytesec: 86-0-218 at 755068; Skipped                  |
11:02:11 PM: Nate: | phpbb.phpbb2_posts | repair | info    | Wrong bytesec: 55-97-101 at 768836; Skipped                |
11:02:12 PM: Nate: | phpbb.phpbb2_posts | repair | info    | Wrong bytesec: 0-5-0 at 771232; Skipped                    |
11:02:12 PM: Nate: | phpbb.phpbb2_posts | repair | info    | Wrong bytesec: 64-24-254 at 786216; Skipped                |
11:02:12 PM: Nate: | phpbb.phpbb2_posts | repair | info    | Wrong bytesec: 218-64-52 at 776504; Skipped                |
11:02:12 PM: Nate: | phpbb.phpbb2_posts | repair | info    | Wrong bytesec: 1-1-1 at 794868; Skipped                    |
11:02:12 PM: Nate: | phpbb.phpbb2_posts | repair | info    | Wrong bytesec: 220-64-49 at 783232; Skipped                |
11:02:12 PM: Nate: | phpbb.phpbb2_posts | repair | info    | Wrong bytesec: 53-54-55 at 789512; Skipped                  |
11:02:12 PM: Nate: | phpbb.phpbb2_posts | repair | info    | Wrong bytesec: 0-1-0 at 789748; Skipped                    |
11:02:12 PM: Nate: | phpbb.phpbb2_posts | repair | info    | Wrong bytesec: 1-1-1 at 792360; Skipped                    |
11:02:12 PM: Nate: | phpbb.phpbb2_posts | repair | info    | Wrong bytesec: 1-0-177 at 794888; Skipped                  |
11:02:12 PM: Nate: | phpbb.phpbb2_posts | repair | info    | Wrong bytesec: 1-1-1 at 793352; Skipped                    |
11:02:12 PM: Nate: | phpbb.phpbb2_posts | repair | info    | Wrong bytesec: 0-1-0 at 793876; Skipped                    |
11:02:12 PM: Nate: | phpbb.phpbb2_posts | repair | info    | Wrong bytesec: 225-64-52 at 803524; Skipped                |
11:02:12 PM: Nate: | phpbb.phpbb2_posts | repair | info    | Wrong bytesec: 91-0-239 at 800560; Skipped                  |
11:02:12 PM: Nate: | phpbb.phpbb2_posts | repair | info    | Wrong bytesec: 98-100-0 at 801660; Skipped                  |
11:02:12 PM: Nate: | phpbb.phpbb2_posts | repair | warning  | Duplicate key for record at 738804 against record at 734812 |
11:02:12 PM: Nate: | phpbb.phpbb2_posts | repair | warning  | Duplicate key for record at 738840 against record at 734848 |
11:02:12 PM: Nate: | phpbb.phpbb2_posts | repair | warning  | Duplicate key for record at 738876 against record at 734884 |
11:02:12 PM: Nate: | phpbb.phpbb2_posts | repair | warning  | Duplicate key for record at 738912 against record at 734920 |
11:02:12 PM: Nate: | phpbb.phpbb2_posts | repair | warning  | Duplicate key for record at 738948 against record at 734956 |
11:02:12 PM: Nate: | phpbb.phpbb2_posts | repair | warning  | Duplicate key for record at 738984 against record at 734992 |
11:02:12 PM: Nate: | phpbb.phpbb2_posts | repair | warning  | Duplicate key for record at 739020 against record at 735028 |
11:02:12 PM: Nate: | phpbb.phpbb2_posts | repair | warning  | Duplicate key for record at 739056 against record at 735064 |
11:02:12 PM: Nate: | phpbb.phpbb2_posts | repair | warning  | Duplicate key for record at 739092 against record at 735100 |
11:02:12 PM: Nate: | phpbb.phpbb2_posts | repair | warning  | Duplicate key for record at 739128 against record at 735136 |
11:02:12 PM: Nate: | phpbb.phpbb2_posts | repair | warning  | Duplicate key for record at 739200 against record at 735208 |
11:02:12 PM: Nate: | phpbb.phpbb2_posts | repair | warning  | Duplicate key for record at 739236 against record at 735244 |
11:02:12 PM: Nate: | phpbb.phpbb2_posts | repair | warning  | Duplicate key for record at 739272 against record at 735280 |
11:02:12 PM: Nate: | phpbb.phpbb2_posts | repair | warning  | Duplicate key for record at 739308 against record at 735316 |
11:02:12 PM: Nate: | phpbb.phpbb2_posts | repair | warning  | Duplicate key for record at 739344 against record at 735352 |
11:02:12 PM: Nate: | phpbb.phpbb2_posts | repair | warning  | Duplicate key for record at 739380 against record at 735388 |
11:02:12 PM: Nate: | phpbb.phpbb2_posts | repair | warning  | Duplicate key for record at 739416 against record at 735424 |
11:02:12 PM: Nate: | phpbb.phpbb2_posts | repair | warning  | Duplicate key for record at 739452 against record at 735460 |
11:02:12 PM: Nate: | phpbb.phpbb2_posts | repair | warning  | Duplicate key for record at 739484 against record at 735492 |
11:02:12 PM: Nate: | phpbb.phpbb2_posts | repair | warning  | Duplicate key for record at 739520 against record at 735528 |
11:02:12 PM: Nate: | phpbb.phpbb2_posts | repair | warning  | Duplicate key for record at 739556 against record at 735564 |
11:02:12 PM: Nate: | phpbb.phpbb2_posts | repair | warning  | Duplicate key for record at 739592 against record at 735600 |
11:02:12 PM: Nate: | phpbb.phpbb2_posts | repair | warning  | Duplicate key for record at 739628 against record at 735636 |
11:02:12 PM: Nate: | phpbb.phpbb2_posts | repair | warning  | Duplicate key for record at 739664 against record at 735672 |
11:02:12 PM: Nate: | phpbb.phpbb2_posts | repair | warning  | Duplicate key for record at 739700 against record at 735708 |
11:02:12 PM: Nate: | phpbb.phpbb2_posts | repair | warning  | Duplicate key for record at 739736 against record at 735744 |
11:02:12 PM: Nate: | phpbb.phpbb2_posts | repair | warning  | Duplicate key for record at 739164 against record at 735172 |
11:02:12 PM: Nate: | phpbb.phpbb2_posts | repair | warning  | Duplicate key for record at 739772 against record at 735780 |
11:02:12 PM: Nate: | phpbb.phpbb2_posts | repair | warning  | Duplicate key for record at 739808 against record at 735816 |
11:02:13 PM: Nate: | phpbb.phpbb2_posts | repair | status  | OK                                                          |

[...]

11:02:13 PM: Nate: +--------------------+--------+----------+-------------------------------------------------------------+
11:02:13 PM: Nate: 144 rows in set (0.62 sec)
11:02:13 PM: Nate:
11:02:13 PM: Nate: mysql>
11:05:54 PM: Nate: 20829 rows before and 22170 now
11:05:59 PM: Nate: i'd say that's progress wouldn't you?
11:07:25 PM: Radix: hmm duplicates again


as radix learned more about how phpbb2 stores posts in its database, it occurred to him to copy the topics present in an old forum backup from 2003 into the file he was reconstructing. these topics had been deleted months earlier when someone accidentally turned on old topic pruning. now those topics could be restored along with the posts made after 1 may that radix was recovering from the trashed disk. being very protective of old topics on the forum, i was thrilled with his idea, and so we moved from simply recovering what marie had taken from us to not only recovering it, but actually making it better than it was before the incident.

Quote:
10:37:40 PM: Trebor: how much have we got back so far?
10:37:48 PM: Nate: we think we're in the middle of june
10:37:55 PM: Nate: but it's hard to tell
10:40:22 PM: Radix: i'm at the end of june
10:40:26 PM: Radix: because i saw the thread about bulleta's 55
10:40:29 PM: Radix: which was june 24
10:40:32 PM: Radix: so there can't be much left
10:40:33 PM: Trebor: ah, cool


finally, on the evening of 14 july, our long toil was nearly done:

Quote:
10:34:15 PM: Nate: mysql> REPAIR TABLE phpbb2_posts USE_FRM;
10:34:15 PM: Nate: +--------------------+--------+----------+----------------------------------------+
10:34:15 PM: Nate: | Table              | Op    | Msg_type | Msg_text                              |
10:34:15 PM: Nate: +--------------------+--------+----------+----------------------------------------+
10:34:15 PM: Nate: | phpbb.phpbb2_posts | repair | warning  | Number of rows changed from 0 to 22317 |
10:34:15 PM: Nate: | phpbb.phpbb2_posts | repair | status  | OK                                    |
10:34:15 PM: Nate: +--------------------+--------+----------+----------------------------------------+
10:34:15 PM: Nate: 2 rows in set (0.67 sec)
10:34:15 PM: Nate:
10:34:15 PM: Nate: mysql>
10:34:20 PM: Radix: woooooooooooooooooooo
10:35:14 PM: Radix: woooooooooooooooooooooooooooooooooooooooooooooooooootttttttttttttttttt


the database file containing the posts table was repaired successfully for the first time, meaning he'd done it - he'd taken the thousand pieces and put them back together exactly as they were before marie destroyed the filesystem on the disk. on top of that, he had successfully integrated the topics deleted in the unintentional forum prune without disrupting any of the topics made since.

now it was almost over. i hurriedly updated phpbb to version 2.0.9, which had been released since the forum was erased, and radix composed the first new post, giving a cursory description of the work that had gone on over the past two weeks.

to see the forum back better than it had ever been before was unreal to us, like some kind of apparition on the screen. what radix did was not supposed to be possible except in situations involving thousands of dollars in data recovery fees, and even then, i wondered if any data recovery firm in the world would have taken any amount of money to trudge through all of those tiny blocks like radix had:

Quote:
02:17:11 AM: Radix: I guess now I can put "data recovery expert" on my resume
02:17:20 AM: Nate: heh i was talking about that to dad
02:17:28 AM: Nate: i wonder how many people outside of mysql know what you know
02:17:31 AM: Radix: i lost count long ago...
02:17:37 AM: Radix: but i'll bet posts_text was near 1000 fragments
02:17:53 AM: Nate: yeah i was telling people in my family "over a thousand fragments"
02:17:54 AM: Radix: the very start had a 4mb fragment
02:17:58 AM: Radix: but after that it was mostly 4k/8k
02:18:04 AM: Radix: with some 24k for quote tunnels


with everything back in place and the ghost of marie buried forever, i took a moment to relax and think about where i wanted to take the reborn site. working in 'recovery mode' for so long had caused me to forget about what had been going on before the would-be apocalypse.

i knew, though, that i would never see the forum the same way after having watched what radix did for it and for the people on it. it was no longer just a place to post things, but a venerated meeting place with history as well as purpose. working on it now would be the highest privilege.

long live the metroid 2002 forum.
Thread title: 
Anywhere, everywhere
Quote from nate:
long live the metroid 2002 forum.

Hear, hear!!!

Just don't get hacked like scu...
I hadn't actually appreciated the volume of manual labour that went into the job before now, so really I should probably reiterate my first post on this forum.
Wow... I remember hearing about this when I first arrived here (Since I joined M2K2 in Aug 04, two months after this disaster ended) but I never realized the amount of work and dedication you two put in to bringing M2K2 back from the dead. I can't even begin to imagine how hard it was to put everything back tofether, and within two weeks of the site collapsing.  Shocked

As nate said...

LONG LIVE M2K2!!!
l'appel du vide
Word.  Great posts, Nate.
Too many fangirls to count
Wow, I just read all three parts. *pops back* wow I don't have anyone here's skills. Long live the forum. May my idiocy not corrupt it.
Haha, I thought Radix never recieved enough credit before, but after reading that...  Shocked  Shocked /me bows to Radix's determination

On another note, I'm just curious (re: you don't have to answer), but when was the last backup completed?
last backup of the forum? well, i usually make it a policy not to discuss security matters in public. so let's just say i'm most protective of areas where i've been hurt the worst.
Ah yes, I was just curious, and should have known it was a security issue.  Embarassed
Samus Lauren
Wow...And to think i was on my break from you guys during this...

Not like i wouldve known about all of THIS at the time...i never really pay attention to the technical aspects of keeping a forum and a site. But jeez...

...

*takes nate and hides him forever from marie*

So whats part four going to cover? I know i'm not going to be there, since dont technically show up till...early 2005ish...

...Wait, this is all on the speed demo history? Well then screw me ever being a part of THAT.  :P
:)

right now i'm thinking all the stuff that happened that made me become an sda slave, basically how i started capturing stuff outside of metroid, then started making the dvds, etc, so basically late 2004 partway through 2005. so except for metamorphosis (12 december 2004) i think the next part will mostly focus on sda.
Really nice job and everything, quite a feat. Only thing I don't get is why, no offense or anything but this forum's past isn't worth spending that much time on. :x
At the time, two months was a much larger chunk of the history (the forum was less than a year old at that time), and also, there were still a lot of important discoveries being made that were being announced and discussed here first, and that would have meant failing a lot of people to give up.
Really enjoyed reading through that, as well as the other two blocks. Think i'll have a look at the old gfaqs topics again.

Great stuff.
The people that made the discoveries would have memorized them if they were important enough. :P
Not about losing the discoveries themselves, but about first of all having to rewrite all of the "how to" text, and second, never knowing what the original discovery post looked like. I think it's the same reason why getting all the old GameFAQs topics back was such a huge thing for some people.
my umbrella goes directly to Bankai
whoa awesome rebuilding skills :O

Radix sure has many talents

i would have given up fast if that happened to me.

the history of this site never stops surprising me.
long live the metroid 2002 forum

Breath-taking, nothing else to say.
red chamber dream
I don't have anything to say that someone else hasn't, so yeah, that is quite amazing. Can't believe I joined just a few months after that all happened. o_O
(user is banned)
i had no idea. o_O

my question is, why did you guys REALLY fear marie/DJ after s/he/it deleted the... all? I mean, unless the person has absolutely NO LIFE, they wouldn't come and stalk the site waiting to delete everything again...

i know that I wouldn't... i dont care about people across the world (or even 300 kilometres south/north... well, 1,000 west i do, considering i know a person who just moved there, but still) think of me... they don't know who i am, what i look like, or if im even lying in their faces with them being oblivious. :P

but i had no idea that this happened, considering how nobody actually mentions it on the forums, and i dont go to this area normally. o_O


you could just make a "fake mod" position, and give it to people that you're suspicious of, that has absolutely no power, but it makes it seem like they do. :P


I'm never using the excuses "its a long story" or "you won't understand half of the shit thats coming out of my mouth" again (to you guys).  laugh new
unless you're a paranoid schizophrenic, trying to put yourself in its shoes probably will not help you understand.
(user is banned)
Quote from nate:
unless you're a paranoid schizophrenic, trying to put yourself in its shoes probably will not help you understand.



that, is probably a very good point.
Quote:
my question is, why did you guys REALLY fear marie/DJ after s/he/it deleted the... all?


You just got your server rm rfed and you don't know if the person who did it still has a grudge against you. :/

Not a good situation there...
Bangaa Bishop
Forgive me, but I just stumpled upon this and didnt actually read part 1 yet... so can someone summarize why exactly this person hated you and deleted the whole filesystem?

It looks like the first few lines say why but I dont quite get it.

I read part one, and I still dont get it. :(

-edit- NVM, this is part 3. Wow, i'm an idiot.

-edit again- wow..... now I understand....
Every Bit Counts
wow. I never knew this happened to this site, and would have never gussed it. when I read it, it seemed like a fictional story for a launguage arts class, olny it was real. just wow. I've always respected nate and Radix, but what they did for this site was just insane. and what does rm rf stand for? again, just wow.

long live the metroid 2002 forum