SoylentNews History Log
NCommander (Today at 1:02 PM)
@BlackCoffeeDrinker @Wiki Volunteer here, talking about how slashdot and soylentnews broke off, one of the major changes of the guard of the 90s
So, some context, Slashdot was formed in the very early 90s by CmdrTaco, and CowboyNeal, pretty much the first tech news of the early Internet
This was in the days of the dotcom boom, and it got handed over probably a dozen times; it was an Internet landmark, like SourceForge, or Tucows was
(RIP tucows)
(please ask questions, I'm recounting this all from memory, and some of this is a decade old)
Eventually Slashdot got acquired by DICE Enterainment, after a very long period of changing hands
Like all the original editors were gone
BlackCoffeeDrinker (Today at 1:05 PM)
slashdot's logo was /. because you know...
NCommander (Today at 1:05 PM)
http:///..com
emotional support mom friend (Today at 1:05 PM)
That's not confusing at all lol
NCommander (Today at 1:05 PM)
since browsers back in those days you needed to put http://
BlackCoffeeDrinker (Today at 1:05 PM)
I wanna say it was right-leaning? Hated microsoft and loved open-source
NCommander (Today at 1:06 PM)
Well, Gopher was still active, and you'd still frequently find news: and telnet: links
Like, I remember when those were on theopen net
NCommander (Today at 1:06 PM)
Much more liberarian than anything else
sirocyl (Today at 1:06 PM)
I remember when ftp:// was relatively common, too
NCommander (Today at 1:06 PM)
you mean today?
BlackCoffeeDrinker (Today at 1:06 PM)
Yes, news: was common
emotional support mom friend (Today at 1:06 PM)
B zhzhmfskgdzdhkzhldzlhdslhds
NCommander (Today at 1:06 PM)
Anyway
emotional support mom friend (Today at 1:06 PM)
More so than now?
sirocyl (Today at 1:06 PM)
my thoughts exactly
emotional support mom friend (Today at 1:07 PM)
Yranhfsksfumfsldgujzxgn
NCommander (Today at 1:07 PM)
DICE had made a lot of unpopular changes, but the straw that broke the camels back was "Beta"
sirocyl (Today at 1:07 PM)
oh no
BlackCoffeeDrinker (Today at 1:07 PM)
at some point users revolted because they didn't like changes
CEO of Email, Inc. (Today at 1:07 PM)
Never goes well
NCommander (Today at 1:07 PM)
Like it was this horrid whitespace meme; it turned a very information dense site into a wordpress block
no, it was just bad
sirocyl (Today at 1:07 PM)
it introduced gradients and rounded corners, too
NCommander (Today at 1:08 PM)
It was like if every page of the print New York Times was double spaced
sirocyl (Today at 1:08 PM)
it was awful
CEO of Email, Inc. (Today at 1:08 PM)
Did it go back then? I noticed Soylentnews looks the same as Slashdot does now
BlackCoffeeDrinker (Today at 1:08 PM)
yes, everything still loaded fine in netscape 3 before that
NCommander (Today at 1:08 PM)
Someone should look it up on Wayback
To think we need a screenshot of that
BlackCoffeeDrinker (Today at 1:08 PM)
let me check
sirocyl (Today at 1:08 PM)
damn archiveorg slow today
NCommander (Today at 1:09 PM)
Anyway, Slash (the underlying software) had been open source at one point, but had been left to die in a old CVS repo
(might have been git? Either way, it was heading towards death)
emotional support mom friend (Today at 1:09 PM)
I'm picturing the pharmacy here which makes it funnier
sirocyl (Today at 1:09 PM)
... wasn't git, I'm sure. was it?
BlackCoffeeDrinker (Today at 1:09 PM)
CVS i think
or SVN
CEO of Email, Inc. (Today at 1:09 PM)
Oh, so it got abandoned before you picked it up, then?
sirocyl (Today at 1:09 PM)
mercurial maybe?
CEO of Email, Inc. (Today at 1:09 PM)
The software I mean
NCommander (Today at 1:09 PM)
DICE was still using it, there are some trademark tells like the X-Fry/X-Bender headers
brb
NCommander (Today at 1:10 PM)
back
Regardless or not, enough was enough, so a bunch of us basically got together in IRC on Freenode
The channel was #alphaslash or something like that
The idea was to create a replacement. A few others had launched, like pipedot.org, which was basically a reimaginating with a new PHP codebase, and technocrat relaunched briefly
I think there was some efforts to see if kurosh5n could be risen from the dead
There were folks who remembered that Slash had been open source at once point, and the original archives were still up (it might still be there https://slashcode.com/)
CEO of Email, Inc. (Today at 1:13 PM)
Just redirects to sourceforge 😔
emotional support mom friend (Today at 1:13 PM)
:/
NCommander (Today at 1:13 PM)
and oh boy, Slash is a crankly SoB
This was the Perl CMS
CEO of Email, Inc. (Today at 1:13 PM)
Perl I'm guessing?
NCommander (Today at 1:13 PM)
the original
CEO of Email, Inc. (Today at 1:13 PM)
Yepp
Called it
NCommander (Today at 1:13 PM)
it required a Makefile.PL and has a CPAN bundle
CEO of Email, Inc. (Today at 1:13 PM)
🥖
NCommander (Today at 1:14 PM)
and was written in an era when CPAN expected to be installed as system libraries
But the worst part?
It used parts of mod_perl that aren't $r
oh god did it use parts. It basically kept a scorecard in Apache's shared memory space
to do synchrozation across tasks
it has a cron replacement written in Perl
BlackCoffeeDrinker (Today at 1:15 PM)
Someone that's good in perl wrote this
CEO of Fish Inc. (Today at 1:15 PM)
Horrifying
NCommander (Today at 1:15 PM)
and I mean daemon
Well, how it worked was insane
CEO of Email, Inc. (Today at 1:15 PM)
But why
NCommander (Today at 1:15 PM)
because it generated static pages for everything
CEO of Email, Inc. (Today at 1:15 PM)
Huh, I guess that was probably wise when it came out
NCommander (Today at 1:16 PM)
You basically had the #1 site on the early Internet, by being run by hobbyest with 1993 tech
CEO of Email, Inc. (Today at 1:16 PM)
Sounds like quite the feat honestly
emotional support mom friend (Today at 1:16 PM)
honestlu
NCommander (Today at 1:16 PM)
Like, I can't think of anything than Slashdot. It used to be whenever a server landed on Slashdot - BOOM
BlackCoffeeDrinker (Today at 1:16 PM)
Slashdot currently has 16 web servers all of which are running Red Hat 9. Two serve static content: javascript, images, and the front page for non logged-in users. Four serve the front page to logged in users. And the remaining ten handle comment pages. All web servers are Rackable 1U servers with 2 Xeon 2.66Ghz processors, 2GB of RAM, and 2x80GB IDE hard drives. The web servers all NFS mount the NFS server, which is a Rackable 2U with 2 Xeon 2.4Ghz processors, 2GB of RAM, and 4x36GB 15K RPM SCSI drives.
NCommander (Today at 1:17 PM)
Yeah
CEO of Email, Inc. (Today at 1:17 PM)
Well, that's where the "Slashdot effect" comes from
Which became the Reddit hug of death
Idk what the equivalent is now with widespread cdn use etc
NCommander (Today at 1:17 PM)
this was a codebase as written by the early wizards
not for elegance, but for pure functionality
Like, the only webapp that I can think is more hardcore is the original SF2.5 source code
BlackCoffeeDrinker (Today at 1:18 PM)
Besides the 16 web servers, we have 7 databases. They currently are all running CentOS 4. They breakdown as follows: 2 Dual Opteron 270's with 16GB RAM, 4x36GB 15K RPM SCSI Drives These are doing multiple-master replication, with one acting as Slashdot's single write-only DB, and the other acting as a reader.
emotional support mom friend (Today at 1:18 PM)
Wow
CEO of Email, Inc. (Today at 1:18 PM)
It's funny that they ended up in the same hands
(assuming you mean sourceforge?)
NCommander (Today at 1:18 PM)
We didn't have social media like that back then, it was basically articles, and AIM
NCommander (Today at 1:18 PM)
Yeah
BlackCoffeeDrinker (Today at 1:19 PM)
2 Dual Opteron 270's with 8GB RAM, 4x36GB 15K RPM SCSI Drives These are Slashdot's reader DBs. Each derives data from a specific master database (listed above).
Lastly, we have 3 Quad P3 Xeon 700Mhz with 4GB RAM, 8x36GB 10K RPM SCSI Drives which are sort of our miscellaneous 'other' boxes. They are used to host our accesslog writer, an accesslog reader, and Slashdot's search database.
16 + 7 + 2 + 3
NCommander (Today at 1:19 PM)
Remind me to document just how I ended up in Savannah documentation
But, let's get back to it
BlackCoffeeDrinker (Today at 1:20 PM)
Perl that had to manage running on 28 servers
NCommander (Today at 1:20 PM)
There had been a few efforts to try and get the original code running. It was "decently documented" for what was an internal codebase
But it was tied to Apache 1.3, it needed a very specific configuration of mod_perl, and more
And this was 2014(ish?), Apache 1.3 was dead and buried at that point
CEO of Email, Inc. (Today at 1:21 PM)
Were the undocumented parts things your people had to figure out themselves? Or were the original slash people still reachable somehow
NCommander (Today at 1:21 PM)
We had no help from the original founders
Anyway, I decided to take on the challenge
Back in middle school, I loved Slashdot, I even tried to run it ontop of Mac OS, going as far as learning how to compile Perl, and Apache, etc.
I never actually got it running, cause of CPAN problems, but I knew pretty much what it was going to take
CEO of Email, Inc. (Today at 1:23 PM)
Heh, your slash adventures started early, then
NCommander (Today at 1:23 PM)
I ran Linux on my macs before we got Mac OS X
I was in middle school? It was the greatest thing to do
I love understanding systems
CEO of Email, Inc. (Today at 1:23 PM)
I hear you there, it's fun and satisfying to understand
NCommander (Today at 1:23 PM)
Like, push comes to shove? I could probably do a LFS type system without the book
Anyway, getting off topic
CEO of Fish Inc. (Today at 1:24 PM)
This idea of using Perl to manage a website sounds horrifyingly similar to this ticketing application I helped support at work for a little bit
NCommander (Today at 1:24 PM)
I bought a linode instance, install MySQL
and started to climb that cliff
I compiled a freestanding version of Perl
BlackCoffeeDrinker (Today at 1:24 PM)
I know what you're talking about 😛
NCommander (Today at 1:24 PM)
(because only Python is worse about virtual environments)
Has to be a threading one
I compiled Apache 1.3, with some patching, mod_perl 1.3, and then began the grind of fixing the CPAN bundle
A lot of CPAN was already dying to atrophy by that time
CEO of Fish Inc. (Today at 1:25 PM)
So glad to know I’m not alone https://github.com/bestpractical/rt
NCommander (Today at 1:25 PM)
bugzilla is another old guard one
... mod_perl too, oof
BlackCoffeeDrinker (Today at 1:26 PM)
first job was rt + bugzilla + cvs
NCommander (Today at 1:26 PM)
Although not as bad as Slash
CEO of Email, Inc. (Today at 1:26 PM)
Wait, is Soylent still apache 1.x?
NCommander (Today at 1:26 PM)
no
that was the itch I couldn't let stand
Like
I said at the bginning "we're going to do that"
and that was the last major thing I did in that code base
CEO of Email, Inc. (Today at 1:27 PM)
Well, 2014. 1.3 would have been fucking ancient
NCommander (Today at 1:27 PM)
Yeah, I'll get back to that
By and large, we decided the risk was "acceptable" since we ended up putting the web frontends behind an nginx SSL terminator
oh god, I forgot about that
Slashdot used to make SSL a paid feature
BlackCoffeeDrinker (Today at 1:27 PM)
yes!
NCommander (Today at 1:28 PM)
We wanted to go HTTPS by default, the whole thing was a fucking passion project; it was going to be perfect
Let me find the articles
Like
BlackCoffeeDrinker (Today at 1:28 PM)
(before you say WHY... https used to be slow back then)
NCommander (Today at 1:29 PM)
I wrote about this
BlackCoffeeDrinker (Today at 1:29 PM)
(but also, /. didn't need https)
NCommander (Today at 1:29 PM)
It was considered to be high cost for CPU
BlackCoffeeDrinker (Today at 1:29 PM)
I remember we used to host the minimum amount over HTTPS
CEO of Email, Inc. (Today at 1:29 PM)
Understandable in a way, encryption is expensive
This was before knowledge of the NSA dragnets were public, and also, SSL was seen as relatively pointless by and large; you could still get away with a self signed certificate on the public web
It was really the bandwidth more than anything
Let me find these articles
NCommander (Today at 1:31 PM)
https://soylentnews.org/search.pl?op=stories&author=2
Everything I wrote on SN
Oh fucking hell I forgot about a lot of this
NCommander (Today at 1:31 PM)
https://soylentnews.org/article.pl?sid=14/02/16/2220240 - so it was a major meme on Slashdot how the antique the thing was
Like, we were going to one up old Slashdot so hard
BlackCoffeeDrinker (Today at 1:32 PM)
https://cdn.discordapp.com/attachments/1014650516854296697/1039970670257377290/image.png
lol
NCommander (Today at 1:32 PM)
That was actually a Mission Accomplished moment
Cause well
... UTF-8 has a lot of edge cases
When you need to protect against spam
I think we stlil filtered right to left /left to right markers
But I remember my personal driving energy was very much "/. didn't have it, we're doing IPv6"
@N's Life
(talking about how Internet history happened)
Anyway, the "lead" of the project was someone with the handle of Barbassas?
Fucking weird guy, like, he would quote machovili, and absolute control freak
emotional support mom friend (Today at 1:34 PM)
Ew
NCommander (Today at 1:35 PM)
Basically threatened to tank the project through mismanagement
I forced him out
I felt like a fucking ass, but I did most of the tech work, I had gotten the site running in the cloud
NCommander (Today at 1:35 PM)
We didn't have a name for it then, itwas litterly li69-342
or something like that
https://soylentnews.org/article.pl?sid=14/04/09/1925245 - oh god, the Slashdot port scanner
SN had code in the web frontend that it would actually portscan connecting hosts for adding to an internal block list
emotional support mom friend (Today at 1:36 PM)
I'm sorry, the what —why
NCommander (Today at 1:36 PM)
and yes, I did take advantage of the Unicode
emotional support mom friend (Today at 1:36 PM)
Of course you did
NCommander (Today at 1:36 PM)
CmdrTaco actually commented on it, it was an anti-spam measure
emotional support mom friend (Today at 1:37 PM)
Ah
NCommander (Today at 1:37 PM)
It made the top of hackernews
... like the first time I was on hackernews ...
... I didn't even register the signifance of that at the time
but this was a codebase
https://soylentnews.org/article.pl?sid=14/03/06/0728230 - I actually wrote up about John's departure
Like we were all on IRC at the time
John had us set up our own IRCd; I would have perferred to stay on Freenode
Slack was a thing at that point, but electron apps were really crummy then
emotional support mom friend (Today at 1:38 PM)
And it thought that was a download
NCommander (Today at 1:39 PM)
Look, some of us predate pretty URLs
emotional support mom friend (Today at 1:39 PM)
Gbargbsfhngdjgcn.
NCommander (Today at 1:39 PM)
I still feel kinda bad about it, but basically, the site hadn't an actual name at that point
We had promised to do a naming contest, there was no leadership
I was paying for it literially out of pocket on Linode
On my personal account
BlackCoffeeDrinker (Today at 1:40 PM)
$$$ much
NCommander (Today at 1:40 PM)
I was still at Canonical at the time I think, but I was right at the end
Yeah, it was pricy
John had actually blackmailed us
And sold the soylentnews.org domain without giving it to staff
Matt Angel bought it for 3k
https://soylentnews.org/article.pl?sid=14/03/10/1129232/
Right, we forced him out, and then he sold the domain
Well, I
fucking hell this is a story
Anyway, Matt basically was our venture captilist, and I was the CTO
And I basically said "we're going to incorporate this, and we're going to do it right"
and we did
Fitting, that we got our incorporation papers back on July 4th
https://soylentnews.org/article.pl?sid=14/07/04/2326238
Like, it's surprisingly hard to know how to make a corporation that has no physical assets
Like, its the definition of a .com company
https://soylentnews.org/article.pl?sid=14/07/16/0253256 - rereading this is a trip
Past N did a good job
... wow, I just proved that comments really do help the next person in the future
which uh ... was me ...
I think someone tallied it up? I had written like 100k words in the first two years. It was an absolute meme that the editors would fear whenever NCommander opened the editing console
and it was fast like, we had a private #chillax channel, and I say that "I'm going to write an article" and 3k words later would appear in 20 minutes
emotional support mom friend (Today at 1:47 PM)
And well, that checks out
NCommander (Today at 1:47 PM)
I should actually see if the quotes database is still up, I'm about 40% of #staff
But Matt and I got it incorporated; and there was a bunch of us out in the northeast so we had a major meet and greet day with a BBQ
and then I brought the party to everyone
https://soylentnews.org/article.pl?sid=14/06/17/0059210 this was one of the big things, like I had drafted up subscriptions, and then the pushback was really bad
Essentially, the original plan was to basically make something akin to SDF
All the editors had legit shell accounts, there was people.soylentnews.org
that actually wasn't an intentional wordplay
Debian has people.debian.org which is used to host personal projects on their namespace; Canonical had people.canonical.com
Looks like they revoked my SSH key; which is fine
Haven't been involved a very long time
sirocyl (Today at 1:51 PM)
oh my god
NCommander (Today at 1:51 PM)
like I didn't even notice it until jsut now
... right
Let ie be known that I can infact do it to myself
https://soylentnews.org/article.pl?sid=14/05/05/0541206
Like, if I was going to make a remake of the original Slashdot, I was going to make it a pinnacle to the gods
The li694-22 Domain
I've mentioned this on comments, and its on the wiki as well, but we use an internal gTLD for referencing nodes throughout the backend. Every node can access each other at hostname.li694-22. The name itself is a reference to the original private URL which we used for bringing up Slashcode way back before SN was decided as our temporary name. We have full forward and reverse resolution available, and only publish AAAA records for normal services. Oh yeah, about that ...
sirocyl (Today at 1:53 PM)
wait. shelter island?
NCommander (Today at 1:53 PM)
?
Linode
... wow
Is that actually the shelter island area code
Anyway, after incorporation, I had fairly large plans for Soylent, but that's when health things started to get worse
https://soylentnews.org/article.pl?sid=14/05/12/0335243
I had wanted to be able to do actual journalism instead of just a news aggitator
sirocyl (Today at 1:55 PM)
Image
wait, 11694 my b
emotional support mom friend (Today at 1:55 PM)
headpats
NCommander (Today at 1:55 PM)
Someone needs to remember to run /log when I finish
so the bot grabs it
sirocyl (Today at 1:56 PM)
Image
Rockaway Park
NCommander (Today at 1:56 PM)
Anyway, SN didn't quite pan out the way I had hoped
Like, I had to justify to the Internet why we were doing things, there was major concerns about freedom of speech
This was in 2014, the Snowden leaks had just happened
sirocyl (Today at 1:56 PM)
mmhm
NCommander (Today at 1:57 PM)
https://soylentnews.org/article.pl?sid=14/05/12/0335243
Wrong link
Like we ran our own Tor exit node: https://soylentnews.org/article.pl?sid=15/05/26/062240
It got a surprising amount of use
https://soylentnews.org/article.pl?sid=15/04/01/0212256 - the eBBQ
This was a milestone event. This was our first year up
Like, this was a celebration
I hope to play as a 24-hour event, from 0:00 to 23:59 EST, though that might change depending on how I'm feeling. I plan to setup a Skype session so members of the staff can join in and perhaps field a question or two. If anyone wants to live update for me (watch the stream and keep notes on what I've done) or something similar, drop me a Private Message on either the SoylentNews IRC channel or on Freenode (I'll be in #nethack while I'm streaming).
This was in the early days of Lets plays. Twitch was still pretty small ... I don't think I was at Beam yet
BlackCoffeeDrinker (Today at 1:59 PM)
justin.tv you mean
NCommander (Today at 2:00 PM)
No, it was twitch at this point
I say so as much in the post
This must have been right after I left Canonical, but before I got hired at MCProHosting
Or before they moved me over
There was like a 1-2 month period where I did actual sysadmin work
Note to streamers: For those watching via Twitch, there is a 20-30 stream delay, so if I don't immediately respond, assume that's the reason.
wow
I don't know if that was foreshadowing or not
... I might have been at Mixer at that point?
BlackCoffeeDrinker (Today at 2:01 PM)
20 to 30 !?!
NCommander (Today at 2:01 PM)
Or it was a sure thing
NCommander (Today at 2:01 PM)
That was the average
BlackCoffeeDrinker (Today at 2:01 PM)
I never realize it was that bad
NCommander (Today at 2:01 PM)
It could go minutes long unoptimized
Packet delay is cummative over the life of the stream
BlackCoffeeDrinker (Today at 2:02 PM)
I rememeber that long stream had A/V desync
NCommander (Today at 2:02 PM)
Yeah I think I was already thinking about latency at this point
What got me hired at MCProHosting was a discussion with Matt the CEO
Who was on my hiring call
... I should have questioned that at the time ...
and Matt had asked me questions about it; I had listened low latency experience on my resume
It was in terms of system processing, not networking, but "eh, close enough?"
BlackCoffeeDrinker (Today at 2:04 PM)
HAHAHAHHAHA
NCommander (Today at 2:04 PM)
I mean, I didn't know how different they were at the time
I understood things like packet delay due to TCP/IP
But I never really thought about it beyond that.
I should give some context here
In system processing, latency is generally in reference to what is known as realtime processing
Basically, it means you need to do X amount of work in Y amount of time
Most computing is not realtime, but its fast enough for humans not to notice
i.e., keyboard entry is not real time
The big SoB being audio
BlackCoffeeDrinker (Today at 2:06 PM)
DMA Audio was a big deal
NCommander (Today at 2:06 PM)
Video you can get away with more
(I think this is worth talking about, even though its off topic)
BlackCoffeeDrinker (Today at 2:07 PM)
Video was so unobtanium that a different approach was needed
NCommander (Today at 2:07 PM)
Well, by and large, it was geared toward animation, you had Smacker, and then compositing
Kodak used to charge a very pretty penny to make high quality analogue coversions
fuck, I don't think I ever brought up Castle Infinity
I haven't even thought about them in ... has to be decades
Uh, that was the period of my life when you can literially say I had the job title "Architect of Infinity"
BlackCoffeeDrinker (Today at 2:08 PM)
the MMO ?
NCommander (Today at 2:08 PM)
Yeah
I didn't play it back in the Starwave days
But the servers had gotten tossed out
Literially tossed out
One of the diehard fans, Kevin Quilt, dumpster dived for them
and put the game online
They had full source code, and gotten a legal agreement out of Starwave to run it
(it might have been Go? They got bought by Disney)
The game was made by Starwave, then owned by GO, and then Disney
It was a 9x era MMOG, and everything that implies
The original thing ran on NT 3.51, and the client on 9x, and it abused things like IPC and COM
and it did *cute coding
BlackCoffeeDrinker (Today at 2:10 PM)
😭
NCommander (Today at 2:11 PM)
case 'ABCD': /* this compiles, because a four byte char can be cast to an int, and VC++6 will not complaint*/
It had this horrid heap manager from a third party company, it used RAD game tools, and a few other bits
BlackCoffeeDrinker (Today at 2:11 PM)
HAHAHAHA AHAHAHAHAH AHAHAHHAHA YES !! YES!!
NCommander (Today at 2:12 PM)
The level editor was a MFC application
oh god, SP4k
BlackCoffeeDrinker (Today at 2:12 PM)
God VC++6 was amazingly bad
NCommander (Today at 2:12 PM)
that ... that thing was bad, it was a C++ codebase from hell
Because the code started life on VC++4, been poorly migrated to VC++6, and then I kicked it screaming to VC++2005
SED (Today at 2:12 PM)
Like Hammer:blep:
BlackCoffeeDrinker (Today at 2:12 PM)
VC++2005 didn't compile what VC++6 did...
how did they migrate that ?
NCommander (Today at 2:13 PM)
How did I migrate it
BlackCoffeeDrinker (Today at 2:13 PM)
Poorly ?
oh
NCommander (Today at 2:13 PM)
well, it still takes case syntax
No, it was still on VC++6 when I got there
I was in high school at the time
Actually, remind me to come back to castle infinity @N's Life, let me finish SoylentNews
BlackCoffeeDrinker (Today at 2:13 PM)
I did my final school project on VC++6
NCommander (Today at 2:14 PM)
Because I want to recount my last bit
emotional support mom friend (Today at 2:14 PM)
So, that first, then infinity
NCommander (Today at 2:14 PM)
This was the last major coding exercise
https://soylentnews.org/article.pl?sid=15/03/13/044211
We had called our branch fo Slash, slashcode 15.03
(you can see the Ubuntu influence)
We didn't do timed releases, I just gave them version numbers to tag
The plan was to do a final Apache 1.3 release
I had gone into what can be described as a manic mood
This was about a year after we launched
And I was going to get us off Apache 1.3, and onto 2.0
(2.2 didn't have all the CPAN modules we needed)
This is not soething you do lightly
Like
this is a fucking effort
mod_perl was wrapped around rehash's codebase like a cancer
And it used a global state table and an API that didn't exist in 2.0
Most of the codebase was clean, the slashd daemon ran without the mod_perl components, but the entire frontend was dependent on it
I had ripped a lot of the static code generators out; it was 2015, we could afford the processing port
It's a thing of beauty when you delete a lot of dead code
Like, I think I purged a good 40k LoC from that beast
and we had a commitment not to use frontend javascript
It was a testimonent to the old Internet
unless its changed, privacy badger was still 0
Back at that time, I actually considered leaving NCommander as a handle to whomever replaced me
Ala dread pirate roberts from the movie, the Silk Road hadn't happened yet
We had a development box, lithium
All the machines were named after elements, starting at hydrogen, and going up
Canonical actually used the same naming scheme
The cdimage build master was called antimony, which was appropiate
To even start porting the monster, I had to sort out the mess of CPAN dependencies
and I still had to have a self-contained Apache build system
https://github.com/SoylentNews/rehash/blob/master/Makefile#L341 - I wrote a massive makefile to do it
Including patching a few CPAN bits to work with mod_perl 2
Well, the makefile was already there
but I really made it there
This was going to be a reproducible build damn it
I also think we migrated to MySQL cluster at the same time
Like
holy fuck
what was I thinking
A little known feature of the codebase is that its built on a plugin architecture which makes expansion easier and allows for multi-hosting. That is to say, we can have multiple sites out of a single installation; i.e., soylentnews.jp and soylentnews.org can both point to the same installation, but show a different mainpage. While we're still somewhat far off from supporting this kind of user-generated content, this upgrade sets the framework for allowing it to happen. The mainpage of SN will remain as it is, but allow the community to form and discuss any topic they wish, all handled under the same familiar interface you use now.
Actually, this is a bit important, there were other slashdot forks, there was slashdot.jp, and barrapuento.com, both on the original Slash code, I think there was a few others, but those were the two I remmber trying to reach
https://soylentnews.org/article.pl?sid=15/05/19/0740226 - we even discucssed i18n/l10n efforts
https://soylentnews.org/meta/article.pl?sid=15/06/01/0657256 - fucking hell
I actually did do both at the same time
Rewrote large amounts of the site to migrate to Apache 2, mod_perl 2, and perl 5.20.
This was a massive undertaking. I did a large part of the initial work, but paulej72, and TheMightyBuzzard did lots to help fix a lot of the lingering issues. Major props to Bytram for catching many of the bugs pre-release
Nexus Support (finally).
Currently we have the Meta and Breaking News nexii, with the possibility of adding more in the future, such as a Freshmeat replacement.
Nexii can be filtered in the user control panel under the Homepage tab. At the moment, this functionality is hosed due to unexpected breakage, but should be functional within the next 24-48 hours
IPv6 support - the AAAA record is live as we speak
Themes can be attached to a nexus independent of the "primary theme" setting; user choice overrides this
Squashed More UTF-8 Bugs
Migration to MySQL Cluster (more on this below)
Rewrote site search engine to use sphinx search and (in general) be more useful
Long comments properly collaspe now
Support for SSL by default (not live yet)
Fault tolerance; the site no longer explodes into confetti if a database or webfrontend goes down unexpectedly; allows for much easier system maintenance as we can offline things without manual migration of services
Improved editor functionality, including per-article note block
Lots of small fixes everywhere, due to the extended development cycle
Like, I did IPV6, mod_perl 2, and database clustering on a single upgrade cycle
and I mean
it worked
@BlackCoffeeDrinker what ... how did I do this?
emotional support mom friend (Today at 2:26 PM)
Not something you'd recommend but it did work apparently
NCommander (Today at 2:26 PM)
you should realize the sheer absurdity of that changelog
That's the fucking marketing statement, not the actual engineering changelog
emotional support mom friend (Today at 2:27 PM)
O.o
NCommander (Today at 2:27 PM)
In our year and a half of dealing with slashcode, we had also identified several pain points; for example, if the database went down even for a second, the site would lockup, and httpd would hang to the point that it was necessary to kill -9 the process. Although slashcode has support for the native master-slave replication built into MySQL, it had no support for failover. Furthermore, MySQL's native replication is extremely lacking in the area of reliability. Until very recently, there was no support for dynamically changing the master database in case of failure, and the manual process is exceedingly slow and error prone. While MySQL 5.6 has improved the situation with global transactions IDs (GTID), it still required code support in the application to handle failover, and a specific monitoring daemon to manage the process, in effect creating a new single point of failure. It also continues to lack any functionality heal or otherwise recover from replication failures. In my research, I found that there was simply bad and worse options with vanilla MySQL in handling replication and failover. As such, I started looking seriously into MySQL Cluster, which adds multi-master replication to MySQL at the cost of some backwards compatibility.
OH GOD I FORGOT ABOUT THIS
Right, so because the code was so crankly, if the MySQL daemon ever went down
Everything else would lock up
And I mean, we would have to killall -9 httpd
and this happened fairly often because Linode used to restart boxes regularly for downtime
BlackCoffeeDrinker (Today at 2:28 PM)
Indeed how
are you like 5 ppl in one ?
NCommander (Today at 2:28 PM)
https://soylentnews.org/meta/article.pl?sid=16/08/25/073218 - I'm a fucking overachiever
BlackCoffeeDrinker (Today at 2:29 PM)
Those weren't established standards
emotional support mom friend (Today at 2:29 PM)
Did…that got fixed, right?
NCommander (Today at 2:29 PM)
Well
I fixed it by making the database not go down
ever
emotional support mom friend (Today at 2:29 PM)
I mean that works
NCommander (Today at 2:29 PM)
Like, the fucking site had to have a 4 nines uptime
BlackCoffeeDrinker (Today at 2:29 PM)
Very nice
NCommander (Today at 2:29 PM)
And for a legacy mod_perl site
in an era before containers or dockerization
Kubernetes hadn't been made yet
Docker still was in its infancy
BlackCoffeeDrinker (Today at 2:30 PM)
we did have red/blue where we'd ship a server with the next version to replace the server that was in prod
NCommander (Today at 2:30 PM)
Like, I had to rewrite the etnire site search engine to do that migration
The original search engine used MySQL FULLTEXT tables
I rewrote it to run sphinx
You'd probably use Elastasearch these days, but ES is the cloud sucks
BlackCoffeeDrinker (Today at 2:31 PM)
Word tables
NCommander (Today at 2:31 PM)
The primary cause of the slowdown was due to the fact that rehash did large JOIN operations on text columns in MySQL. This is bad practice in general due to performance reasons, but it causes a drastic slowdown with MySQL cluster, which prevents the query optimizer from doing what's known as a "pushdown", and allowing the query to execute on the NDB nodes. This caused article load to be O(n*m), where n was the number of articles in the database and m was the number of articles with the neverdisplay attribute set. The revised queries now load at O(1). Instead it had to do multiple pulls from the database and assemble the query data on the frontend, a process that took 4-5 seconds per problematic query. The problem was compounded that there are limited number of httpd daemons at any given moment, and any database pull that hit a problematic query (which were in index.pl and article.pl) would cause resource exhaustion.
Like
THAT WAS A CRANKLY MIGRATION
June 02 2015
... WTF, I WAS DOING FTL AT THE SAME TIME
CEO of Fish Inc. (Today at 2:33 PM)
HOW
how did you have the time
NCommander (Today at 2:33 PM)
https://github.com/microsoft/ftl-sdk/commit/09fc8aadc86a7fe0d99c9a997b9a7198333bc013
no, I didn't do it yet
emotional support mom friend (Today at 2:33 PM)
What weren't you doing at the time?
NCommander (Today at 2:33 PM)
This was right in the Canonical/FTL gap
This was the last passion project, I wanted SN to succeed and to be able to live on this; the job at MCProHosting was the easy paycheck
Yeah
and it was the thing I promised I do
The last thing I said to the community that we were going to modernize this codebase
emotional support mom friend (Today at 2:34 PM)
And well
NCommander (Today at 2:36 PM)
Yeah
I honestly remember being utterly disappointed at the time that it didn't go smoothly
The site was migrated over a 2 window period, and then we had about two days of intense debugging
and about another month of smacking the gremlins out
It was like 200k loc when I got it?
I think I cut about 75k loc out of it
Just a lot of dead code
But that was a literial trial by fire
What I didn't realize at the time was hydrogen's earlier failure had not been resolved as I thought, and it gave truly abysmal performance, with 10+ second page loads. As soon as this was realized, I quickly pressed fluorine, our 'normal' frontend server into service, and site performance went from horrific to bad. A review of the logs showed that some of the internal caches used by rehash were throwing errors; this wasn't an issue we had seen on dev, and such was causing excessive amounts of traffic to go to the database, and causing Apache to hang as the system tries to keep up with the load. Two hours of debugging have yet to reveal the root cause of the failure, so I've taken a break to write this up before digging into it again
OH I REMEMEBR NOW
Ok, so the problem was one of the two database nodes was in a failed state
I thought it had been resolved, so I had ended up restoring the DB back into a flakely node which then failed hard
So I was trying to get a cluster running with only half its brain
sorry, frontend
flourine was the frontend
hydrogen and flourine
We had been in a 1x2 configuration, one web frontend, two database nodes
I had put hydrogen back into the service, but due to very very heavy caching with varnish and memcached, we weren't seeing the lag
rehash had modified a lot of the rendering paths so the site would be more dynamic
the cost of processing a request is cheap if you never hit the disk
... I think that's what inspired FTL
BlackCoffeeDrinker (Today at 2:41 PM)
^^
NCommander (Today at 2:41 PM)
... I actually understood the rules of the game even at that point, I wanted to prove you could do it
BlackCoffeeDrinker (Today at 2:41 PM)
abuse cache, never the disk
NCommander (Today at 2:41 PM)
when you can treat the network as reliable
BlackCoffeeDrinker (Today at 2:41 PM)
the Database should be Write only
NCommander (Today at 2:42 PM)
Because each request would enter nginx load balancer, get passed back to an application server, and our dataset was small, only a few gigs
put the whole thing in memory
And I mean the entire blasted database
That's how mysql cluster works
BlackCoffeeDrinker (Today at 2:42 PM)
one should only read the DB when the server boots and then only write changes to it - never hit the db
NCommander (Today at 2:42 PM)
Yeah, that's how memcached worked
BlackCoffeeDrinker (Today at 2:42 PM)
yup
NCommander (Today at 2:43 PM)
And to prevent a bottleneck, each web frontend could connect to any master, because we had no serialization concerns
BlackCoffeeDrinker (Today at 2:43 PM)
Designed enough system to have that drilled in my head
NCommander (Today at 2:43 PM)
Well, we needed read isolation, but any database has that
BlackCoffeeDrinker (Today at 2:43 PM)
mysql didn't always have it
NCommander (Today at 2:43 PM)
if you understand the rules of the game, you can break them
Like, SN gets. apretty good amount of traffic
oh fuck, alexa is gone
but we were up there
https://soylentnews.org/hof.pl - like, you can see it
800+ comments, on a article from 2022 https://soylentnews.org/meta/article.pl?sid=22/07/09/0547216 (incidently, addressing one of the main issues of why I essentially faded out)
and it ran on five relatively small Linodes,
Whole thing used centralized kerberos and hesiod database
We were on LDAP
https://soylentnews.org/meta/article.pl?sid=15/07/13/0255214
https://wiki.soylentnews.org/wiki/The_Slashcott - oh the wiki has useful info
The Slashcott
Here's the original Slashcott
I mean
People were fucking pissed about the state of slashdot
https://arstechnica.com/information-technology/2014/02/slashdots-new-interface-could-kill-what-keeps-slashdot-relevant/ - THANK YOU ARS
THERE IT IS
https://cdn.discordapp.com/attachments/1014650516854296697/1039989964831719514/unknown.png
BETA
^- @Wiki Volunteer @N's Life this is what we revoked against on Slashdot. the Beta interface. Make sure this gets saved
BlackCoffeeDrinker (Today at 2:49 PM)
OH GOD I REMEMBER THAT
NCommander (Today at 2:49 PM)
shit, I almost think we should have made this a seperate channel
Damn it
fuck it
We don't get enough discussion here, and we'll pin the start point
and log it with the bot
https://slashdot.org/submission/3326505 - YES
Trying to figure out time zones is starting to make my brain hurt, but apparently in a bit over 6 hours somewhere on the other side of globe from Greenwich the Week of Slashcott will begin, as Midnight arrives for anyone in that zone, and then it travels west, where I will encounter it in about 23 h...
THIS IS WHERE IT ALL STARTED
I mean, look at the old UIDs that came out of the woodwork
AKDT!
https://cdn.discordapp.com/attachments/1014650516854296697/1039990736680128572/image.png
NCommander (Today at 2:55 PM)
https://wiki.soylentnews.org/wiki/NewName - this is how the name came to be
and we did a poll by email, automated, with GPG keys to vote on it
like
... hotdam
https://wiki.soylentnews.org/wiki/Confessions_of_an_Ex-Slashdot_Beta_User there were people who were really really fucking pissed about beta
sirocyl (Today at 2:56 PM)
wow
NCommander (Today at 2:57 PM)
https://wiki.soylentnews.org/wiki/Archive - this was the prelaunch archive
The wiki was the first thing to actually get online
Because no one knew how we were going to replace Slashdot
it wasn't clear if running slashcode was even practical; I remember the general sentiment being it wasn't possible
I wasn't the only one who did it, robinld got it working on RHEL ... I guess it would have been 6?
I wiped the database once in pre-launch stuff, like it had been up privately for a few days on golive
and there were a lot of notes, I regret that in hindsight
I think I did it because no one really expected a bunch of development notes to be published publicly
I very much set the standard for transparency
https://soylentnews.org/meta/article.pl?sid=17/04/20/1055218 - not all of them were good decisions
I think we only migrated two machines
Getting moderation to work was hard
Like really hard https://soylentnews.org/article.pl?sid=14/02/13/0412219
It basically assumed there was tens of thousands active users
@BlackCoffeeDrinker https://soylentnews.org/~NCommander/journal/1096 - this is incredible
Like
I can feel the black magic
https://soylentnews.org/~NCommander/journal/1032
and just say I'm a certifiable madman.
.... AHHHHHHHHHHH
I fucked pwned myself across space and time
I actually wrote a postmodern https://soylentnews.org/meta/article.pl?sid=15/06/18/0045246
... ok then ...
um
... you know, I'm going to eat something
Folks, ask me questions
because uh ...
I don't know what I just read, and I wrote it
Sourdough and spam is amazing, change my mind
Maybe should have had some godha to go w/
damn now I raelly want crackers with gohda
I think SoylentNews, and FTL (which immedately followed) was basically an end result of being so fed up with Canonical that I needed to do something functional
to prove that I could
I mean, there was nothing saying I had to do this, I wasn't paid
I mean, we brought in about 5k per year, which for a site that runs no adversiting
is obscene
like its litterally a text aggreator
NCommander (Today at 3:18 PM)
https://soylentnews.org/communityreviews/article.pl?sid=18/10/25/0850204
The other is what I'm calling tentatively calling Root Zone in a Box, a series of shell scripts, instructions and docker containers to automatically recreate a simulation of the DNS root zone, and other core internet functionality to allow testing of potential changes to DNS, as well as help study and debug various issues related to Internationalized Domain Names.