We are in control.

Is it just me, or did other people read this and immediately imagine ita doing an evil laugh?

Not just you.

Not just you. Although it was the "we don't need Perl here" that really sent a shiver up my spine.

It's all about the power.

Ooh, so you could say "hmm, I'm on post 40, and there are 300 more posts. I want to see post 40-150 on one page"? That would be SO COOL!

There's a fundamental point that needs explaining here -- I might err on the side of the patronising here, but it's in a good cause! -- we control this place. It's ours!

With TT/WX, there was a database behind the scenes, which we could only interact with by clicking on things and hoping the right stuff came out. The database was their secret, and the only way we could get stuff out of it was with a browser.

The Perl script I wrote, and the search engine someone (Tom?) wrote, and the number-of-times-we-mentioned-porn counts, were all done by imitating a browser and getting the stuff from their database.

It's like when you go to buy drugs from one of those places where there's a steel-reinforced door with a small hole in it. You shove your money through the hole, ask politely and hope for the best. You've got no idea what's going on. Someone can take your money and give you nothing in return, take your money and give you baking soda in return, take your money and shoot you, or just plain not take your money at all.

So when we create tools to grab content from WX or TT, we're the customer.

On the Phoenix board, we're the dealer as well.

We've got our people on the other side of the door.

So, when people plaintively, endearingly, wistfully ask "would it be possible, maybe, one day, to make a search engine which searched this site? Please?" -- yes, you should imagine ita, and indeed everyone else doing an evil laugh of total mastery, because yes, we can do it right away, in fact it's already done!

yes, we can do it right away, in fact it's already done!

Umm, no?

I wrote the search engine. It took me time. It was not a done deal, just because we run the database. I had just written it before anyone asked.

I wrote the search engine. It took me time.

I'm sorry, I didn't mean to diminish the work you'd done.

But if someone wanted to know some searchengine-type thing right away and it was important, you could actually log in and get it from the command-line just with a one-liner, couldn't you?

Fundamental point still stands. The workings of this site, the database contents, the storage setup are not hidden from us the way they used to be.

To write Mister Pointy, I had to get all the transcripts one by one from the site in Germany, and do some complicated scripting on them to count all the words, sort the counts, list the sorted counts by document name, blah blah blah, and why? So I could reduce the contents of those pages to something I could put into a database -- because a database is the best way to look stuff up quickly ... here everything's already in a database.

if someone wanted to know some searchengine-type thing right away and it was important, you could actually log in and get it from the command-line just with a one-liner, couldn't you?

Well, yeah, but the premise behind constructing and executing a single SQL format and coming back and telling the results isn't the same as making a search engine, which is what your post described.

We have more direct access to the data, but it would be misleading to imply that adding a tool is trivial. Jon B designs the HTML, we go back and forth while I get it right, it gets fully integrated with the existing template, etc, etc.

Sure it can (and probably will) be done. We're out of hacking, is all, and into production.

Again, sorry to make it seem like it was trivial to do all the wonderful work you've done, and I know its not -- all I wanted to do is make people aware that we finally have a board where the workings are not obscure to us.

This should really be divided into two parts:

• People are asking for cool site features. They should respect the fact that those features will take work to implement.
• They're asking for them in a way that almost makes me sad, because they seem not to understand the complete coolness of what you've already done, that we have taken control of the data once and for all.

So plaintively asking "can we have a search engine?" or any other feature, makes me want to do the evil laugh thing -- of course we can, because we own the data and we know where everything is.

"Can we have one right away?", different question.