A modest suggestion

Surely it has occurred to many of you that it
might prove useful & informative to collect
all the digests of this forum in a single
compressed rapidly searchable file.

80 kilobytes per tuning digest * 3000 digests
= 240 megabytes.

Having no experience in the creation of ultra-
large searchable text databases, it is not obvious
into which format (presumably compressed)
the entire transcript of this forum might be
converted so as to speed a keyword search.
Presumably plain ascii is not on.

Has anyone specific expertise in this area which
they might share with us?

Is the entire transcript of this forum already
available in compressed readily searchable
(not plain ascii txt) format as a single
downloadable file, rather than the 50,000
individual messages currently residing on
yahoo's servers?

If not, might I suggest this as a collective
project which might prove worth peoples' time?

________________________________________________________________________
Yahoo! Messenger - Communicate instantly..."Ping"
your friends today! Download Messenger Now
http://uk.messenger.yahoo.com/download/index.html

>Surely it has occurred to many of you that it
> might prove useful & informative to collect
>all the digests of this forum in a single
>compressed rapidly searchable file.

It has indeed.

>80 kilobytes per tuning digest * 3000 digests
>= 240 megabytes.

And plaintext compresses quite well.

>Having no experience in the creation of ultra-
>large searchable text databases, it is not obvious
>into which format (presumably compressed)
>the entire transcript of this forum might be
>converted so as to speed a keyword search.
>Presumably plain ascii is not on.

XML seems like an obvious choice.

One party who has experience in the creation of
ultralarge searchable databases is Google, and if
you use site-restricted search to groups.yahoo.com
you can do ok, if you know what you're looking for
and you don't need to go much earlier than 1999.

>Has anyone specific expertise in this area which
>they might share with us?

Not I.

>Is the entire transcript of this forum already
>available in compressed readily searchable
>(not plain ascii txt) format as a single
>downloadable file, rather than the 50,000
>individual messages currently residing on
>yahoo's servers?

For a while I was making available all the digests
I had as a single 25MB zipped text file. I may make
that available again sometime.

>If not, might I suggest this as a collective
>project which might prove worth peoples' time?

It's been suggested before, and Robert Walker took
up the task, but his project was marred by the fact
that he only included the posts of people who
explicitly gave him permission.

-Carl

--- In tuning@yahoogroups.com, Jean Lawton <mlkofmusic@y...> wrote:

> Surely it has occurred to many of you that it
> might prove useful & informative to collect
> all the digests of this forum in a single
> compressed rapidly searchable file.

It only takes a few minutes to search all of it using the search
feature on the list's website.

--- In tuning@yahoogroups.com, "Carl Lumma" <ekin@l...> wrote:

> >If not, might I suggest this as a collective
> >project which might prove worth peoples' time?
>
> It's been suggested before, and Robert Walker took
> up the task, but his project was marred by the fact
> that he only included the posts of people who
> explicitly gave him permission.

This involved (at least the one I'm aware of) the *mills* tuning
digests, the ones *prior* to what exists on this website (which is
all of the onelist/egroups/yahoogroups posts). Hopefully we could get
more people to give their permission so that Robert's mills archive
would be more complete; right now you can only see about 6 members'
posts.

>> Surely it has occurred to many of you that it
>> might prove useful & informative to collect
>> all the digests of this forum in a single
>> compressed rapidly searchable file.
>
>It only takes a few minutes to search all of it using the search
>feature on the list's website.

Which means it is impossible to use said search feature for
interactive learning.

-Carl

>This involved (at least the one I'm aware of) the *mills* tuning
>digests, the ones *prior* to what exists on this website (which is
>all of the onelist/egroups/yahoogroups posts). Hopefully we could get
>more people to give their permission so that Robert's mills archive
>would be more complete; right now you can only see about 6 members'
>posts.

I believe all of the Mills posts are available on an ftp server
over there somewhere. Does anyone remember the URL?

It has been established time and time again that searchable archives
serve the public to the extent that, at the very best contributors
have the right to have their contributions *removed* upon request.

-Carl

--- In tuning@yahoogroups.com, Carl Lumma <ekin@l...> wrote:
> >> Surely it has occurred to many of you that it
> >> might prove useful & informative to collect
> >> all the digests of this forum in a single
> >> compressed rapidly searchable file.
> >
> >It only takes a few minutes to search all of it using the search
> >feature on the list's website.
>
> Which means it is impossible to use said search feature for
> interactive learning.

Please elaborate on what you're thinking here.

--- In tuning@yahoogroups.com, "wallyesterpaulrus" <paul@s...> wrote:

> It only takes a few minutes to search all of it using the search
> feature on the list's website.

And you trust Yahoo to always make this so?

>> Which means it is impossible to use said search feature
>> for interactive learning.
>
>Please elaborate on what you're thinking here.

People use various strategies for managing information. One
of them is search-and-find -- what I was calling "interactive
learning" here. It requires *instant* search before it is
viable. Google makes it viable to use the web like this, and
if you've mastered it you can be what librarians 10 years ago
would have called a "god".

An alternative strategy is to make a hierarchy of links. In
the case of the web, making bookmarks and putting them into
categorized folders is another strategy.

Anyway, this is all HCI, and off-topic.

-Carl

--- In tuning@yahoogroups.com, Carl Lumma <ekin@l...> wrote:
> >This involved (at least the one I'm aware of) the *mills* tuning
> >digests, the ones *prior* to what exists on this website (which is
> >all of the onelist/egroups/yahoogroups posts). Hopefully we could
get
> >more people to give their permission so that Robert's mills
archive
> >would be more complete; right now you can only see about 6
members'
> >posts.
>
> I believe all of the Mills posts are available on an ftp server
> over there somewhere.

No, that's only about the first year, all before I joined.

>> I believe all of the Mills posts are available on an ftp server
>> over there somewhere.
>
>No, that's only about the first year, all before I joined.

If you happen to have the url maybe I can gank them if it's still
up. I could then combine them with my Mills archives, and see if
that amounts to anything.

-Carl

--- In tuning@yahoogroups.com, Carl Lumma <ekin@l...> wrote:

/tuning/topicId_51788.html#51826

> >This involved (at least the one I'm aware of) the *mills* tuning
> >digests, the ones *prior* to what exists on this website (which is
> >all of the onelist/egroups/yahoogroups posts). Hopefully we could
get
> >more people to give their permission so that Robert's mills
archive
> >would be more complete; right now you can only see about 6
members'
> >posts.
>
> I believe all of the Mills posts are available on an ftp server
> over there somewhere. Does anyone remember the URL?
>
> It has been established time and time again that searchable archives
> serve the public to the extent that, at the very best contributors
> have the right to have their contributions *removed* upon request.
>
> -Carl

***You know, it seems like somebody nowadays with a good knowledge of
XML could probably figure out how to transcribe this entire list into
a large text archive, or some such, for safekeeping. I've done a
little work with XML, but not enough to consider something like this,
though... :(

J. Pehrson

--- In tuning@yahoogroups.com, Carl Lumma <ekin@l...> wrote:
> >> I believe all of the Mills posts are available on an ftp server
> >> over there somewhere.
> >
> >No, that's only about the first year, all before I joined.
>
> If you happen to have the url maybe I can gank them if it's still
> up. I could then combine them with my Mills archives, and see if
> that amounts to anything.
>
> -Carl

http://www.mills.edu/LIFE/CCM/ftp/tuning/list/archive/

from

http://www-math.cudenver.edu/~jstarret/resources.html

>http://www.mills.edu/LIFE/CCM/ftp/tuning/list/archive/
>
>from
>
>http://www-math.cudenver.edu/~jstarret/resources.html

Thanks. I'll see what I can do.

-C.