subnet info, was Re: [HECnet] _PROVISIONAL_ map of HECnet, courtesy largely of Brian H.

[Johnny Billquist]

On 2013-01-08 17:15, Brian Hechinger wrote:
On 1/8/2013 9:24 AM, Johnny Billquist wrote:
Data mining is difficult, since there are different systems, with
different possibilities of extracting it, and in different formats.

Sampsa's goal here is mapping HECnet. My goal is to write a data mining
service that just happens to provide the data that a mapper would use.

To that end, I'll be stashing all this data in a database of some sort
(details unknown as yet, see below)

Right. So, once we have the data, it can be used by other people in various ways. So lets focus on the data.

A centralized repository of data is nice in many ways, but it is a
headache to manage.

Absolutely it is, but if people are already putting INFO.TXT files out
there they are doing 99% of the work already, we just need to get the
data in a single place.

I think it's better to keep the information separate. INFO.TXT was created for one purpose, this is trying to reuse it for another, additional purpose. Better to create a separate file. In addition, we can make some better design choices as we go about this.

That said, I could be convinced of setting something semi-automatic
up. A reasonable way would be for people to give me machines to poll,
and then I'd setup an automated process to poll those machines for
files in a specific format. I can then create a database out of that,
and make it available through the web, as well as over DECnet, and
also as a summarized file. Anything would be pretty easy if we just
have the data collected.

I think this would be fantastic.

Ok. So let's set about working on this. Anyone else want to join? We should probably keep this off list, as it will be rather technical, and present a solution when we have it.

I already have something of a start for this in the form of my
database of nodes in HECnet. I'd need to extend it with more fields,
but that would be pretty easy. It's all in Datatrieve today, and that
should be accessible over DECnet right now (even though I seem to
remember that VMS hosts had some problems with that).

I'll have to learn how to access that db.

It should be trivial. Datatrieve have a programming interface. Should be callable from any language.

I'm already extracting information from that database for the hecnet
web-page on MIM (accessible as Madame).

So, if we can just decide on what we want, and how to make the
information available, I'll sit down and write the code to fix it.

Do you want to also store my data or should I do that myself? I might do
it myself at least for now until I know what exactly I need/want to save.

Let's start by talking exactly what we want to store, and how to retrieve it, and possible uses of it, to make it meaningful. We can then go about how and where to store it. I even think that it wouldn't be a problem to store it in several places, scrape the source from several places, and present data and services based on this from several places.

	Johnny