How to get HG data

[Yomi]

This thread is for people making tools related to HG. It describes the APIs available to get data for things like messages, players online, market data, and so on.

This has been floating around in PMs and chat logs, but it's long past time to post it.

A note about the URLs. info.hgweb.org/ is currently equivalent to hgweb.acaos.webfactional.com/info/. In theory the first should be used since it is more generic, but it's pointed to the second for a long time now, so use whichever you'd like.


Data is retrieved from the web server, which gets it from the game server. One nice consequence of this is that load on the web server has no effect on the in-game performance. The current setup has a time delay, where the web server has snapshots of the game behavior. The current settings are:

Messages are updated every 20 seconds.
Uptimes, player data, and quests are updated every 60 seconds.
Market data is updated every 120 seconds.

Note that for messages, if no messages on any channel have been posted, it doesn't update anything on the web server. Somewhat related, the guild static pages are only generated when activity happens.

[Yomi]

Static Information

There is quite a bit of information generated statically every time the web data is updated. If this works for your program, it's preferred since retrieving it is very lightweight for the web server.

HTML:
* messagelog.html Global channel (3 hours)
* bazaarlog.html Market channel (9 hours)
* chatlog.html Chat channel (9 hours)
* newbielog.html Newbie channel (9 hours)
* marketlist.html All the market items in table format with rollups
* playerlist.html Player list
* quests.html Quests

Guilds :
* guild/<guild name>/messagelog.html Guild messages (24 hours)
* guild/<guild name>/messagelog.xml Guild RSS feed

Note that the guild message logs are password protected, are only set up for guilds that request them, and only have inter-server messages. That is, if you say "/g I really wish PersonX wasn't in the guild." it does not go into the message log. If you use "!guild" it goes out to everyone. Some guilds use this to announce runs, knowing that even people not logged into HG can see the message.


messagelog.xml is the global message RSS feed.


xdata.xml. This is of particular importance for tool writers. This XML file contains all meta, user, and messages for all public channels for the last hour. Tools that do more-or-less real time updating are encouraged to use this. It's basically everything going on on the servers in the last hour, except market data.

[Yomi]

Dynamic Information

A cgi script lets on perform dynamic queries on the web server. It has quite a few arguments. The main script is:

info.hgweb.org/webout.cgi

The main arguments you need to give it are:

type=XXX where XXX={messages,market,players,quests,meta,xdata,all}

and

format=XXX where XXX={html,xml,json,rss,atom}


Formats supported by the types:
* messages: HTML, RSS, Atom, XML, and JSON.
* market: HTML and XML
* players: HTML and XML
* quests: HTML
* Meta: XML
* xdata: XML
* all: XML

All means:
meta, players, messages for channels [0,-2,-3,-4], and market.

xdata means:
meta, players, messages for channels [0,-2,-3,-4].


Other arguments:

Option	Arguments	Type	Format
channel=#	0 = global, -2 = bazaar, -3 = newbie, -4 = chat	Messages	All
channels=#	comma separated list, or 'public	Messages	JSON (should be All)
maxmes=#	limit number of messages to #	Messages	All
maxage=#	limit age of messages to # hours	Messages	All
fadesec=#	used for HTML shading	Messages	HTML
minfade=#	used for HTML shading	Messages	HTML
norefresh	Don't add a refresh tag for HTML	Messages	HTML
refresh=XXX	HTML reloads every # seconds	Messages	HTML
highlight	Adds HTML bold/color	Messages Market	HTML
l33t	HG pwn3d by hax0rs	Messages	HTML
showguild	Shows guild in HTML	Players	HTML
localtime	Tries to convert UTC times to browser time	All	HTML
newsince=XXX	Experimental option for limited retrieval	All	XML
callback=XXX	Returned JSON will call this function	All	JSON

(Note that xdata and all will use message settings for their message portions)

This isn't as clean as I'd like, but it's all a work in progress.


Some examples:

info.hgweb.org/webout.cgi?type=xdata&format=xml&maxage=12&maxmes=100

XML of all data except market, 12 hours worth of data, limited to 100 messages per channel. This is a pretty good way to initialize an app that wants to display HG data, which will then make calls to get the static xdata.xml afterwords for updates.

Simple metadata (lightweight): info.hgweb.org/webout.cgi?type=meta&format=xml

Messages: info.hgweb.org/webout.cgi?type=messages&format=xml

Players: info.hgweb.org/webout.cgi?type=players&format=xml

Market: info.hgweb.org/webout.cgi?type=market&format=xml

Everything: info.hgweb.org/webout.cgi?type=all&format=xml

Almost everything (all except market data which is quite large): info.hgweb.org/webout.cgi?type=xdata&format=xml Note that this is very similar to the static file though it has an age limit of 1 hour, rather than the default 3 hours.

[Yomi]

Details of the output.

I wouldn't rely on parsing the HTML or RSS/Atom output, as it may change without notice. This post is describing the XML and JSON output, which are meant for programatic parsing.

....to be written....

[Yomi]

Examples of parsing.

Perl:

... will add various examples ...


AHK (AutoHotKey):

See the file src/arf2.ahk from bhao's [url=http://highergroundpoa.proboards.com/index.cgi?board=software&action=display&thread=18526[/url]ARF2[/url]. It uses the dynamic XML calls.

AJAX:

would sure be nice if someone wants to write an example. I ran out of time to play with it.

[dirk]

Hey Yomi, is it possible something blew up in the XML tonight? I just noticed my HG Webdash broke some time in the last few hours, and I also can't call up the XML or webcgi directly in my browser.

For example this throws an error for invalid XML char:
info.hgweb.org/webout.cgi?type=messages&format=xml

I wonder if it's a funky character in someones message?

EDIT:
Yes I think it's in the Feywild shout, the error actually points it out clearly:

<fullmsg>Run forming on server 214. Contact Gyro's Wuckin Jammy Fizard (GyroKut) if interested: Feywild, smack down and

I'm not sure if I can sanitize that before parsing or not. Will have to look into it later on. :(

-db

[Yomi]

It was a control-Z that made it into a message. My XML control character remapping was missing some cases. Control characters (x00-x1F) for both HTML and XML should now be remapped to ४ (code 2410).

[dirk]

Awesome, thanks Yomi. Nice post here by the way, I meant to shout kudo's to you on this awesome interface to the HG data.

-db

[arek]

I don't know if anyone's still working on this, but if so could HG Quest data be made available in XML format? Currently it's only available as a rather parser-unfriendly HTML page.

--Arek