Month: April 2005

The Eight (K. Neville)

The female counterpart to ‘The Name of the Rose’? I don’t think so. But at the same time, a well-written book that succeeds in translating a numerological fixation into a compelling tale of adventure. Kudos to the author!

I should mention: chess is the non-numerological fixation. Everything in chess revolves around the number 8, so that the combination of game and number actually buffers the book pretty well.

(more…)

New Mambo Site

Homegrown is nice, but why do your own thing when the rest of the world has moved on?

I was fond of my site tool, but after evaluating a ton of other ones, I settled on Mambo, and surely don't regret it!

The City of Lost Children (1995)

I really wasn’t expecting much from this movie. I had rented it from Netflix because of an affinity hint, but I had gotten some really bad movies that way. This one sounded a little too childish to entrance me, and I was not really too fond of the idea of putting it into the DVD player. Bad reaction!

(more…)

wget – the Universal Web Retrieval Tool

wget – the Universal Web Retrieval Tool

If you spend a lot of time on the web researching information, you have probably wished you could store some of the HTML pages you find locally on your machine. Sometimes the site you are going to is really slow, but you have to consult it frequently; other times you know you won’t be able to get to the web, but need to information on the road; and in the third and worst case, it is good to have your diagnostic information handy if your connectivity ever goes down.

Wget solves this problem. In its easiest incarnation, you can simply download, say, the Yahoo! homepage by typing:

wget http://www.yahoo.com

You’ll get a file called index.html with the current content (as of download time) for the page you were asking.

Invocation

Wget runs from the command line. Its behavior is determined by the command line arguments and by two initialization files, /usr/local/etc/wgetrc for global initialization and ~/.wgetrc for user specific selections. Options specified on the command line override those in the initialization files.

Wget is fairly fast and always tries to do the ‘right thing’, continuing where it was interrupted and retrieving files even when they are not exactly where the URL specified they would be.

Use Case 1: Downloading a single URL

Note: see below for a complete reference.

As mentioned above, to download a single URL you just type wget followed by the full URL. You can skip the protocol part (http://) if the protocol is HTTP. You can use HTTP, HTTPS and FTP as protocols.

Use Case 2: Downloading a whole directory

Not much more difficult. You simply specify the name of the directory, and the options -r -l 1 -np. If the web server returns a directory listing, then these three options turn on recursion, set the recursion depth at 1, and disallow referencing the parent (-np stands for "no parent". Note that the web server must be able to return the directory listing for this to work.

Example: wget -r -l 1 -np http://diva.homelinux.org/images/icons

Use Case 4: Downloading a whole site

This is actually fairly easy. You specify that you want infinite recursion (-r -l inf) and start at the top. There is a handy short version of this, the option -m (for ‘mirror’).

Example: wget -m http://msdb.microsoft.com/

Use Case 5: Make a local copy of a site

You know the problem: your ISP has a wonderful help site where they explain what to do if your connection is down. Unfortunately, if your connection is down, you can’t get to the site. So it’s good to have a local copy of the site for your reference. Wget makes that easy, by providing the option –convert-links.

Example: wget –convert-links http://help.earthlink.net/connections

Use Case 6: Checking bookmarks

If you are diligent like me, after a while you have hundreds of bookmarks or favorites. When it gets to that point, you soon find out that some of the bookmarks point to nowhere, and it’s time to clean them up. Wget can help there. If you export your bookmarks to a file, say bookmarks.html, you can download (that is, check) every URL in that file, which gives you a good idea of what still works and what doesn’t. To do so, you need to combine the -i option (input) with the –spider option. The latter checks whether the URL still is available, without actually downloading anything.

Example: wget –spider -i bookmarks.html

Use Case 7: Incremental backup of a web site

If you have a web site that needs to be backed up locally once in a while, you probably want to do incremental backups and download only the changes. Wget offers two options to do so, -nc (no clobber) and -N (timestamping). The former will simpy refuse to download a page if it is already present on disk. The latter will check the remote and local copies and download the remote only if it is newer than the local copy.

By using -nc, you ensure a minimum effort backup. The option -N, instead, gives you a realistic picture of your web site.

Example: wget -N -r -l inf http://diva.homelinux.org

Use Case 8: Downloading from a authenticated site

If you have a site that requires authentication, you have two options with wget:

  1. Work with web server authentication using basic or digest authentication methods
  2. Use cookies as identifiers

The methods are implemented completely differently, and we’ll discuss the first one here, the other one in the next use case.

(more…)

srdgraph

Prerequisites

Well, the utility is a command line tool that uses Perl – so that’s a good start for requirements. You need some recent version of Perl: I used the tool only with 5.8.0 and appreciate feedback.

The packages required to run the utility are:

  • GD::Graph
  • POSIX
  • S710

GD::Graph

GD::Graph is the graphical core of the utility. It takes care of generating a graph given the data, and you as a user don’t have to do more than presenting it and – voila! – it’s there.

Unfortunately, the ease of use causes a little pain, since you’ll have to download and install it. If you have access to an APT repository, then you should be fine after you type:

apt-get install perl-GD-Graph

Oh, and of course, if you don’t have APT installed – then run and get it!

POSIX

The POSIX package is used for date and time manipulations, so it’s not really a requirement and you could easily modify the code to do without. You can install it the same way as the other package , but POSIX should be installed by default on your Perl distribution.

So far, I have been unable to find an implementation of Perl POSIX on Windows, so if you are trying to get this to run on that OS, good luck!

S710

That’s Dave’s Perl package that wraps the S710 library. To install it, you’ll probably want to go to his site and download both the S710 Linux software and the Perl wrapper (you’ll need both to run the wrapper).

Dave did a splendid job at packaging the wrapper, and now it is extremely easy to use and does pretty much everything you’d want. (I think Dave likes it that we are using his software, so don’t forget to stop by and say thanks!).

Download

You can download the software from the following links. For users of RedHat 9.0, I created two RPMs for the binary pieces:

hrmcat

How Is It?

I called the project hrmcat because I started thinking it would end up being a set of parallel utilities that would complement the existing srd* utilities. Then I found that it made little sense to proceed this way and simply added the capability of reading HRM files to the srd* utilities. Indeed, the change was so minimal that aside from a new C file (for the parsing of the HRM file itself), the modifications to Dave’s code consisted only of a minor change to the file reader. If it fails to parse a file as SRD file, it falls back to HRM. And since the formats are completely incompatible, you risk not one file being mistaken for the other format.

Download

If you want to get HRM files to be read, you need first version 0.12 of Dave’s code. You can find it on his web site, or directly on this one (hope he doesn’t mind) here.

Next, you’ll need to apply a patch. This patch will add the fallback to HRM files as described above. Additionally, it will ensure the hrm.c is compiled and built along with the rest of the files. You can download the patch here.

Finally, you’ll need to download the C file itself and place it in the /src/libs710 directory. To get the file, click here.

Applying the Patch

Nobrainer! You unpack the software:

cd ~/downloads tar xvzf s710-0.12.tar.gz

Now you copy the hrm.c file into its destination location:

cp /tmp/hrm.c ~/downloads/s710-0.12/src/libs710

Finally, you apply the patch:

patch < s710-0.12-hrm.patch

Building

Since Dave is a pro, configuration, compiling and building is really simple:

cd s710-0.12 ./configure make all make install

Running

Now that you have everything compiled and installed, you have three new utilities at hand:

srdcat
the utility used to dump the file information to a human readable format
srdhead
same as srdcat, but doesn’t spit out the raw data, only the initial summary
srdplot
the core utility used to generate the graphs

To run any of them (say srdplot) on an HRM file, you just specify the HRM file as the argument of the call. Say you happen to have a file called /home/marco/.polar/data/2003/01/03012902.hrm. Then to get the graph for the file use the command:

srdplot /home/marco/.polar/data/2003/01/03012902.hrm

Couldn’t be much simpler, huh?

For the Lazy

Well, why do all the work if someone else has already done it? I can get you jump-started with an RPM file for RedHat 9, how about that? Here is the download location.

As usual, feedback always welcome!

HRM File Format

Overview

The Polar S7xx heart rate monitor comes with Windows software to download and display workout data. That’s nice, but for those of us that don’t use Windows, it’s really not going to work.

Dave Bailey, a physicist turned to the dark side of programming like me, has written software to download and display workout data on Linux. Unfortunately, the format he uses to store data is not the same as the format used by the Windows software.

In a different section, I describe my attempts at creating software to add support for the Windows format to the Linux utilities, so as to make transition between the two operating systems easier.

Here, you’ll find all I found out about the Windows file format, called HRM after the extension .hrm

General: Format and Conventions

First the good news: the HRM file format is plain text, and you can look at your workouts in any text editor. The format actually follows the conventions of .INI files, with sections delimited by square brackets (‘[]’), like in the following example:

[Params] Version=106 Monitor=12 [ExtraData] [Summary-123] 0       0       0       0       0       0 180     160     80      70 

 

The bad news next: HRM does not capture all the data recorded in a workout session. It records almost all of it, though.

More news: HRM is a variable format, meaning that it looks slightly different depending on what you were recording on a given workout. To give you an example of what I mean, if you turn off altitude data, you’ll have one less column in the data section. Since you can turn on and off a lot of features, this means that parsing is a bit of a problem.

Sections are always stored in a fixed sequence, starting with Params and ending with HRData.

Sections

Since there is no official documentation, all of the following are guesses. Since some of them are wilder than others, I decided to put those items in italic that seemed more off the mark than others.

[Params]

This section is always at the top of the file and contains the general parameters for the workout.

Version
Currently set to 106, this is the version number of the software.
Monitor
I get 12 here, and assume this is the marker for the heart rate monitor (S710 in my case).
SMode
This is the recording mode. It is a list of bits that mark different functions used in the recording. See table below for the meaning of the bits.
Date
The start date of the workout in 4 digit year, 2 digit month, 2 digit day format.
StartTime
The start time of the workout. Like all times in this format, it’s n digit hours, 2 digit minutes, 2 digit seconds and 1 digit tenth of second, with the separators “:” and “.” – as in: “1:23:45.6” for one hour, twenty-three minutes, fourty-five seconds and six tenths.
Length
The duration of the workout, same format as above.
Interval
???
Upper? and Lower?
The upper and lower heart rate zones.
Timer?
Evidently, this heart rate monitor must have timers. 🙂
ActiveLimit
???
MaxHR, RestHR
The maximum and resting heart rates as entered into the settings panel.
StartDelay
???
VO2Max
Maximum oxigen intake volume. Another option set with the watch settings.
Weight
Your current weight, used for power output computations.

[Notes]

The comments you added in the software to this particular workout.

[IntTimes]

No information available.

[ExtraData]

No information available.

[Summary-123]

This section deals with the heart rate zones and with how much of the workout you spent in any of them.

[Summary-TH], [HRZones], [SwapTimes]

No information on these is available at this time.

[Trip]

Summary information about the trip. This contains the following items in order. Items are not described in any detail, they are just listed in fixed order. Current information about this section is surely wrong. If you have more information, please add it to the forum.

[HRData]

This is the set of core data as coming from the monitor. This is structured as one sample per line, with each line containing information in the following order:

  1. heart rate
  2. speed
  3. cadence
  4. altitude
  5. power

Unrecorded items are just omitted from the list. To know which items were included, you need to understand the SMode field above.

SMode

This bit field indicates what items were recorded and how. Here is a lineup of bits:

speed
bit 7 (highest, first character of string) indicates that speed has been recorded.
cadence
bit 6 indicates that cadence has been recorded.
altitude
bit 5 indicates that altitude has been recorded.
power
bit 4 indicates that power output has been recorded.
bike
bits 1-3 represent the bike number.
units
bit 0 (lowest, last character of string) is set to 1 for imperial, to 0 for metric units.

Polar HRM

What's this?

The Polar S7xx family of bike computers has a handy feature that allows you to download your exercise data to a PC running Windows. Thanks to the work of Dave Bailey, the Polar family now runs on Linux, too. Dave provided software to read Polar data into the PC and to display it in graphs. Later, he added a Perl interface to the library that makes his work accessible from scripts.

srdgraph

I am not really the greatest Perl programmer in the world, but there were a few things I didn't like about Dave's graphs: they are fixed size, which means a short ride gets the same width as a really long one, and that the altitude changes are not reflected in the graphs. This means that a ride with an elevation gain of 100 feet may look on a graph just as steep as one across the Sierras.

So far so good. Turns out, though, that the Perl script to create the graphs that I wanted was really short. Even after I added a few twists (like stitching workouts together for those pesky days when the monitor decides that your 'short' rest warrants a new workout), it ran just above 4k. How's that for efficiency? (more…)

Reading Windows HRM Files

Some of us have started working out on the Polar series before Dave had completed his software. Then we are stuck with two different file formats that are entirely incompatible, since the Windows software saves a stripped down amount of information into files with the extension .hrm after download.

The HRM file structure is fairly well documented in a related project, called Polar Viewer. The author, Stefan Saring, did a wonderful job of creating a C# UI for the viewer using the MONO project framework. If you can use that, please do so – the UI is much easier to use than the text mode utilities here. I found getting Mono to run a little awkward, and I wanted automatic generation of web pages (see project HTML Tree), so I wrote an extension to the srd utilities to get the job done. (more…)

If you want to read up on the file format of HRM files, or if you want to contribute to decoding them, please check here.

Half Dome Trail, Yosemite, CA

{moszoomimglink:Nothing stops you for}Yosemite in the early summer is a wonderland of incredible beauty. The wildflowers are still in bloom, the waterfalls are strong and powerful, and the temperatures are high enough to allow for a quick dip in the water. What better time to explore a great day hike than this?

(more…)

Four Mile Trail, Yosemite, CA

{moszoomimglink:First view of Yosemite – El Capitan and Half Dome}Have you ever gone to Yosemite? Well, I had been living in California for years before I took the time and drove there. It was a chilly autumn morning, the leaves were already brightly turning and I had nothing better to do.

I left early and stayed no time. To a certain degree, I only wanted to say I had been there. Tell myself, that is, since there was nobody else who would have cared.

A year later, it was Thanksgiving, I had just come back from Hawai’i. My housemate surprised me by saying he had rented a room at Yosemite lodge for the holiday, just to escape the cooking. We would drive up on Thursday morning, have dinner at the lodge, and maybe do a hike or two.

{moszoomimglink:El Capitan}Things worked out beautifully. It was unseasonably warm, maybe five degrees above average. We got there in the early afternoon, went on a short hike to Vernal Fall and had dinner at the main restaurant. Mountain air, excellent preparation and the comparison with home cooking made it a real feast. The room was understandably full, but waiting times were close to nil – reservations were not required.

(more…)