Oliver Roup's Blog » Code

Keyboard modified clicks

oroup — Mon, 29 Mar 2010 23:36:03 +0000

In case you’re ever wondering, here is a full list of the “key modified clicks” that the major browsers look for:

(Note that on Windows, “cmd” is “ctrl”)

Chrome, Firefox:
shift: new active window
cmd: new tab
cmd-shift: new active tab

Safari:
cmd: new tab
cmd-shift: new active tab
cmd-opt: new window
cmd-opt-shift: new active window

IE:
shift: new active window
ctrl-shift: new active tab

Note that you also need to deal with right clicks and “open in new window”, “open in new tab”, etc….

Wow Google Spreadsheets

oroup — Thu, 03 Jul 2008 16:54:55 +0000

I’ve long been a fan of Google Spreadsheet. Like all disruptive technologies, it doesn’t meet the needs of most customers of the incumbent product, Excel. I’m no banker, but even my relatively neophyte hands keep trying to hit F4 to lock a cell reference or hit F2 to edit a cell. I miss being able to right click to format a cell, the lag time of javascript sometimes annoys the crap out of me, etc… I would not try to convince a Wall Street number jockey to switch from Excel just yet. No way.

BUT, also like all disruptive technologies Google Spreadsheet is quietly getting good at things that Excel can only dream of. Since it launched, Google Spreadsheet has had great sharing and concurrent editing capabilities. A team at school used a shared Google Spreadsheet as a “factory information system” in a simulated factory exercise and it was amazingly powerful.

More recently, Google has added amazing data extraction techniques. Put =GoogleFinance(“Oil”, “Price”) in a cell and you’ll get the price of oil updated in realtime. The following will pull all the headlines off Techmeme: =importxml(“http://www.techmeme.com”, “/html/body/div[2]/div/div[4]/div/div[2]/div/div/div/strong/a”)

There are also functions for trolling arbitrary HTML, for parsing RSS and Atom feeds and for importing CSS feeds.

Where the mind really starts to boggle is when you think about how every cell in every google spreadsheet effectively has an URL. So it should be almost as easy to publish from your sheet as it is to pull in from somebody else’s. Suddenly, there are network effects for spreadsheets, where your work can leverage off the work of others. This ability to ref a cell in someone else’s sheet doesn’t seem to be enabled just yet, but I can’t imagine Google isn’t working on it. I’m sure there are some issues around permissioning and detection of dependency loops.

Throw in a little work on keyboard shortcuts and a little Google Gears magic to make the app more responsive and workable offline and Excel will be starting to feel the heat…

PicasaWeb Downloader

oroup — Tue, 20 May 2008 10:55:29 +0000

In the genuinely useful software dept., here is a little app written in C# (source included!) that will download an entire google web album. Great for grabbing copies of all of your friends photos.

Theme weirdness

oroup — Mon, 14 Jan 2008 06:32:58 +0000

I upgraded WordPress to the latest version and my theme seemed to be interacting poorly with the new Schema. So I’ve reverted to the default theme and I’ll find a new one I like at some point soon.

Computing distances with Lat/Lon

oroup — Sun, 14 Jan 2007 19:45:01 +0000

Have you ever wanted to compute distances with latitude and longitude? At first it seems so simple. We all learned the pythagorean theorem in elementary school:
d=sqrt((x2 – x1)^2 + (y2 – y1)^2)
So if we have our two points (x1,y1) and (x2,y2) then we should be able to find (x2-x1,y2-y1) and find the distance using the Pythagorean theorem, right? Well, no.

The Pythagorean theorem finds the straight line distance between two points. However, for two points any appreciable distance from each other on earth, trying to get to your destination would involve a lot of digging. Assuming you want to travel overland, you must travel further than the linear distance to account for the curvature of the Earth.

A little further research yields the Haversine Formula which is useful for computing surface distances on a sphere and correctly accounts for points that are very close together:

Haversine Formula (from R.W. Sinnott, “Virtues of the Haversine”, Sky and Telescope, vol. 68, no. 2, 1984, p. 159):

dlon = lon2 – lon1
dlat = lat2 – lat1
a = sin²(dlat/2) + cos(lat1) * cos(lat2) * sin²(dlon/2)
c = 2 * arcsin(min(1,sqrt(a)))
d = R * c

But what is R, the radius of the Earth?

It turns out the Earth is not a sphere after all. The spinning motion compresses the planet so that the equitorial radius is considerably larger than the polar radius. The shape of the Earth is well approximated by an oblate spheroid with a polar radius of 6357 km and an equatorial radius of 6378 km.

R’ = a * (1 – e²) / (1 – e² * sin²(lat))^(3/2)

where a is the equatorial radius, b is the polar radius, and
e is the eccentricity of the ellipsoid = (1 – b²/a²)^(1/2).

Wow! So there you have it.

Of course if you’re trying to compute these values in client side javascript, it may take a while. It turns out that for short-ish distances at latitutudes within the continental United States, just pretending the Earth is a sphere yields errors of roughly 20 meters. That’s good enough for me..

(All from Google and http://www.faqs.org/faqs/geography/infosystems-faq/ )

KML in Google Maps and the challenge of standards

oroup — Wed, 27 Dec 2006 16:53:05 +0000

The Google Maps API is generally a pleasure to work with. After more or less inventing the whole Web 2.0 thing, the people behind Google Maps have continued to innovate, recently adding a Geocoder to their API, allowing address information to be translated into coordinates in Latitude and Longitude. Naturally having this information expands considerably the number of things you can do with Google Maps. Even better, it doesn’t just return Coordinates for an address, it “normalizes” a query and returns the address in a structured way.
There’s no doubt that creating a global XML standard for addresses must have been a challenging thing to try and do. Some countries use street names in addresses others do not. Some nations use island names, others have no islands. Some (like America) has county names but do not use them in addresses. So Kudos to OASIS for creating a standard called xAL and kudos to Google for trying to use it in their Google Maps API. Unfortunately, the devil is always in the details.

First, OASIS makes reading the standard harder than it should be by making you download a 10 megabyte zip file containing a whole bunch of standards, not all of which are currently relevant to me. Law #1 in the web age: Google is the source of all* traffic. The only reason I even found them is because Google helpfully links them.

While it’s clearly better than nothing, the following quote from the spec tells much of the story: xAL can be used to define addresses in simple terms or in complex terms. It is up to the user to decide how they want to implement xAL.

I’d be very surprised if anyone anywhere has implemented software that can read and do something useful with the full range of XML that technically conforms to the spec. There’s just too much of it.
Unfortunately (and as with all things this complex) there appear to be outright bugs in the Google implementation. Take two address examples:

Example #1: 91 N. MAIN ST., WARSAW NY 14569 This address works beautifully and returns results as you might expect. There’s a hierarchy and each element in the hierarchy maps to a US centric address element:

Country (Country)

Administrative Area (State)

Sub Administrative Area (County)

Locality (City)

ThoroughFare (Street)
PostalCode (Zip)

Beautiful. Google has mapped US addresses nicely into the standard and it’s easy to write code that does useful things. Unfortunately, in my small sample, Google only returns results of this form about 87% of the time.

Example #2: 760 HOOSICK RD., TROY NY 12180 This address also works well but returns a different hierarchy:

Country (Country)

Administrative Area (State)

Sub Administrative Area (City)

ThoroughFare (Street)
PostalCode (Zip)

Notice that County information is missing. This is not a fatal problem and is to be expected with a dataset as large and noisy as addresses. However, rather than simply leaving Sub Administrative Area unnamed (as the spec seems to suggest with its examples) the API remaps the City up to the Sub Administrative Area and omits the Locality element. Notice also that ThoroughFare and PostalCode are no longer a child of Locality but of Sub-Administrative Area.

I’ve sent a link to this post to the folks at Google Maps. I’ll be interested to see whether they agree that it’s a bug or think they’re adhereing to the letter of the spec. That’s the challenge with standards. They either say to much or to little.

Note: The examples above link to cached versions of what Google returns when you connect with a valid Maps Key. I cannot easily figure out what a key for your host might be so you’ll need to get your own key and make your own queries to see the results for yourself.

The joys of screenscraping

oroup — Sun, 05 Nov 2006 15:58:10 +0000

A while back, I ran across a great HackDiary entry extolling the virtues of using TagSoup and XPATH to do screenscraping from the web. TagSoup is a library that coerces all the ugly nasty HTML you find out in the wild into well-formed (although not necessarily valid) XML. While there’s no guarantee that the results are semantically the same as the input, it lets you use all your nice XML tools like XPATH to extract data. The entry does a great job of showing you how to use TagSoup with Xalan. However, the JDK has been updated with it’s own XPATH parser so it’s no longer necessary to import the Xalan library. Below is a code sample for using TagSoup and the default XPATH parser to retrieve the stock price of Google from Google Finance. Note that the whole “MutableNamespaceContext” implementation is just a workaround for a missing JDK method as documented in JDK Bug 5101859. If that bug gets fixed, the code could be simplified substantially.

The usual disclaimers apply about this all being sample quality code. All error handling has been punted to keep the example length short, but you’d never really want to do that. Also, I’m having trouble preserving indenting in this HTML View, I’ll work on that. This code assumes JDK 1.5. Click through for the code itself.

Does this code support the argument that Java is WAY too verbose? Absolutely.

 /**

 * Demo of screenscraping using TagSoup and XPATH as described at 

 * http://blog.oroup.com/2006/11/05/the-joys-of-screenscraping/

 * 

 * This example class downloads the content of a page from Google

 * Finance and parses it for the Google stock price. It completely

 * omits all error handling for brevity. Also a lot of objects

 * should be cached and re-used if you were really going to call

 * this multiple times.

 * 

 * @author Oliver Roup <oroup@oroup.com>

 */



import java.io.InputStream;

import java.io.StringWriter;

import java.net.URL;

import java.net.URLConnection;

import java.util.ArrayList;

import java.util.HashMap;

import java.util.Iterator;

import java.util.List;

import java.util.Map;



import javax.xml.namespace.NamespaceContext;

import javax.xml.transform.OutputKeys;

import javax.xml.transform.Result;

import javax.xml.transform.Source;

import javax.xml.transform.Transformer;

import javax.xml.transform.TransformerFactory;

import javax.xml.transform.dom.DOMResult;

import javax.xml.transform.dom.DOMSource;

import javax.xml.transform.sax.SAXTransformerFactory;

import javax.xml.transform.sax.TransformerHandler;

import javax.xml.transform.stream.StreamResult;

import javax.xml.xpath.XPath;

import javax.xml.xpath.XPathConstants;

import javax.xml.xpath.XPathFactory;



import org.w3c.dom.Node;

import org.w3c.dom.NodeList;

import org.xml.sax.InputSource;



// The Tagsoup library.

import org.ccil.cowan.tagsoup.Parser;



public class QueryHtml {



  public static void main(String[] args) throws Exception {



    // Get the page and coerce it to an XML DOM. This loads the whole

    // thing into memory so massive pages should be cut down first

    // using SAX or something similar.

    Node node = getHtmlUrlNode("http://finance.google.com/finance?q=GOOG");



    // Create a mutable namespace context. This should really be provided

    // by the JDK, but the default implementation does not allow new entries

    // to be added.

    MutableNamespaceContext nc = new MutableNamespaceContext();



    // Set the prefix "html" to correspond to the xhtml namespace.

    // This can be called multiple times with different prefixes.

    nc.setNamespace("html", "http://www.w3.org/1999/xhtml");



    // This is the query we run against the DOM coereced from the web page.

    // If the HTML changes in a relevant way, it will break this query. 

    final String QUERY = "/html:html/html:body/html:table[2]/html:tr" + 

      "/html:td/html:table/html:tr[2]/html:td[2]/html:div/html:table" + 

      "/html:tr/html:td/html:span/html:span/text()";



    // Run the xpath query against the DOM.

    NodeList result = xPathQuery(node, QUERY, nc);

    

    // Print out the result.

    System.out.println(dumpNode(result.item(0), true));

  }



  /**

   * @param urlString The URL of the page to retrieve

   * @return A Node with a well formed XML doc coerced from the page.

   * @throws Exception if something goes wrong. No error handling at all

   * for brevity.

   */

  public static Node getHtmlUrlNode(String urlString) throws Exception {



    SAXTransformerFactory stf = 

      (SAXTransformerFactory) TransformerFactory.newInstance();

    TransformerHandler th = stf.newTransformerHandler();



    // This dom result will contain the results of the transformation

    DOMResult dr = new DOMResult();

    th.setResult(dr);



    Parser parser = new Parser();

    parser.setContentHandler(th);



    URL url = new URL(urlString);

      URLConnection urlConn = url.openConnection();

      InputStream stream = urlConn.getInputStream();

    

    // This is where the magic happens to convert HTML to XML

    parser.parse(new InputSource(stream));

    return dr.getNode();

  }

  

  /**

   * @param node An XML DOM Tree for query

   * @param query An XPATH query to run against the DOM Tree

   * @param nc The namespaceContext that maps prefixes to XML namespace

   * @return A list of nodes that result from running the query against

   * the node.

   * @throws Exception If anything goes wrong. No error handling for brevity

   */

  public static NodeList xPathQuery(Node node, String query, 

      NamespaceContext nc) throws Exception {

    XPathFactory xpf = XPathFactory.newInstance();

    XPath xpath = xpf.newXPath();

    xpath.setNamespaceContext(nc);

    return (NodeList) xpath.evaluate(query, node, XPathConstants.NODESET);

  }



  /**

   * @param node A node to be dumped to a string

   * @param omitDeclaration A boolean whether to omit the XML declaration

   * @return A string representation of the node.

   * @throws Exception If anything goes wrong. Error handling omitted.

   */

  public static String dumpNode(Node node, boolean omitDeclaration) 

    throws Exception {

    Transformer xformer = 

      TransformerFactory.newInstance().newTransformer();

    if (omitDeclaration) {

      xformer.setOutputProperty(OutputKeys.OMIT_XML_DECLARATION, "yes");

    }

    StringWriter sw = new StringWriter();

    Result result = new StreamResult(sw);

    Source source = new DOMSource(node);

    xformer.transform(source, result);

    return sw.toString();

  }

  

}



/** There is a bug in the JDK which omits the setNamespace declaration

 * from implementations of NamespaceContext. We have to create our

 * own implementation to work around it. Documented here:

 * http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=5101859

 * @author Oliver Roup 

 */

class MutableNamespaceContext implements NamespaceContext {



  private Map map;



  public MutableNamespaceContext() {

    map = new HashMap();

  }



  public void setNamespace(String prefix, String namespaceURI) {

    map.put(prefix, namespaceURI);

  }



  public String getNamespaceURI(String prefix) {

    return map.get(prefix);

  }



  public String getPrefix(String namespaceURI) {

    for (String prefix : map.keySet()) {

      if (map.get(prefix).equals(namespaceURI)) {

        return prefix;

      }

    }

    return null;

  }



  public Iterator getPrefixes(String namespaceURI) {

    List prefixes = new ArrayList();

    for (String prefix : map.keySet()) {

      if (map.get(prefix).equals(namespaceURI)) {

        prefixes.add(prefix);

      }

    }

    return prefixes.iterator();

  }

}

Postfix

oroup — Sat, 04 Nov 2006 17:07:53 +0000

My rimu host hosts a number of domains, including oroup.com, openrelay.com, cadabraco.com and drugmaps.com. I want to be able to send and receive email to those domains, but (at least for now) I don’t really want to set up dovecot locally and actually deal with mail. So the ideal thing is just to forward your mail to another email address. Postfix is a little daunting at first, but the key instructions turn out to be quite simple.

1 /etc/postfix/main.cf:
2     virtual_alias_domains = example.com ...other hosted domains...
3     virtual_alias_maps = hash:/etc/postfix/virtual
4
5 /etc/postfix/virtual:
6     postmaster@example.com postmaster
7     joe@example.com        joe@somewhere
8     jane@example.com       jane@somewhere-else
9     # Uncomment entry below to implement a catch-all address
10     # @example.com         jim@yet-another-site
11     ...virtual aliases for more domains...

Presto, instant email!