a nifty way to automatically grab title or description tags from the destination page of a url

29 comments 7 links

So I recently discovered a nifty feature on Facebook where you can enter a link into your status bar and it automatically populates the title and description for the web page that can be found at that link. Smart.

Concurrent to but not as a result of this, at some point, it occurred to me that it might be useful to attach links, raw links, to the articles here on do{block}.

I remembered too, reading over and over again about an HTML parser called Hpricot, developed by Ruby's resident rockstar _why the lucky stiff, which, while I've never used it before, seemed like it would get the job done for me.

And it did. Allow me to elaborate:

hpricot

Either a hip apricot, or a hyper-icot, or for your purists, just a hpricot and I should go jump off a bridge for suggesting otherwise. Whatever the case, it's a pretty useful HTML parser which, astonishingly enough, you can install like this:

$ sudo gem install --remote hpricot

Gee, wacky. This, along with Ruby's built in "open-uri" library, will give us ample tools to grab the html from a remote location, parse out that html into a Hpricot object/array, and extract specific bits of data from whichever tags our hearts desire.

Lovely.

But before we take another bite out of said hpricot, or elaborate upon why you can't find any literature about it at all except for its rdoc and this simple documentation page, let's talk a little bit about _why, the mysterious rubyist who ran away.

why, _why?

_why the lucky stiff, for those of your unfamiliar with him, has been a bit of an icon in the Ruby community for the past several years.

He published and illustrated _why's poignant guide to ruby, an insightful demi-fictional-Ruby-how-to-novella that was touted on the Rails homepage and was how I and many others were first drawn into the language and the Ruby community.

He's responsible for the try ruby! interactive Ruby demo on the Ruby homepage, and myriad open source applications such as camping, RedCloth, Shoes (which I've used extensively and love), and many others that you've heard of and have long been staples in the Ruby community.

Anyway, he's gone. Apparently at some point he kind of closed up shop and left town without a word, having shut down ALL of his web properties in the meantime, and leaving many (including us) in the dark without any of his well-illustrated, post-modern instruction.

Just telling you in case you go looking for any trace of him. All that's left are some mirrors that dutiful ruby-ites have put up of his past work. But that's it, all else has vanished. Weird.

getting on with our lives and this tutorial

Okay, so _why has taken something of an abrupt sabbatical, but he's left hpricot in his wake and that's good enough for me for the time being. The weight of the world gets to all of us at some point.

Now -- suppose that you've got a link model that you want to attach to articles, or even that you want to devise some nifty system in which links attached to or within articles automatically input the title of the destination page as the title for that link tag. Both of these would be very useful, but for either you'd probably want to integrate the action into the Link Model itself. It might look something like this:

#!/app/models/link.rb

class Link < ActiveRecord::Base

  attr_reader :html
  def html(sync=false)
    unless sync || @html
      @html = Link::HTML.new(url)
    else
      @html
    end
  end
  
  class Link::HTML
    # require html-parsing library
    require 'rubygems'
    require 'hpricot'
    require 'open-uri'
    
    attr_reader :title, :desc, :doc
    def initialize(link)
      begin
        @result = open(link)
        @doc = Hpricot(@result)
        @result.close
      rescue
        @doc = nil
      end        
    end

    def title
      unless @doc.nil?
        @title = @doc.at("title").inner_html
        return @title unless @title.nil?
      end
      return ""
    end

    def desc
      unless @doc.nil?
        @desc = @doc.search("//meta[@name='description']").first
        return @desc["content"] unless @desc.nil?
      end
      return ""
    end
  end


end

Let's look at the Link::HTML class first. The first thing that we do is to ensure that rubygems, hpricot, and the open-uri library are all or have already been required for our use. With that in place we can set up the Link::HTML#initialize method, which will grab and parse all of the html for the destination page at the link provided:

def initialize(link)
    begin
      @result = open(link)
      @doc = Hpricot(@result)
      @result.close
    rescue
      @doc = nil
    end        
  end

The open() method is actually provided by the open-uri library, which returns the destination html file for link. We then pass that file through the Hpricot() constant/method, which will parse the contents of that html file into a nifty Hpricot object/array that contains all of the page's html tags in order. This Hpricot object/array is then stored in the @doc instance variable so we can play with it later.

The method will also rescue this process from any errors that might occur in the event that (link) is not a valid url or if open() fails in grabbing the html from that destination.

But before we go into actually fetching the title and description for the page we just fetched, let's first look at the html method in the Link model that we created to access Link::HTML. The html method, or Link#html, is used to ensure that a new Link::HTML object is created only when one doesn't exist, or when it's called with the html(true) parameter. This saves us from fetching the destination html anew everytime you want to perform actions on it.

Also, and this is really, REALLY IMPORTANT, this function assumes that you've got a field called url in your Link model, so that you can fetch the destination html by calling:

@html = Link::HTML.new(url)

If you don't have a url field in your Link model, or if you don't have a Link model at all, just replace url with whatever variable contains the url for the destination page you're trying to retrieve.

Whew, I'm glad we cleared that up. With this in place you can now call @link.html from a Link object and return a formatted array of the html from the destination page. Nifty.

describing the indescribable

We've now got this massive array of html bits in the form of a Hpricot object stored in @doc which, while perhaps not immediately useful to us, will pay dividends very soon. Specifically, we'll be able to use methods within the @doc/Hpricot object to return useful bits like the title and the meta description from the page. Here are those methods from the above code:

def title
  unless @doc.nil?
    @title = @doc.at("title").inner_html
    return @title unless @title.nil?
  end
  return ""
end

def desc
  unless @doc.nil?
    @desc = @doc.search("//meta[@name='description']").first
    return @desc["content"] unless @desc.nil?
  end
  return ""
end

To get the title we use the built-in Hpricot#at method, which returns just the first instance of the tag that we pass in as a parameter, in this case "title". We use #at here because we know the page should have only one title tag, and we don't want it to work any harder than it needs to. Then, once we've got that title tag, we use the inner_html method to just grab the text within the tag, and not the tags themselves. Simple.

Getting the description, or desc as I've named the method above, is a little more convoluted. As opposed to there being one single