How’s that for a run-on sentence?

Alright, so in addition to being too lazy to click on every link, I’m also too lazy to write fancy code for this project. What I wanted was something quick and dirty. Something that got me 80% of the way there.

feedextractor.py is where I ended up.

This little python script uses the wonderful BeautifulSoup xml/html parsing library from Crummy Software. I highly recommend the soup and Lewis Carroll’s Alice in Wonderland.

Using feedextractor.py

1. Export you current list of feeds in opml format.
2. Rename the export file to “mysubscriptions.opml”
3. Place the export file in the same directory as feedextractor.py
4. Change the baseurl variable in feedextractor.py to the url of the page you would like start from
5. Run feedextractor.py (i.e. [python feedextractor.py])

This will create a file called newfeeds.opml with all of the spidered feeds that do not appear to be in your current list of feeds.

Known “problems”

1. The script only takes the first feed from a found site. If there is an RSS and an ATOM feed, the script will grab whichever one is at the top of the file. This means that multiple feeds are ignored. You might think this is a bad thing. If so, feel free to change it. I set it up this way because I didn’t want to comb through the found feeds and delete what amounts to duplicates.

2. The script has just one exception block. If something happens while trying to pull back a page, the script skips that site. Could be more elegant.

3. The script does not bother with parameters. Would be nice if you could just pass in a url… I know this is simple, but again I am in a hurry. I just wanted it to work. I’m not making a project here.

4. urllib2 gets rejected by some sites. True enough. Some web servers will reject a request from urllib2. If you want to go to the trouble of adding a user agent header, be my guest.

It works

This script comes as-is. Use it to your heart’s content. I’m not planning updates or anything else. Just a fun bit o’ code I whipped up to suit a need.

But it does work, and quite efficiently too (even for some sloppy-quick hacking).

Download feedextractor.py

Raw source after the jump…

#!/usr/bin/env python
# encoding: utf-8
"""
feedextractor.py

Created by Jamie Grove on 2008-01-30.

"""

from BeautifulSoup import BeautifulSoup
import urllib2
from xml.dom import minidom
from urlparse import urlparse
import socket

timeout = 15
socket.setdefaulttimeout(timeout)

subscribedsites = []
subscribedfeeds = []

# put your seed url here
baseurl = 'http://www.please-change-to-some-site.com/or-full/url.html'

# loadsubscriptions - imports your current opml list
def loadsubscriptions():
	global subscribedsites,subscribedfeeds
	dom = minidom.parse('mysubscriptions.opml')
	for node in dom.getElementsByTagName('outline'):
		subscribedsites.append(node.attributes['htmlUrl'].value)
		subscribedfeeds.append(node.attributes['xmlUrl'].value)

# gethtml(url) - fairly obvious, right?
def gethtml(url):
	html = urllib2.urlopen(url).read()
	return html

# extractlinks(html)  - pulls out all the anchor tags from the html, skips sites you already have
#   uses beautifulsoup to extract links
#   1) checks to see if the netloc of the anchor is in the list of subscribed sites
#   2) checks to see if the netloc of the anchor is in the list of links (keeps out the dupes)
#   3) checks to see if the netloc of the anchor is in the seed url
def extractlinks(html):
	global subscribedsites,subscribedfeeds,baseurl
	soup = BeautifulSoup(html)
	anchors = soup.findAll('a')
	links = []
	for a in anchors:
		o = urlparse(a['href'])
		if len([s for s in subscribedsites if o.netloc in s]) == 0 and len([s for s in links if o.netloc in s]) == 0 and o.netloc not in baseurl:
			links.append(a['href'])
	return links

# getfeed(html) - looks for feed URLs in the html you pass in
#   uses beautifulsoup to extract links
#   same basic logic as extract links to make sure you only get feeds you don't have
def getfeed(html):
	global subscribedsites,subscribedfeeds,baseurl
	soup = BeautifulSoup(html)
	linkedfiles = soup.findAll('link')
	feed = []
	for l in linkedfiles:
		if l.has_key('rel'):
			if l['rel'] == 'alternate':
				o = urlparse(l['href'])
				if len([s for s in subscribedfeeds if o.netloc in s]) == 0 and len([s for s in feed if o.netloc in s['href']]) == 0:
					feed.append({'href':l['href'],'title':l['title']})
	return feed

# main - unimaginative?  yes, but it works
#   1) creates a opml stub
#   2) grabs the seed page and parses it for new links
#   3) goes out and gets feeds (if they exist)
#   4) adds feeds to the stub opml
#   5) writes the opml file out for import elsewhere
def main():
	global subscribedsites,subscribedfeeds,baseurl
	loadsubscriptions()
	html = gethtml(baseurl)
	links = extractlinks(html)
	xml = minidom.Document()
	opml = xml.createElement('opml')
	opml.appendChild(xml.createElement('head'))
	body = xml.createElement('body')
	print '%d links' % len(links)
	counter = 0
	for l in links:
		counter = counter + 1
		print 'processing link %d - %s' % (counter,l.encode('latin-1'))
		try:
			html = gethtml(l)
			feed = getfeed(html)
			if len(feed) > 0:
				for f in feed:
					outline = xml.createElement('outline')
					outline.setAttribute('title',f['title'])
					outline.setAttribute('htmlUrl',l)
					outline.setAttribute('xmlUrl',f['href'])
					body.appendChild(outline)
		except:
			print 'Could not get %s'% l.encode('latin-1')
	opml.appendChild(body)
	xml.appendChild(opml)
	fp = open("newfeeds.opml","w")
	# writexml(self, writer, indent='', addindent='', newl='', encoding=None)
	xml.writexml(fp, "    ", "", "\n", "UTF-8")

if __name__ == '__main__':
	main()

P.S. Looking for a good book on Python Network Programming? I highly recommend John Goerzen’s Foundations of Python Network Programming. John’s style is engaging and easy to read. His examples are practical and clear (way better than the shoddy code I wrote above).

This book will have you spinning ideas and code so fast you’ll wonder how you got along without it.

Python just happens to have a basic HTTP server as part of the base language (see SimpleHTTPServer). There is also an extension of the module that runs CGI (see CGIHTTPServer). So, a quick script and you have an instant web server that:

• runs on port 8000 (by default)
• uses the current directory as the root htdocs
• can run CGI scripts

[Note: I'm not the first to think of using this built-in python function. I mean, that's what it's for, right? I'm just passing this along as a tip.]

If you place said script on your path (as I have), you can fire up a mini web server in an instant from anywhere.

Here is the script:

#!/usr/bin/env python
# If necessary, replace above with your path to python

import CGIHTTPServer

CGIHTTPServer.test()

I called my script cgiserver for ease of use. Also, if you are new to shell scripting, be sure to make your shiny, new script executable:

chmod +x [script_name]

Below is a clip from my terminal window where I fired up the server and browsed http://localhost:8000. To close the server, just hit control-c to kill the script.

jgmbpro:~/Documents/Work/FGTP/site jamiegrove$ cgiserver
Serving HTTP on 0.0.0.0 port 8000 ...
localhost - - [15/Sep/2007 09:21:02] "GET / HTTP/1.1" 200 -
localhost - - [15/Sep/2007 09:21:02] "GET /styles/stylemain.css HTTP/1.1" 200 -
localhost - - [15/Sep/2007 09:21:02] "GET /styles/scal.css HTTP/1.1" 200 -
localhost - - [15/Sep/2007 09:21:02] "GET /javascripts/prototype.js HTTP/1.1" 200 -
localhost - - [15/Sep/2007 09:21:02] "GET /javascripts/scriptaculous.js HTTP/1.1" 200 -
localhost - - [15/Sep/2007 09:21:02] "GET /javascripts/builder.js HTTP/1.1" 200 -
localhost - - [15/Sep/2007 09:21:02] "GET /javascripts/effects.js HTTP/1.1" 200 -
localhost - - [15/Sep/2007 09:21:02] "GET /javascripts/dragdrop.js HTTP/1.1" 200 -
localhost - - [15/Sep/2007 09:21:02] "GET /javascripts/controls.js HTTP/1.1" 200 -
localhost - - [15/Sep/2007 09:21:02] "GET /javascripts/slider.js HTTP/1.1" 200 -
localhost - - [15/Sep/2007 09:21:02] "GET /javascripts/sound.js HTTP/1.1" 200 -
localhost - - [15/Sep/2007 09:21:03] "GET /javascripts/scal.js HTTP/1.1" 200 -

When using urllib, you can set the debuglevel directly by using something like this:

import urllib, httplib

httplib.HTTPConnection.debuglevel = 1
urllib.urlopen(”http://www.somesite.com”)

However, when using urllib2 you need to create a handler install it for use. The sample below creates the lovely debuglevel handler.

import urllib2
h=urllib2.HTTPHandler(debuglevel=1)
opener = urllib2.build_opener(h)

Here’s the original post on mail.python.org

P.S. Looking for a good book on Python Network Programming? I highly recommend John Goerzen’s Foundations of Python Network Programming. John’s style is engaging and easy to read. His examples are practical and clear. This book will have you spinning ideas and code so fast you’ll wonder how you got along without it.

SQLAlchemy is a database “toolkit” for python. In many ways, it is like Hibernate from the Java world. Both kits are focused on providing high-performance object relational mapping, but they also provide some nice database abstraction functions as well. The SQLAlchemy document is bit dense, but then this is serious stuff. Still, after I played with it for awhile, I would recommend it over SQLObject.

The SQLAlchemy Philosophy:

SQL databases behave less and less like object collections the more size and performance start to matter; object collections behave less and less like tables and rows the more abstraction starts to matter. SQLAlchemy aims to accommodate both of these principles.

SQLAlchemy doesn’t view databases as just collections of tables; it sees them as relational algebra engines. Its object relational mapper enables classes to be mapped against the database in more than one way. SQL constructs don’t just select from just tables—you can also select from joins, subqueries, and unions. Thus database relationships and domain object models can be cleanly decoupled from the beginning, allowing both sides to develop to their full potential.

Like I said, it’s a bit dense, but check out Robin Munn’s article on SQLAlchemy. What appears below is a much summarized version of Munn’s sample code:

from sqlalchemy import *
from sqlalchemy.orm import *

#accessing a database
db = create_engine('mysql://root@localhost/test')

#metadata object used for binding
metadata = BoundMetaData(db)

# creating a table
users_table = Table('users', metadata,
	Column('user_id', Integer, primary_key=True),
	Column('user_name', String(40))
)

#metadata.engine.echo = True
try:
	users_table.create()
except exceptions.SQLError:
	print 'TABLE \'users\' already exists.'

# loading definitions automatically
users_table = Table('users', metadata, autoload=True)

# printing a column
print list(users_table.columns)[0].name

#create a holding class
class User(object):
	def __repr__(self):
		return '%s(%r,%r)' % (
			self.__class__.__name__,self.user_name,self.user_id)

	def wager(self):
		return 'betting on it'

# map the holding class to the table definition
mapper(User, users_table)

#create an instance of the class
u1 = User()

#see, it automatically maps class and fields.  Slick.
print u1.user_id
print u1.wager()

The DB-API PEP is good if you already know what you’re doing (you can also look at the sqlite module doc). Below is my very simple example. Paul DuBois has an extensive article on using MySQL with python.

from MySQLdb import *

# connect to the database
conn = connect(
	host='localhost',
	user='test',
	db='test'
)

table_def = """
CREATE TABLE samples (
	sample_id int PRIMARY KEY,
	sample_name varchar(20),
	sample_url varchar(250)
)
"""

try:
	cursor = conn.cursor()
	cursor.execute(table_def)
	conn.commit()
except:
	print 'Table already exists.' #obviously, there could be other errors too.

conn.close()

OPML as defined by Dave Winer way back in 2000:

OPML an XML-based format that allows exchange of outline-structured information between applications running on different operating systems and environments.

More specifically, OPML is the format used by most RSS aggregators to spit out lists of feed subscriptions. That list usually contains the name of the feed, the feed’s URL, and the site’s URL.

On one of my little sites, I wanted to keep a quick list of links showing the feeds I subscribed to on that particular subject. As I add feeds to my aggregator at a pretty good clip, updating the site would be a tedious process. So, rather than use the tools available in my aggregator I went ahead and wrote a little python script to grab the OPML and convert it into an HTML stub.

# opml2html.py - sample code for converting OPML to HTML
from xml.dom.minidom import parse, parseString
import urllib2

#dom1 = parse('mah_links.opml') # parse an XML file by name - uncomment if you want to draw from a file
dom1 = parseString(urllib2.urlopen('http://share.opml.org/opml/top100.opml').read()) #use this to parse a feed

links = dom1.getElementsByTagName('outline')

f = open('links.html','w')

for link in links:
	linktext = '<a href="' + link.getAttribute('htmlUrl') + '">'
	linktext += link.getAttribute('title') + '</a><br />\n'
	print linktext
	f.write(linktext)
	f.flush()

f.close()

Pretty simple, eh? I included the method to parse a live feed too just for reference. By the way, if you’re looking for lists of feeds, Winer’s share.opml.org is great.