How’s that for a run-on sentence?

Alright, so in addition to being too lazy to click on every link, I’m also too lazy to write fancy code for this project. What I wanted was something quick and dirty. Something that got me 80% of the way there.

feedextractor.py is where I ended up.

This little python script uses the wonderful BeautifulSoup xml/html parsing library from Crummy Software. I highly recommend the soup and Lewis Carroll’s Alice in Wonderland.

Using feedextractor.py

1. Export you current list of feeds in opml format.
2. Rename the export file to “mysubscriptions.opml”
3. Place the export file in the same directory as feedextractor.py
4. Change the baseurl variable in feedextractor.py to the url of the page you would like start from
5. Run feedextractor.py (i.e. [python feedextractor.py])

This will create a file called newfeeds.opml with all of the spidered feeds that do not appear to be in your current list of feeds.

Known “problems”

1. The script only takes the first feed from a found site. If there is an RSS and an ATOM feed, the script will grab whichever one is at the top of the file. This means that multiple feeds are ignored. You might think this is a bad thing. If so, feel free to change it. I set it up this way because I didn’t want to comb through the found feeds and delete what amounts to duplicates.

2. The script has just one exception block. If something happens while trying to pull back a page, the script skips that site. Could be more elegant.

3. The script does not bother with parameters. Would be nice if you could just pass in a url… I know this is simple, but again I am in a hurry. I just wanted it to work. I’m not making a project here.

4. urllib2 gets rejected by some sites. True enough. Some web servers will reject a request from urllib2. If you want to go to the trouble of adding a user agent header, be my guest.

It works

This script comes as-is. Use it to your heart’s content. I’m not planning updates or anything else. Just a fun bit o’ code I whipped up to suit a need.

But it does work, and quite efficiently too (even for some sloppy-quick hacking).

Download feedextractor.py

Raw source after the jump…

#!/usr/bin/env python
# encoding: utf-8
"""
feedextractor.py

Created by Jamie Grove on 2008-01-30.

"""

from BeautifulSoup import BeautifulSoup
import urllib2
from xml.dom import minidom
from urlparse import urlparse
import socket

timeout = 15
socket.setdefaulttimeout(timeout)

subscribedsites = []
subscribedfeeds = []

# put your seed url here
baseurl = 'http://www.please-change-to-some-site.com/or-full/url.html'

# loadsubscriptions - imports your current opml list
def loadsubscriptions():
	global subscribedsites,subscribedfeeds
	dom = minidom.parse('mysubscriptions.opml')
	for node in dom.getElementsByTagName('outline'):
		subscribedsites.append(node.attributes['htmlUrl'].value)
		subscribedfeeds.append(node.attributes['xmlUrl'].value)

# gethtml(url) - fairly obvious, right?
def gethtml(url):
	html = urllib2.urlopen(url).read()
	return html

# extractlinks(html)  - pulls out all the anchor tags from the html, skips sites you already have
#   uses beautifulsoup to extract links
#   1) checks to see if the netloc of the anchor is in the list of subscribed sites
#   2) checks to see if the netloc of the anchor is in the list of links (keeps out the dupes)
#   3) checks to see if the netloc of the anchor is in the seed url
def extractlinks(html):
	global subscribedsites,subscribedfeeds,baseurl
	soup = BeautifulSoup(html)
	anchors = soup.findAll('a')
	links = []
	for a in anchors:
		o = urlparse(a['href'])
		if len([s for s in subscribedsites if o.netloc in s]) == 0 and len([s for s in links if o.netloc in s]) == 0 and o.netloc not in baseurl:
			links.append(a['href'])
	return links

# getfeed(html) - looks for feed URLs in the html you pass in
#   uses beautifulsoup to extract links
#   same basic logic as extract links to make sure you only get feeds you don't have
def getfeed(html):
	global subscribedsites,subscribedfeeds,baseurl
	soup = BeautifulSoup(html)
	linkedfiles = soup.findAll('link')
	feed = []
	for l in linkedfiles:
		if l.has_key('rel'):
			if l['rel'] == 'alternate':
				o = urlparse(l['href'])
				if len([s for s in subscribedfeeds if o.netloc in s]) == 0 and len([s for s in feed if o.netloc in s['href']]) == 0:
					feed.append({'href':l['href'],'title':l['title']})
	return feed

# main - unimaginative?  yes, but it works
#   1) creates a opml stub
#   2) grabs the seed page and parses it for new links
#   3) goes out and gets feeds (if they exist)
#   4) adds feeds to the stub opml
#   5) writes the opml file out for import elsewhere
def main():
	global subscribedsites,subscribedfeeds,baseurl
	loadsubscriptions()
	html = gethtml(baseurl)
	links = extractlinks(html)
	xml = minidom.Document()
	opml = xml.createElement('opml')
	opml.appendChild(xml.createElement('head'))
	body = xml.createElement('body')
	print '%d links' % len(links)
	counter = 0
	for l in links:
		counter = counter + 1
		print 'processing link %d - %s' % (counter,l.encode('latin-1'))
		try:
			html = gethtml(l)
			feed = getfeed(html)
			if len(feed) > 0:
				for f in feed:
					outline = xml.createElement('outline')
					outline.setAttribute('title',f['title'])
					outline.setAttribute('htmlUrl',l)
					outline.setAttribute('xmlUrl',f['href'])
					body.appendChild(outline)
		except:
			print 'Could not get %s'% l.encode('latin-1')
	opml.appendChild(body)
	xml.appendChild(opml)
	fp = open("newfeeds.opml","w")
	# writexml(self, writer, indent='', addindent='', newl='', encoding=None)
	xml.writexml(fp, "    ", "", "\n", "UTF-8")

if __name__ == '__main__':
	main()

P.S. Looking for a good book on Python Network Programming? I highly recommend John Goerzen’s Foundations of Python Network Programming. John’s style is engaging and easy to read. His examples are practical and clear (way better than the shoddy code I wrote above).

This book will have you spinning ideas and code so fast you’ll wonder how you got along without it.

In corporations Open source is thriving, and not because its the strategy:

In large corporations if you need to buy a tool for development, especially one no one has used before good luck in completing the order. The developer wanting the tool will have to justify it over and over and the odds of getting it aren‘t dependent on the need but whether the department (and hence what time of the year it is) has any money. Big purchases tend to be much easier because they have to be approved further up in the organisation. Little purchases just drag on and on. I don‘t know why it is the finance mechanism in companies causes so much grief, but the truth is they are not designed for quick purchases for strategic reasons. Any CFO that tells you otherwise is lying or has no idea. The real reason for finance control is to stop money leaking out and the company loosing money and not knowing why. The fact that redeveloping the library will take you 6 months is unfortunately lost on them as your an expected cost.

[Via: JRoller]

Paul goes on to expand this discussion to show how Open Source fits into the financing process which is a key reason for its continuing adoption by corporations.

Of course, this doesn’t explain why some IT shops are afraid of open source, but my gut tells me it’s about responsibility. IT shops unwilling to take responsibility for their own decisions and solutions tend to rely on vendors who are ready to serve as punching bags during a crisis.