Python-Readability-lxml Version 0.3.0.7 ======================================= Forked from Yuri Baburov <[email protected]> This code is under the Apache License 2.0. http://www.apache.org/licenses/LICENSE-2.0 This is a python port of a ruby port of arc90's readability project http://lab.arc90.com/experiments/readability/ In few words, Given a html document, it pulls out the main body text and cleans it up. It also can clean up title based on latest readability.js code. Based on: - Latest readability.js ( https://github.com/MHordecki/readability-redux/blob/master/readability/readability.js ) - Ruby port by starrhorne and iterationlabs - Python port by gfxmonk ( https://github.com/gfxmonk/python-readability , based on BeautifulSoup ) - Decruft effort to move to lxml ( http://www.minvolai.com/blog/decruft-arc90s-readability-in-python/ ) - "BR to P" fix from readability.js which improves quality for smaller texts. - Github users contributions. Installation:: Dev Version: wget https://github.com/appscluster/python-readability/archive/master.zip 7z x master.zip ( if you don't have 7z : sudo apt-get install p7zip-full ) cd python-readability-master/ python setup.py install Base Version: easy_install readability-lxml or pip install readability-lxml Usage:: from readability.readability import Document import urllib html = urllib.urlopen(url).read() readable_article = Document(html).summary() readable_title = Document(html).short_title() Command-line usage:: Default: python -m readability.readability -u http://pypi.python.org/pypi/readability-lxml Add User-Agent: python -m readability.readability -u http://www.bbc.co.uk/newsbeat/30520643 -a "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36" Using positive/negative keywords example:: python -m readability.readability -p intro -n newsindex,homepage-box,news-section -u http://python.org Document() kwarg options: - attributes: - debug: output debug messages - min_text_length: - retry_length: - url: will allow adjusting links to be absolute - positive_keywords: the list of positive search patterns in classes and ids, for example: ["news-item", "block"] - negative_keywords: the list of negative search patterns in classes and ids, for example: ["mysidebar", "related", "ads"] Updates - 0.3.0.7 Encoding updated based on buriy's update - 0.3.0.6 Added option for User-Agent inclusion