How to Fetch Hacker News from news.ycombinator.com?

Some time ago I familiarized myself with the topic of web scraping. As a programming language, the decision quickly falls on Python, since there are good libraries available for this task. When dealing with a new topic, it is not a bad idea to do so creating a self-developed program. The result is: fetch-hackernews.

$ pip3 install fetch-hackernews # macOS, Linux
$ pip install fetch-hackernews  # Windows

As the name suggests, the hacker news from news.ycombinator.com can be displayed in the terminal. The libraries requests and beautifulsoup4 are used for this purpose. (Yes, I know there are already some similar programs you can download via pip.)

The list of headings and the associated links are stored locally as an HTML file. So, all news will be read from this local index.html file. This program will search for such a file. If no file has been created yet, it will create this file, download the content (using requests) and save it. After that, the content will be parsed using BeautifulSoup. By default, the index.html file is only updated every six hours. This reduces requests to the server from news.ycombinator.com. On macOS the index.html file is stored in the following directory:

~/.config/hackernews

The program can be started using this command:

$ fetch_hackernews

This will display the 30 most recent messages in the terminal. The output looks similar to the one shown below:

If you do not use Terminal on macOS, but the alternative iTerm, the links are clickable (after clicking the command key). This will open the default browser showing the corresponding article.

Last updated on September 30th, 2024