Some time ago I familiarized myself with the topic of web scraping. As a programming language, the decision quickly falls on Python, since there are good libraries available for this task. When dealing with a new topic, it is not a bad idea to do so creating a self-developed program. The result is: fetch-hackernews.
$ pip3 install fetch-hackernews # macOS, Linux $ pip install fetch-hackernews # Windows
As the name suggests, the hacker news from news.ycombinator.com can be displayed in the terminal. The libraries
beautifulsoup4 are used for this purpose. (Yes, I know there are already some similar programs you can download via
The list of headings and the associated links are stored locally as an HTML file. So, all news will be read from this local
index.html file. This program will search for such a file. If no file has been created yet, it will create this file, download the content (using
requests) and save it. After that, the content will be parsed using BeautifulSoup. By default, the
index.html file is only updated every six hours. This reduces requests to the server from news.ycombinator.com. On macOS the
index.html file is stored in the following directory:
The program can be started using this command:
This will display the 30 most recent messages in the terminal.
If you do not use Terminal on macOS, but the alternative iTerm, the links are clickable. This will open the default browser showing the corresponding article.
Even if you are not interested in hacker news, this program may be a good starting point for your own project.