html2rss-web

🕸 Generates RSS feeds of any website & serves to the web! Automatic scraping. Ready to use configs. Write your own. Rolling Docker releases for speedy updates.

Description

This is a tiny web application to expose HTTP endpoints which deliver RSS feeds built by the html2rss gem.

Out of the box you'll get all configs from html2rss-configs. You can - optionally - create your own configs and keep them private.

Programming language: Ruby

License: MIT License

Tags: Rss     Feed Readers     RSSminer     Docker     Scraper    

README

html2rss-web

This is a small web application to deliver RSS feeds built by html2rss via HTTP.

Features:

• serves your own feeds: set up your feed configs in a YAML file. See html2rss' README for documentation.
• comes with all html2rss-configs included.
• handles caching and HTTP Cache-Headers.

This web application is distributed in a rolling release fashion from the master branch.

💓 Depending on this application? Feel free to donate! Thank you!

Using the included html2rss-configs

Build the URL like this:

The feed config you'd like to use:
lib/html2rss/configs/domainname.tld/whatever.yml
‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌^^^^^^^^^^^^^^^^^^^^^^^^^^^

The corresponding URL:
http://localhost:3000/domainname.tld/whatever.rss
‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ^^^^^^^^^^^^^^^^^^^^^^^^^^^

👉 See file list of all html2rss-configs.

Deployment with Docker

Install Docker CE and docker run -d -p 3000:3000 gilcreator/html2rss-web.

To use your private feed configs, mount a feed.yml into the /app/config/ folder.

docker run -d --name html2rss-web \
  --mount type=bind,source="/path/to/your/config/folder,target=/app/config" \
  -p 3000:3000 \
  gilcreator/html2rss-web

Automatic updating

A primitive way to automatically update your Docker instance is to set up this script as a cronjob:

#!/bin/bash
set -e
docker pull -q gilcreator/html2rss-web
(docker stop html2rss-web && docker rm html2rss-web) || :
docker run -d --name html2rss-web --restart=always -p 3000:3000  \
  --mount type=bind,source="/home/deploy/html2rss-web/config,target=/app/config" \
  gilcreator/html2rss-web

The cronjob for updating every 30 minutes could look like this:

*/30 *  * * * /home/deploy/html2rss-web/update > /dev/null 2>&1

Heroku one-click deployment

Since this repository receives updates frequently, you'd need to update your instance yourself.

Run it locally

1. Install Ruby >= 2.6.
2. gem install bundler foreman
3. bundle
4. foreman start

html2rss-web now listens on port 5000 for requests.

Build and run with docker locally

This approach allows you to play around without installing Ruby on your machine. All you need to do is install and run the Docker daemon.

# Build image from Dockerfile and name/tag it as html2rss-web:
docker build -t html2rss-web -f Dockerfile .

# Run the image and name it html2rss-web-dev:
docker run \
  --detach \
  --mount type=bind,source=$(pwd)/config,target=/app/config \
  --name html2rss-web-dev \
  html2rss-web

# Open a interactive TTY with the shell `sh`:
docker exec -ti html2rss-web-dev sh

# Stop and cleanup container
docker stop html2rss-web-dev
docker rm html2rss-web-dev

# Remove the image
docker rmi html2rss-web

Feed configs runtime health checks

Websites often change their markup. To get notified when one of your own configs break, use the /health_check.txt endpoint.

It will respond with success if your feeds are generatable. Otherwise it will not print success, but states the broken config names.

Supported ENV variables

Name	Description
PORT	default: 3000
RACK_ENV	default: 'development'
RACK_TIMEOUT_SERVICE_TIMEOUT	default: 15
WEB_CONCURRENCY	default: 2
WEB_MAX_THREADS	default: 5