Keeping on top of your RSS feeds can be a headache. If you have many feeds that you follow, then trying to keep tabs on the status of each one is just as tedious as difficult. Let’s be honest: having to dig through the online clutter and pull out the feed you want to monitor is a time-consuming process that nobody wants to experience.

These inconveniences created an itch that Sandeep Gupta couldn’t ignore, which spurred him to create an application that pulls all RSS feeds into one location. This provides users with a holistic view of all their RSS feeds, promoting a painless and efficient way of organizing their entire collection. Using different Redis modules, Sandeep was able to create an application that’s hyper-responsive to user commands, minimizing the chances of any lags or delays from happening. 

Using different Redis modules, Sandeep was able to create an application that’s hyper-responsive to user commands, minimizing the chances of any lags or delays from happening. 

Let’s examine how this application was put together. 

Step-By-Step Instructions

This article will cover how to build an app that pulls all your RSS feeds into one location through the following step-by-step instructions:

1. What Will You Build?

2. What Will You Need?

3. Architecture

4. Getting Started

5. Implementing the Redis Commands

6. Accessing the Hacking Capabilities

7. Navigating the app

1. What Will You Build?

You’ll build an application that will gather all of a user’s RSS feeds into one location to provide users with easy access to each feed. From start to finish, everything is about allowing users to gain a tighter grip on each RSS feed they have by making the management process as seamless as possible. 

This is mainly achieved through its dashboard, where users have a complete overview of their collection of RSS feeds. Users can scroll through their list of feeds, organize each one into different categories and gain access to important details such as post updates. 

Other features include:

• Clean and minimalistic UI

• Discover RSS/Atom feeds from a given site

• Add RSS/Atom feeds

• Import feeds and folders from an OPML file

• De-dupe posts with the same content across multiple feeds

• Multiple layouts

• Sort feeds by oldest/newest posts

• Filter to show only read/unread posts

• Mark favorite posts with a star

• Bookmark posts to read later

• View all star/bookmarked posts

• Create new folders

• Categorize feeds into folders

• View statistics on a graph to get a clear visualization of how frequently a feed updates their posts

• View self-reading habits (e.g. how many posts you read every day)

Below we’ll lay everything out for you to make building this application as easy as possible.

Ready to get started?  Ok, let’s dive straight in. 

2. What Will You Need?

• RedisCore: Used to check on all operations related to keys as well as storing post data

• RedisJSON: Used as the main document store; all post data and details on feeds are stored as JSON documents once they’ve been crawled

• RediSearch: Provides querying, secondary indexing, and full-text search for Redis

• RedisTimeSeries: Used to store user activity as well as the publishing behavior of feeds

• RedisBloom: Used to de-duplicate entries across all feeds 

• JDK 11: Used as a software development environment for building Java applications and applets

• Spring Boot: Used as an open-sourced Javascript framework

• TypeScript: Used as the preferred programming language to build large applications

• React 17: Provides you with gradual React upgrades

• Bootstrap 5.0: Used for creating a responsive, mobile-first website

• Apache Maven (for building server): Used to help build and manage Javascript projects

• NPM/Yarn (for building clients): A popular package manager for the JavaScript programming language

3. Architecture

   +-------+              +--------+                     +------------+
   | React |    REST      | Java   |  Redis Protocol     | Redis with |
   |  UI   |------------> | Server |-------------------> | modules    |
   +-------+              +--------+                     +------------+
                              |
                              | Continual
                              | polling          +------------+
                              +----------------> | Feed Hosts |
                                                 +------------+

This is a classic 3-tiered application, where the web tier connects with the service tier over the rest of the services. All data persistence happens inside Redis. The app utilizes continual polling for fetching new posts from RSS/Atom feeds. 

This is limited since there is no easy way of receiving webhooks on http://localhost for feeds with PubSubHubBub enabled. The code can however be extended to use tunneling via ngrok and localtunnel. If not, the app will make use of PubSubHubBub to reduce polling.

Note: Not all feeds/sites support hooks, and therefore, polling might still be required. 

4. Getting Started

Prerequisites:

Step 1: Clone the Repository

git clone https://github.com/redis-developer/reread

Step 2: Start the Docker

docker run -d -p 6379:6379 redislabs/redismod

Step 3. Install the Dependencies

Change directory to reread repository and install the dependencies.

$ cd web-ui

$ npm install

Step 4. Create the Build for the Web Client

$ npm run build

$ cd ..

Step 5. Create the Build for the Server

$ cd server

$ cp -r ../web-ui/dist/* src/main/resources/static

$ mvn clean package

$ cd..

Step 6. Start the Project

$ java -jar server/target/reread.jar

5. Implementing the Redis Commands

When the Server Starts Up

This is used to illustrate the activity of the graph as well as the Kickstarter of the application. 

// create startup time

SETNX $reread-start-time {currentSystemTime}

// create search index

FT.CREATE post-search SCHEMA {...fields with weights}

Loading the Home Page

Use the command below to load the homepage. 

// Get the `FeedList` entity for the user
JSON.GET me {feedList}

// If it does not exist, create one
JSON.SET me . {feedList}

//Next get the first page of `all` timeline for the user
ZREVRANGE timeline:$all 0 {pageSize}

// For each ID get the actual post
GET {postID}

Adding a New Feed

Use the command below to add a new feed. 

// First the feed is discovered using HTTP call to URL

// once user chooses the feed to be added

// convert the feed to MasterFeed entry

JSON.GET masterFeed:{id}

 

// if not present, create one

JSON.SET masterFeed:{id} . {masterFeed}

 

// get the feed list

JSON.GET me {feedList}

 

// If it does not exist, create one

JSON.PUT me {feedList}

 

// feed crawling starts

// refer to section on "Feed Crawling" to take a look at its commands

 

// once feed is crawled

// update the feed list

JSON.SET me . {feedList}

Adding a Feed in a Folder

The commands used in this feature are a combination of the commands used in the "adding a feed" command above. The extra commands used here are:

// get a list of all feeds in the folder

JSON.GET feedList:me

// now create a merged union store of all timelines

ZUNIONSTORE timeline:{folderID} {feedID1} {feedID2} {feedID3} {feedID4}

Unsubscribing a Feed

This was designed to use ZDIFFSTORE to remove entries efficiently, but the command is available to start Redis 6.20. However, the redismod Docker image is available on the latest head that contains Redis 6.0.1. 

// get feed list

JSON.GET feedList:me



// read all entries from this timeline

ZRANGE timeline:{feedID} 0 -1



// remove the feed from folder timeline

ZREM timeline:{folderID} {...entries}



// remove the feed entries from all timeline

ZREM timeline:$all {...entries}



// update the feed list

JSON.SET feedList:me . {feedList}

Importing an OPML File

Use the command below to import an OPML file. 

// get feed list

JSON.GET feedList:me



// read all master feeds - this is more of performance improvement

KEYS masterFeed*

GET masterFeed:{masterFeedID}



// check if master feed already exists

// if not create a new master feed

JSON.SET masterFeed:{masterFeedID} . {masterFeed}

Exporting an OPML File

Use the command below to export an OPML file. 

 JSON.GET feedList:me

Crawling a Feed

The command below incorporates all of the Redis components in the architecture.

// get the master feed

JSON.GET masterFeed:{masterFeedID}

 

// get previously crawled details

JSON.GET feedCrawlDetails:{feedID}

 

// crawl the feed here

 

// once crawled, update details like etag/last modified time

JSON.SET feedCrawlDetails:{feedID} .lastCrawled {currentSystemTime}

 

JSON.SET masterFeed:{feedID} .title {parsedFeed.title}

JSON.SET masterFeed:{feedID} .siteUrl {parsedFeed.siteUrl}

 

JSON.SET feedCrawlDetails:{feedID} .lastCrawled {parsedFeed.lastCrawled}

JSON.SET feedCrawlDetails:{feedID} .lastModifiedHeader {parsedFeed.lastModifiedHeader}

JSON.SET feedCrawlDetails:{feedID} .lastModifiedTime {parsedFeed.lastModifiedTime}

JSON.SET feedCrawlDetails:{feedID} .etag {parsedFeed.eTagHeader}

 

// filter already existing posts

BF.EXISTS bloom:hash {post.hash}

BF.EXISTS bloom:text {post.text}

BF.EXISTS bloom:uniqueID {post.uniqueID}

 

// now start saving all filtered posts

BF.ADD bloom:hash {post.hash}

BF.ADD bloom:text {post.text}

BF.ADD bloom:uniqueID {post.uniqueID}

 

// save each post

SET {postID} {post}

 

// index each post

FT.ADD postSearch {postID} 1 FIELDS {...fields}

 

// send for analytics

TS.MADD timeseries-feed:{feedID} {post.updated} 1

 

// update timelines as needed

ZADD timeline:{feedID} {postID} {post.updated}

ZADD timeline:{folderID} {postID} {post.updated}

ZADD timeline:$all {postID} {post.updated}

Marking a Post Read/Unread

The JSON.GET/JSON.SET command was originally used, but due to an encoding bug in the JreJson driver, the GET/SET command had to be used instead. The bug has been filed as:

https://github.com/RedisJSON/JRedisJSON/issues/37

// marking read

SET {postID} {post}

// marking unread

SET {postID} {post}

Viewing a Feed Timeline

The below command is used to view a feed timeline.

// when viewing by newest, first page

ZREVRANGE timeline:{feedID} 0 {pageSize}

 

// when viewing by newest, second page

ZRANK timeline:{feedID} {lastPostID}

ZCARD timeline:{feedID}

ZREVRANGE timeline:{feedID} {card - rank + 1} {card - rank + pageSize}

 

// when viewing by oldest, first page

ZRANGE timeline:{feedID} 0 {pageSize}

 

// when viewing by oldest, second page

ZRANK timeline:{feedID} {lastPostID}

ZRANGE timeline:{feedID} {rank + 1} {rank + pageSize}

Viewing a Folder Timeline

Use the command below to view a folder timeline.

// when viewing by newest, first page

ZREVRANGE timeline:{folderID} 0 {pageSize}

// when viewing by newest, second page

ZRANK timeline:{folderID} {lastPostID}

ZCARD timeline:{folderID}

ZREVRANGE timeline:{folderID} {card - rank + 1} {card - rank + pageSize}

// when viewing by oldest, first page

ZRANGE timeline:{folderID} 0 {pageSize}

// when viewing by oldest, second page

ZRANK timeline:{folderID} {lastPostID}

ZRANGE timeline:{folderID} {rank + 1} {rank + pageSize}

Viewing Feed Details

The command below is used to view feed details. 

// get the master feed

JSON.GET masterFeed:{feedID}

 

// get details about feed when was last crawled

JSON.GET feedCrawlDetails:{feedID}

 

// get number of total posts

ZRANK timeline:{feedID}

 

// also get the chart data

ZRANGE timeline:{feedID} 0 0

 

// get the post

GET post:{postID}

 

// get analytics

TS.RANGE timeseries-feed:{feedID} {post.updated} {currentSystemTime} COUNT 60000

Viewing Activity

Use the command below to view activity. 

// read start of reread universe

GET $reread-start-time

 

// get activity chart data

TS.RANGE timeSeries-activity:{activityID} {rereadStartTime} {currentSystemTime} COUNT 60000

Searching for Posts

The current Java driver for RediSearch doesn’t provide an API to query in a specific index, causing a small hiccup. Because of this, the filtering of posts in the given index is done in the JVM. 

Click here and here to see the driver limitations. Below is the command used to search for posts. 

FT.SEARCH {query}

6. Accessing the Hacking Capabilities

The Front-End

The front-end application is written in TypeScript and uses React underneath. Use the following command to make changes to the UI client:

# Install dependencies

$ cd web-ui

$ npm install (or yarn install)

$ npm run watch

This will start a local development server on http://localhost:1234 where the React application is continuously being built and deployed. 

The Back-End

The backend application is written in Java and uses Spring Boot. All custom beans are defined in the SpringBeans.java file. You should also know that all services are wired to their implementation using the @Service annotation. Javadocs are mentioned over the classes as well as the methods that indicate their function. 

7. Navigating the application

Viewing Your RSS Feeds on the Homepage

The homepage loads an aggregated timeline of all of your RSS feeds. The timelines are stored as sorted members in the Redis cache. 

Here you’ll have a complete overview of all your RSS feeds. 

Filtering Your RSS Feeds

To filter your RSS feeds, click on the list view category at the top of your screen. When you do this, you’ll have a drop-down menu that will allow you to filter your RSS feeds based on:

• Magazine View

• Masonry View

• Folder View

• Title Only view

• Feed Details

• Post Read Activity

• Post Bookmarked Activity

Magazine View

Masonry View

Folder View

Title Only

Feed Details

Post Read Activity

Post Bookmarked Activity

Conclusion: Manage All of Your RSS Feeds in One Location With Redis

Trying to manage a disconnected network of RSS feeds is a frustrating process that nobody wants to undergo. It’s time-consuming, it’s tedious, and it’s an experience everyone wants to avoid. These barriers inevitably discourage people from regularly checking on the updates of their RSS feeds, creating more of a disconnect between themselves and the content they wish to follow.

To get a visual insight into how this app was created, watch Sandeep’s YouTube video here. 

Check it out. Be inspired.