Specbee: How to Import External Feeds into your Drupal 9 Website with the Feeds Module

How to Import External Feeds into your Drupal 9 Website with the Feeds Module
Megha Kundar
03 Nov, 2020

As a developer, you will come across several situations when you need to import data from external sources. Sometimes you will need to import data from more than one source into your Drupal 9 website. The Feeds module in Drupal is a simple, no-fuss module that enables even non-technical users to import content from different data sources. What makes it simpler than other Drupal modules like the Migrate module, is that the entire import process is done through Drupal’s UI. It delivers content to you the way you want it as soon as it is available.
Many websites import RSS / XML feeds from third party sites. In this blog, we are going to get a news feed imported through this 3rd party API that provides data in json format with the help of the Feeds module in Drupal 9

Installing the Feeds Module

The Feeds Drupal 9 module lets you import data from different formats like CSV, XML, JSON, RSS, etc. into nodes, users, and taxonomy terms via the UI.
1.Install the Feeds module (Here we will be using composer)

composer require 'drupal/feeds:^3.0@alpha'

2. Here, we are using JSON feeds for importing data. So, we require a JSON parser which will need us to install one more module which is the Feeds extensible parser. Best way to install this module is by using composer because it downloads all required libraries. (You can also skip the first step if you’re downloading using composer as feeds will be a dependency for the feed_ex module).

composer require 'drupal/feeds_ex:^1.0@alpha'

Note : If you’re not using composer to install then the libraries required by this module can also be installed with the Ludwig module. Install the Ludwig module and then go to /admin/reports/packages for further instructions.

3.    Enable the feeds and feeds_ex (Feeds Extensible Parsers) module either by using drush or through UI.
Feeds Module

Configuring and Implementing the Feeds Module

Here, we are going to import the data from website articles (Topics related to “Top headlines from TechCrunch right now”) which is exposed in JSON format shown in the below screenshot. 

To get this news data we need to login to the website and get an API key. Next, select the topics for which you wish to import the data to your website.
feed module configurationLooking at the API, lets create a content type where our data needs to be imported.

Creating a Content Type 

Here, I have created a new content type called as “News”. Choose the field type according to your JSON for which data needs to be migrated. 
content type-feed moduleIn the above screenshot, we have chosen simple fields for importing the data like Author which is Text(plain) field for the importing data from JSON key author
JSON Key AuthorAs you see in the above screenshot, we have added the Feeds item field of type Feed and chosen the feed type accordingly. We will be learning how to create a Feed type in the next section below.

Creating a Feed Type 

Follow the below steps to create a feed type.

  1. Navigate to Feeds overview page (/admin/structure/feeds). You can see all the feeds listed in this page.feed type - feed moduleI have already created 2 feeds. Here I am going to be using News feeds. You can create your own feed type by clicking on Add feed type. 
  2.  When editing feed type or adding new feed type, we will see the below options.
  • Basic settings
    Let’s configure the Basic settings for the feed type, First, provide the Name, Description and Explanation or submission guidelines for the feed type.        
     Basic setting feeds moduleImport period (shown in the below settings tab) helps you choose how often the import should run. This will be done through a cron job. If you select Off then the import will be done manually.

feed modules

  • The Fetcher settings option shows where you’re importing the content or fetching the content from.
     It provides 3 options to choose from,
  1. Directory :- Import the content from a file or a series of files already on your website.
  2. Download from URL :- Import the content from url (Which we will be choosing in our case).
  3. Upload file :- Import content from a file.

                Based on the Fetcher selected, we will have to configure the Fetcher settings. For the secondary option of fetcher settings, we have the below list of settings.

Configure secondary settings accordingly.
Configure - feed modules•    Parser lets you configure the format of the feed. Here we have many options to choose from like CSV, HTML, OPML, JsonPath (which is our setting), etc. Choose the one from the dropdown menu according to your feed format.
Based on the parser selected, we will have to configure the parser settings. In our case we don’t have any parser settings for parsers of type JsonPath.
•    Processor settings show configurations of the type of content the feed will create.

Similarly, we have a secondary settings option for processor as shown in the image below.
feed modulesConfigure your secondary settings according to the requirement.
The below screenshot shows the primary settings of fetcher, parser and processor.
feed modules
3. Mapping sources to target

  • Context 

The help text says base query to run, which shows root or base of the data to be targeted to import the feeds as content. In our example, we have configured as “$.articles.*” because our data resides inside the article key. Look at this path documentation to configure yours.

  • Field Mappings

Now, we can configure the Feeds to decide which value pulled in from the JSON should map to which field on the Drupal news content type.
These targets are the Drupal data fields and are pulled in based on the processor you select when creating the feed type. You can also configure other settings for the particular mappings of the fields like Language, filter format (applicable for text formatted fields), unique (unique identifier used to track if the content is new or existing one), etc.
For example, if we want to import the JSON feed news article content title as Drupal news content title field then target will be the Drupal Title field and source will be JSON news article title key as shown in below screenshot.
JSON FeedsHere we are using the mapping configuration as shown in the below screenshot.
feed modulesYou can also clone the feed type using the Entity clone Drupal module that is also supported in Drupal 9.

Creating a Feed

To create a feed, navigate to content -> Feeds tab -> add feed -> select the feed type
feed modulesI have already created one for News. To create a new feed, click on the + Add feed button and configure the feed as required. For our blog, we have configured as shown in the below screenshot.
feed modulesAfter the configuration, click on Save and Import. This will import the feeds as content. When the data is getting imported, you will see the processing status as shown in the below screenshot.
feed modulesOnce the JSON data gets migrated to content, we can see the content added to the news content type as shown in the screenshot below.
feed modulesNow, let’s compare the imported content with feeds.

feed modules
    Source Content from the techcrunch article
feed modules
      The content imported with the Feeds module

For more detailed information you can refer to this guide.

Additional modules to extend the Feeds module 

•    Feeds Tamper module helps you modify and update he source data before it gets imported to your website.
•    Feeds extensible parsers contains a set of parsers like XPath XML & HTML, JSONPath JSON & JSON lines parser, etc.
•    Feeds Import Preview module lets you see the content in your source before importing it.
•    Commerce Feeds module helps you in importing products to your Drupal Commerce site.

Shefali ShettyApr 05, 2017