Palantir: Federated Search v2.0

Federated Search v2.0
brandt
Thu, 03/21/2019 – 09:46
Ken Rickard

We have released version 2.0 of our Federated Search application and Drupal integration.

Since our initial release, we’ve been doing agile, iterative development on the software. Working with our partners at the University of Michigan and the State of Georgia, we’ve made refinements to both the application and the Drupal integration.

Better search results

Default searches now target the entire index and not the more narrow tm_rendered_item field. This change allows Solr admins to have better control over the refinement of search results, including the use of field boosting and elevate.xml query enhancements.

Autocomplete search results

We added support for search autocomplete at both the application and Drupal block levels — and the two can use the same or different data sources to populate results. We took a configurable approach to autocomplete, which supports “search as you type” completion of partial text. These results can also include keyboard navigation for accessibility.

Since the Drupal block is independent of the React application, we made it configurable so that the block can have a distinct API endpoint from the application. We did this because the state of Georgia has specific requirements that their default search behavior should be to search the local site first, looking for items marked with a special “highlighted content” field.

Enter search terms field with list of suggested results

Wildcard searching

We fully support wildcard searches as a configuration option, so that a search for “run” will automatically pass “run” and “run*” as search terms.

Default facet control

The default facets sets for the application — Site, Content Type, and Date Range — can now be disabled on a per-site basis. This feature is useful for sites that contribute content to a network but only wish to search their own site’s content.

Enhanced query parameters

We’ve added additional support for term-based facets to be passed from the search query string. This means that all facet options except dates can be passed directly via external URL before loading the search form.

Better Drupal theming

We split the module’s display into proper theme templates for the block and it’s form, and we added template suggestions for each form element so that themes can easily enhance or override the default styling of the Drupal block. We also removed some overly opinionated CSS from the base style of the application. This change should allow CSS overrides to have better control over element styling.

What’s Next for Users?

All of these changes should be backward compatible for existing users, though minor changes to the configuration may be required, Users of the Drupal 8.x-2.0 release will need to run the Drupal update script to load the new default settings. Sites that override CSS should confirm that they address the new styles.

Currently, the changes only apply to Drupal 8 sites. We’ll be backporting the new features to Drupal 7 in the upcoming month.

Users of the 1.0 release may continue to use both the existing Drupal module and their current JS and CSS files until the end of 2019. We recommend upgrading to the 2.0 versions of both, which requires minor CSS and configuration changes you can read about in the upgrade documentation.

Special Thanks

Palantir senior engineer Jes Constantine worked through the most significant changes to the application and integration code. Senior front-end developer Nate Striedinger worked through the template design and CSS. And engineer Matt Carmichael provided QA and code review. And a special shoutout to James Sansbury of Lullabot — our first external contributor.

Development
Drupal
Open Source

Dries Buytaert: JSON:API lands in Drupal core

JSON:API being dropped into Drupal by crane

Breaking news: we just committed the JSON:API module to the development branch of Drupal 8.

In other words, JSON:API support is coming to all Drupal 8 sites in just a few short months! 🎉

This marks another important milestone in Drupal’s evolution to be an API-first platform optimized for building both coupled and decoupled applications.

With JSON:API, developers or content creators can create their content models in Drupal’s UI without having to write a single line of code, and automatically get not only a great authoring experience, but also a powerful, standards-compliant, web service API to pull that content into JavaScript applications, digital kiosks, chatbots, voice assistants and more.

When you enable the JSON:API module, all Drupal entities such as blog posts, users, tags, comments and more become accessible via the JSON:API web service API. JSON:API provides a standardized API for reading and modifying resources (entities), interacting with relationships between resources (entity references), fetching of only the selected fields (e.g. only the “title” and “author” fields), including related resources to avoid additional requests (e.g. details about the content’s author) and filtering, sorting and paginating collections of resources.

In addition to being incredibly powerful, JSON:API is easy to learn and use and uses all the tooling we already have available to test, debug and scale Drupal sites.

Drupal’s JSON:API implementation was years in the making

Development of the JSON:API module started in May 2016 and reached a stable 1.0 release in May 2017. Most of the work was driven by a single developer partially in his free time: Mateu Aguiló Bosch (e0ipso).

After soliciting input and consulting others, I felt JSON:API belonged in Drupal core. I first floated this idea in July 2016, became more convinced in December 2016 and recommended that we standardize on it in October 2017.

This is why at the end of 2017, I asked Wim Leers and Gabe Sullice — as part of their roles at Acquia — to start devoting the majority of their time to getting JSON:API to a high level of stability.

Wim and Gabe quickly became key contributors alongside Mateu. They wrote hundreds of tests and added missing features to make sure we guarantee strict compliance with the JSON:API specification.

A year later, their work culminated in a JSON:API 2.0 stable release on January 7th, 2019. The 2.0 release marked the start of the module’s move to Drupal core. After rigorous reviews and more improvements, the module was finally committed to core earlier today.

From beginning to end, it took 28 months, 450 commits, 32 releases, and more than 5500 test runs.

The best JSON:API implementation in existence

The JSON:API module is almost certainly the most feature-complete and easiest-to-use JSON:API implementation in existence.

The Drupal JSON:API implementation supports every feature of the JSON:API 1.0 specification out-of-the-box. Every Drupal entity (a resource object in JSON:API terminology) is automatically made available through JSON:API. Existing access controls for both reading and writing are respected. Both translations and revisions of entities are also made available. Furthermore, querying entities (filtering resource collections in JSON:API terminology) is possible without any configuration (e.g. setting up a “Drupal View”), which means front-end developers can get started on their work right away.

What is particularly rewarding is that all of this was made possible thanks to Drupal’s data model and introspection capabilities. Drupal’s decade-old Entity API, Field API, Access APIs and more recent Configuration and Typed Data APIs exist as an incredibly robust foundation for making Drupal’s data available via web service APIs. This is not to be understated, as it makes the JSON:API implementation robust, deeply integrated and elegant.

I want to extend a special thank you to the many contributors that contributed to the JSON:API module and that helped make it possible for JSON:API to be added to Drupal 8.7.

Special thanks to Wim Leers (Acquia) and Gabe Sullice (Acquia) for co-authoring this blog post and to Mateu Aguiló Bosch (e0ipso) (Lullabot), Preston So (Acquia), Alex Bronstein (Acquia) for their feedback during the writing process.

Wim Leers: JSON:API shipping with Drupal 8.7!

The JSON:API module was added to Drupal 8.7 as a stable module!

See Dries’ overview of why this is an important milestone for Drupal, a look behind the scenes and a look toward the future. Read that first!

Upgrading?

As Mateu said, this is the first time a new module is added to Drupal core as “stable” (non-experimental) from day one. This was the plan since July 2018 — I’m glad we delivered on that promise.

This means users of the JSON:API 8.x-2.x contrib module currently on Drupal 8.5 or 8.6 can update to Drupal 8.7 on its release day and simply delete their current contributed module, and have no disruption in their current use of JSON:API, nor in security coverage! 1

What’s happened lately?

The last JSON:API update was exactly two months ago, because … ever since then Gabe, Mateu and I are have been working very hard to get JSON:API through the core review process. This resulted in a few notable improvements:

  1. a read-only mode that is turned on by default for new installs — this strikes a nice balance between DX (still having data available via APIs by default/zero config: reading is probably the 80% use case, at least today) and minimizing risk (not allowing writes by default) 2
  2. auto-revisioning when PATCHing for eligible entity types
  3. formally documented & tested revisions and translations support 3
  4. formally documented security considerations

Get these improvements today by updating to version 2.4 of the JSON:API module — it’s identical to what was added to Drupal 8.7!

Contributors

An incredible total of 103 people contributed in JSON:API’s issue queue to help make this happen, and 50 of those even have commits to their name:

Wim Leers, ndobromirov, e0ipso, nuez, gabesullice, xjm, effulgentsia, seanB, jhodgdon, webchick, Dries, andrewmacpherson, jibran, larowlan, Gábor Hojtsy, benjifisher, phenaproxima, ckrina, dww, amateescu, voleger, plach, justageek, catch, samuel.mortenson, berdir, zhangyb, killes@www.drop.org, malik.kotob, pfrilling, Grimreaper, andriansyahnc, blainelang, btully, ebeyrent, garphy, Niklan, joelstein, joshua.boltz, govind.maloo, tstoeckler, hchonov, dawehner, kristiaanvandeneynde, dagmar, yobottehg, olexyy.mails@gmail.com, keesee, caseylau, peterdijk, mortona2k, jludwig, pixelwhip, abhisekmazumdar, izus, Mile23, mglaman, steven.wichers, omkar06, haihoi2, axle_foley00, hampercm, clemens.tolboom, gargsuchi, justafish, sonnykt, alexpott, jlscott, DavidSpiessens, BR0kEN, danielnv18, drpal, martin107, balsama, nileshlohar, gerzenstl, mgalalm, tedbow, das-peter, pwolanin, skyredwang, Dave Reid, mstef, bwinett, grndlvl, Spleshka, salmonek, tom_ek, huyby, mistermoper, jazzdrive3, harrrrrrr, Ivan Berezhnov, idebr, mwebaze, dpolant, dravenk, alan_blake, jonathan1055, GeduR, kostajh, pcambra, meba, dsdeiz, jian he, matthew.perry.

Thanks to all of you!

Future JSON:API blogging

I blogged about once a month since October 2018 about JSON:API, to get more people to switch to version 2.x of the JSON:API module, to ensure it was maximally mature and bug free prior to going into Drupal core. New capabilities were also being added at a pretty high pace because we’d been preparing the code base for that months prior. We went from ~1700 installs in January to ~2700 today!

Now that it is in Drupal core, there will be less need for frequent updates, and I think the API-First Drupal: what’s new in 8.next? blog posts that I have been doing probably make more sense. I will do one of those when Drupal 8.7.0 is released in May, because not only will it ship with JSON:API land, there are also other improvements!

Special thanks to Mateu Aguiló Bosch (e0ipso) for their feedback!


  1. We’ll of course continue to provide security releases for the contributed module. Once Drupal 8.7 is released, the Drupal Security Team stops supporting Drupal 8.5. At that time, the JSON:API contributed module will only need to provide security support for Drupal 8.6. Once Drupal 8.8 is released at the end of 2019, the JSON:API contributed module will no longer be supported: since JSON:API will then be part of both Drupal 8.7 and 8.8, there is no reason for the contributed module to continue to be supported. ↩︎

  2. Existing sites will continue to have writes enabled by default, but can choose to enable the read-only mode too. ↩︎

  3. Limitations in the underlying Drupal core APIs prevent JSON:API from 100% of desired capabilities, but with JSON:API now being in core, it’ll be much easier to make the necessary changes happen! ↩︎

Security advisories: Drupal core – Moderately critical – Cross Site Scripting – SA-CORE-2019-004

Project: 
Date: 
2019-March-20
Vulnerability: 
Cross Site Scripting
Description: 

Under certain circumstances the File module/subsystem allows a malicious user to upload a file that can trigger a cross-site scripting (XSS) vulnerability.

Solution: 

Versions of Drupal 8 prior to 8.5.x are end-of-life and do not receive security coverage.

Reported By: 
Fixed By: 

Drudesk: Useful modules to fix duplicate content in Drupal

Modules to fix duplicate content in Drupal

From a quarter to almost one-third of content in the World Wide Web repeats itself. According to Google’s head of search spam, Matt Cutts, around 25-30% of web content is duplicate. Your website is also likely to have duplicate content, even if it follows web content writing rules. In this post, we will touch upon the reasons and risks of duplication, as well as review useful modules that fix duplicate content in Drupal.

wishdesk.com: What to expect in Drupal 8.7: looking at its fresh alpha release

Drupal 8.6 became one of the most interesting releases in Drupal 8’s history. It brought us the oEmbed feature, the Media Library, the Workspaces module, and more. But it’s time to move forward, and in May 2019 we expect Drupal 8.7. Its “alpha” version has just been released. Although an alpha version is not a final one, we will gladly take a look at it and discuss what to expect in Drupal 8.7.

Drupal 8.7: the alpha version

Drupal 8.7.0-alpha1 has come out on March 14, 2019. Alpha versions are far from being ready for production sites. They are just preliminary releases that allow developers to do a good testing, receive feedback, make final preparations, and fix bugs.

Jacob Rockowitz: Webform module now supports importing submissions

Problem

The answer is Drupal’s Migrate API, which is incredibly powerful but can feel overwhelming. When I migrated MSKCC.org from Drupal 6 to Drupal 8, the Migrate API was just being introduced into Drupal 8 core, and I felt more comfortable writing a custom migration script instead of using code that was still under development. Migrate API is now stable and if you are an experienced Drupal developer, you should use it.

The level of expertise required to build and maintain a Drupal 8 website has changed from Drupal 7, mainly because we are creating more ambitious digital experiences. The Drupal community struggles to simplify our flexible and sometimes complex product. My approach is to make the Webform module as flexible and robust as possible, while not forgetting that people need a simple way to start building a form. This is exactly why I include an introduction video on the Webform module’s main page. Besides making the Webform module an awesome tool for experienced Drupal site builders, the Webform module needs to be welcoming to new users and make it easy for them to move their existing forms to Drupal.

Either an organization is starting from scratch and building a new Drupal site, or more commonly an organization has decided they need to provide a more ambitious digital experience and they have chosen to switch to Drupal. In both situations, we need to make it easy for someone to switch from other form builders to Webform.

The problem that needs to be addressed is…

Solution

The simplest way to migrate to the Webform module is to rebuild an external form and then import the existing data. Building a webform is fun and easy, forms are a critical aspect to most websites; it is worth taking the time needed…Read More