Skip to main content

"Headless Drupal" - The Cake is a Lie

“Headless Drupal” - If you are more or less seriously involved with Drupal, chances are you’ve observed this trend or even participated in discussions on the topic. It keeps popping up on social media, in blog posts and during Drupal events around the globe. It’s nothing short of a pandemic. An increasing number of presentations and birds-of-a-feather sessions at DrupalCons and DrupalCamps alike are attracting more and more human beings to spread it even further: By now, surely every other local user group has seen a presentation during one of their meetups and there is even a groups.drupal.org group dedicated to it.

At the time of this writing, a huge part of the community has discovered Angular and/or combinations of Backbone, Ember, Mithril and others. One can only imagine the impact that React will have once it becomes more popular in the Drupalverse.

What does “Headless Drupal” mean?

Before we continue, let’s examine what we are talking about. Simply put, “headless” is the idea of decoupling the frontend and the backend. Instead of generating a themed representation of your website, you are providing an interface for extracting pure, uniform data, e.g. JSON, which can then be transformed and rendered independent from the backing application (Drupal in this case).

This blog post is targeted at the client side or server/client (isomorphic) Javascript driven flavor of what I consider “Headless Drupal”. Of course, you could also say that a static site generator using Drupal as a data hub is “Headless Drupal”. I am sure there are other valid interpretations as well. Also, we are talking about fully headless websites here. If your website contains some arguably decoupled components driven by a subset of the aforementioned libraries and frameworks then that’s not what I understand as “headless”. You can call that “earless” or “eyebrowless” or even “cheekboneless” if you like. When I talk about “headless”, it really means “headless”... All the way.

Why are we experiencing this recent hype?

While the idea of “Headless Drupal” is certainly not new, the WSCCI (Web Services and Context Core Initiative) and its outcome plays a major role in the recent hype. This is how it describes itself:

The Web Services and Context Core Initiative (WSCCI) aims to transform Drupal from a first-class CMS to a first-class REST server with a first-class CMS on top of it.

There is no doubt that the WSCCI initiative is a great undertaking and has produced wonderful advancements that will greatly impact our daily work with Drupal. In Drupal 8 we will be able to build on a thoughtfully architected Router and Serialization API as well as other subsystems like the Entity- and the Typed Data API that are much more sophisticated than their Drupal 7 counterparts. I bow to the fine folks of the community who have made this possible.

When the REST module and the Serialization API finally made their first appearance in Drupal 8, a series of rejoicing blog posts appeared on the web, heralding the era of “Headless Drupal”. There was very little push back or any form of grounded, rational evaluation of the situation and I can’t deny that I was equally, if not more excited about the seemingly endless possibilities that suddenly emerged on the horizon.

Rationalism was being drowned in a huge wave of hype. There was little push back apart from some critical or annoyed Tweets. Looking at it now, the most disturbing part of this ongoing hype is that people suddenly advertise and talk about “Headless Drupal” as a replacement for the built-in Drupal theming layer. No more heavy markup, no more useless divs and classes and certainly no more “The Angry Themer” talks from Morten. It sure sounds nice. But it’s not that simple. It really isn’t.

I hate to spoil the party, but here’s the truth: If you expect to be able to build your next project by simply configuring a few resources and putting together some React components or Angular directives you’re in for a disappointment. It’s going to take a little bit more than that.

Are you ready to do without your favourite features?

Drupal 8 is going to be an immensely powerful CMS. A huge part of the success of Drupal stems from its enormous potential for customization. Drupal has a long standing tradition for enabling a very broad audience to build even complex websites simply through configuration in the backend. It also comes with a very sophisticated multilingual system for both, the content and the interface. I think we can all agree that there are many good reasons for choosing Drupal.

It is important to realize that many of these are, at least partly, nullified once you make the move towards headless.

Multilingual

Fetching the content in the right language is not a big deal. However, since you do not have access to Drupal’s string translation, which is also not desirable, you will have to take care of translating the interface yourself.

Modifications and ongoing maintenance

All modifications you make to your data schema or user interface are most likely going to require changes to your code on both sides of your application. Many of the configurations on the backend become insidious because they will have zero effect on the frontend. Sitebuilding is pretty much reduced to setting up the data schema and access levels.

Editorial user experience

It’s unlikely that you will have any linking between your backend and frontend making editorial tasks cumbersome.

Accessibility

As @_nod points out (thanks!), a lot of effort has been put into making Drupal 8 as accesible as possible. Once you drop the native Drupal rendering and theming layer you will have to take care of this yourself. Crafting a truely accessible website can be really challenging and there are lots of things that can be easily missed.

This is just a short list and there are definitely more things that you will have to wave goodbye.

There are more universal challenges too

Overall performance

This is a huge problem. For a structurally simple website this probably doesn’t hit you as hard. You can always use very sophisticated caching strategies, too. However, it is undeniable that you are producing a considerable overhead through the amount of additional roundtrips to the server. When using standard REST resources you are going to need multiple requests at least for the initial rendering of the page. You are going to see your browser stuttering as it spits out one piece of the website after the other as it receives the requested data one response at a time.

Even subsequent routes are likely to produce more than one request to the server. This all comes at the expense of performance both on the server (extra bootstraps) and the client (extra requests that are potentially even blocking each other).

Application size and complexity

Building a complex application in Javascript is not easy. The Javascript file size for a moderately large application can easily reach a couple megabytes. Structuring the application properly and consistently and a proper fragmentation strategy for lazily loading the required Javascript on demand is non-optional. This requires significant experience in and a deep understanding of the problem space.

Javascript dependency

In the worst case scenario you would impose a client-side Javascript dependency on your users. This is not so much of a problem itself given that search engines are becoming better and better at properly analyzing Javascript driven websites, however, this means that you are now relying on the browser for the rendering of your application. This is bad. Browsers change and so does Javascript. Future versions of your favourite browser might, at some point, drop support for some functionality you are relying on. You can mitigate this problem as well as sluggish, initial rendering of your application by making it isomorphic.

You see, it is not all rainbows and unicorns. Building a truly functional and user-friendly experience with a completely decoupled user interface is hard and demands extensive planning and research of the different solutions for the aforementioned problems.

One of the harder problems: Traditional REST is a dead end

You might find this rather ironic considering that the development of the REST module was one of the key factors that sparked this hype.

However, for a fully decoupled, highly interactive front-end solution for your application, traditional REST and plain entity schema backed resources won’t cut it. I am assuming that in most of these cases your data will be highly relational. Rendering related views with interdependent data requirements will inevitably result in an ever growing number of requests or exponentially increasing complexity of your custom tailored resources which, at that point, probably can’t be called RESTful anymore anyways.

Assuming that you want to keep your resources as agnostic and modular as possible (after all, one of your goals was to decouple the whole thing, right?) you are going to run into performance problems both on the client as well as on the server.

First of all, each browser implements a different limit of parallel HTTP requests per server. The original HTTP/1.1 RFC suggested a limit of 2. While in reality this limit is much higher in modern browsers (Chrome, for example, now allows 6 parallel connections) it is still something to keep in mind. Blocking HTTP requests are nasty. Especially considering that things like images and other assets fall under this rule as well. You can somewhat mitigate this problem by moving your assets to a CDN and using wildcard subdomains. Keep in mind that this will affect your client-side caching.

Blocking HTTP requests will make your application feel extremely slow and completely undermine your user experience, no matter how much work you put into your user interface. This especially affects the initial page load when each of the pieces with data needs is rendered one after the other as the data flows into your view.

But putting aside browser HTTP request limitations, there is still the more serious problem that multiple separate requests will all cause independent bootstraps on the backing application and we all know that Drupal is not particularly famous for it’s shallow bootstrap.

This one is tricky.

There might soon be solution to this problem

What if you could accumulate the data needs for all of your views and express them as a single query that runs through a single, shared resource on your backend. What if that resource could then resolve the query into a set of specific root calls that then get processed accordingly before they are merged back together as a single, structured response?

Two months ago Facebook announced that they are going to release a new client side library called Relay as well as a specification and reference implementation of a DSL called GraphQL.

While neither the specification, nor any of the reference implementations or Relay are publicly available yet, Facebook is providing us with a constant stream of informatino through various different channels. Based on this information, I’ve started working on a few things including a GraphQL Lexer and Parser in PHP5 and, specifically for Drupal, a Content Entity adapter built around a more generic Typed Data adapter.

I am going to write a more in-depth blog post about this specific undertaking once I am comfortable making it public on GitHub and drupal.org. The Drupal specific adapter will be hosted at http://drupal.org/project/graphql.

Wird nicht veröffentlicht
  • One important point not mentioned in the article is accessibility. There was a lot of work to make Drupal frontend accessible, and I'm sure it will be swept under the rug once people do their own thing.

  • Right, absolutely! While Javascript and accessibility certainly don't contradict each other it's definitely one of the more challenging aspects of front-end development. It's another thing that you are basically waving goodbye when walking down this path.

    And there are definitely more things than the ones listed. I do not claim completeness of the list of pain points / things to keep in mind. Consider them examples. I am going to add accessibility to the article though since I think it is one of the more important ones! Thanks

  • Looking forward to your future posts on the topic, Sebastian! I might push back on the question of chattiness by suggesting that an intelligent but abstract media type can reduce the number of HTTP requests required for a pageload, even for the front page of a site. That may be through the use of embedded resources or through a slightly advanced use of a HAL+JSON or Collection+JSON where you request a collection of entities and know how to follow references in the result set.

    For example, in rendering a shopping cart in a mobile app, I need order, line item, and product data, but there's nothing preventing me from using a media type that knows how to embed full entities in another representation (kind of what I did in Commerce Services) or simply is type agnostic so that you can inspect the entity type and IDs to put things together in the right order client side.

    That doesn't really "solve" the issue, as it just creates client-side engineering challenges that we haven't really addressed yet, but it at least removes the "REST is a dead end" blocker (imo) and refocuses us on building the tooling for and making use of more suitable media types.

  • Excellent writeup! It matches what we've found on a number of decoupled projects over the last 4-5 years. (We prefer that to "headless" for clarity purposes…)

    Sites that have been dealing with complex multichannel publishing requirements (driving native mobile apps from the same source data as the web site, for example) have had to grapple with these problems for a while, and it's great to see someone else pointing them out. Decoupling isn't a quick fix for frustrating Drupalisms—it's a way to future-proof site architecture, and it comes with its own set of problems to solve. They aren't insurmountable, by any means, but the solutions aren't yet standardized and baked-in yet.

    Ryan's comment above is a good example of that; consolidating the requests needed to build a page is a great way to reduce the "chattiness" of an API-driven approach to page building! But the tools to make that a reality aren't turnkey, just-install-and-go options the way something like page caching in Drupal is. It'll take time!

    We're in the early days of Drupal decoupling, and the good news is that we're moving in the direction that the rest of the web world is (rather than fighting against it). We've enjoyed the benefits of fully integrated admin/management/delivery for years, and now we're having to figure out what a world with looser connections looks like.

  • Thanks for your comments, Ryan and Jeff! I am glad that you agree with the general verdict and it's good to know that you have experienced the same issues.

    I hope I did get the message across that these problems are all but insurmountable. It's definitely possible to work around some of the problems as you pointed out. However, it's far from simple and that fact is constantly swept under the carpet (at least that is my perception).

    I agree that proper design of the API and leveraging more advanced media types can improve the situation. However, I am still convinced that we can do even better. I want to reach a point where the API can be treated as an implementation detail. This is exactly why I am working on the GraphQL implementation (I am going to be in Montpellier next week and will spent most of my time there working on that).

    Btw. I absolutely agree that "headless" is a horrible word for what we are talking about here. Decoupled is way more fitting. I actually saw a post on d.o arguing for this as well as your Tweet before writing this blog post. But since "headless" is more commonly used (I think?!) I thought it would be better to stick with it at least for this post to get a little more impact on Twitter. I am an attention whore, sorry :P.

  • Let me reply by quoting https://codeascraft.com/2015/04/06/experimenting-with-hhvm-at-etsy/

    The second problem was how tempting it became for engineers to build lots of general API endpoints that could be called from many different mobile views. If you use too many of these endpoints to generate a single view on mobile you end up degrading that view’s performance. Ilya Grigorik’s “Breaking the 1000ms Time to Glass Mobile Barrier” presentation explains the pitfalls of this approach for mobile devices. To improve performance on mobile, we decided to create API endpoints that were custom to their view. Making one large API request is much more efficient than making many smaller requests. This efficiency cost us some reusability, though. Endpoints designed for Android listing views may not have all the data needed for a new design in iOS. The two platforms necessitate different designs in order to create a product that feels native to the platform. We needed to reconcile performance and reusability.

    Either this or what you plan to do with GraphQL are the only solutions I see for the performance problem (i.e. "too many requests, thus too many round trips, exacerbated by high latency").

  • Thanks for the link, Wim. Good read!

    Crafting view-specific resources is certainly a lesser evil, but surely not the final solution. Maintaining view-specific resources comes at a high cost. Once you do this, you are again tightly coupling your front-end implementation with the backend. This drastically slows down development. Every change you make on either side of the application requires you to fiddle with the other side as well. You are going to slow down even further once the application reaches a certain complexity. Your application will also become more error-prone. It's a band-aid solution imho.

  • Oh, absolutely, this comes with a very high cost!

    I wouldn't go so far as saying that any change on the back-end will need the front-end to be updated, but surely many kinds of changes would require that.

    Related, but speaking more generically: all these downsides of decoupled (headless) Drupal/any CMS point to the appeal of an integrated solution like Drupal: the system is designed to be aware of changes in the configuration made by the site builder. I'm certain we could make the front-end (the JS) be aware of configuration changes in a similar way, but that'd require a lot of abstraction and code on the front-end. Which means more code to load, worse performance, and … not so much of an improvement. Most of the appeal from headless/decoupled Drupal is actually in the fact that as a front-end developer, you're completely in control again. Liberation from Drupal's concepts, coding patterns, coding standards, conventions, from all of it! But… all of that "baggage" actually is there for a reason: to have a translatable, overridable, customizable, configurable, accessible, *able system. You also said this, but I think it's worth stressing even more.

    Right now, from what I can tell, the headless/decoupled Drupal "movement" (if that's the right word) is the effect that the "JS renaissance" has triggered with Drupal front-end people. It's an understandable reaction, coming in a large part from the typical frustrations front-end development in Drupal causes. But let's not forget why Drupal's front-end is the way it is: because it is designed to support site builders, who just click & configure everything together, without writing any code.
    In other words: this is another instance of the everlasting "better code vs. configuration" (for developers vs site builders) and "better markup/CSS/JS vs. configuration" (for themers vs site builders) debate/tension. It is incredibly hard to support both configuration and simple/elegant code (front- or back-end) at the same time.
    There's no reason why "regular Drupal" and "headless Drupal" cannot co-exist, and there are use cases for both. But it's definitely no silver bullet.

    All that being said: I am hopeful that better solutions/approaches will arise from headless/decoupled Drupal :) At worst, we'll learn valuable lessons! This is a very interesting problem space to say the least.

    P.S.: I think headless/decoupled Drupal in the forms that it exists today can be captured in a much simpler description: "Drupal as a structured content repository". That IMHO clarifies the extent to which Drupal itself is still involved in sites built this way.

  • One more thing: AFAIK almost every site out there that uses REST APIs doesn't solely use REST to build pages. It uses REST on the back-end, to talk to all the services necessary to build pages. And the JS on the front-end may very well use REST to perform certain actions, but usually not for the majority of content/interactions.

    I wonder if you have counterexamples? Or do you agree with this observation?

  • Wim, I think you're definitely correct about the influence of the JS renaissance on the decoupled/headless movement. It's also important for folks with narrow use cases (the best ones for fully decoupled sites at the moment) to remember the range of use cases that Drupal's highly-coupled front and back ends are intended to cover.

    In our experience, UI localization is one of the biggest adjustments: it's something that many Drupal front-enders have literally never had to deal with because of the supporting structure the tightly coupled solution provides. There are per-framework solutions, but they all mean that the front end team will end up taking much more responsibility for the nitty gritty of stuff like pluralization. (Yayyyyyy!)

    Aggregating and pre-assembling assets per page-type (i.e., 'give me all of the stuff needed for an article page') is really effective, and doesn't have to be treated as a "re-coupling." I tend to think of it as just another component in a decoupled architecture. The "editorial and storage" layer supplies content and metadata for the "assembly" layer, which supplies efficiently-packaged resources for the "presentation" layer.

    Finally, it's worth remembering that decoupling portions of the content management and delivery architecture isn't a new thing at all. Rather, it's a pendulum-swing in the decades-old evolution of digital publishing and content management. Drupal was one of the young upstarts that rose to prominence as the "decoupled era" was losing steam: the approaches made a lot of sense when web publishing was an adjunct to the "real" work that happened in print publishing systems. The dominance of the web made strongly-coupled systems more palatable, but today the growing important of multi-channel publishing is putting a lot of strain on web-first systems like Drupal.

    That, I think, is where a lot of the interest in decoupling is coming from on the technical/tactical side. It brings lots of challenges, but they're challenges that mobile app teams and business API teams were having to grapple with on large projects anyways.

  • One last thought — it was mentioned that "editorial backend integration" was hard to accomplish with a decoupled system. I've actually been chatting with a group that built some fascinating solutions to that problem. Instead of killing themselves to tie the two layers together, they just passed the NID of each article to the front end, stuck it in a data-attribute, and installed a Chrome extension on each editor's machine that looked for the data attribute and enhanced their view of the public front page with editorial links derived from the nid. It was a really elegant solution that worked great for them, but it required getting out of the "slam it all into the HTML payload" mindset that strongly coupled systems make easy.

  • Great post and great discussion. Amitai and I were just talking about this and want to represent this as part of our presentation at DrupalCon LA. Hopefully nobody minds if we lift some quotes? ;)

    I realize the title is meant to be provocative, but I don't think you can call it a "lie" when people have been doing decoupled work with Drupal with great success for several years. However, the idea that this is a simple, straightforward thing is clearly not true.

    To me the biggest benefit is that decoupling mandates that clear and effective decisions are made at a technical level about what a site is doing, and for whom. The architecture enforces the "separation of concerns", and pushes a ton of complexity to the front-end.

    That can be a disaster if you can't/don't make the right decision, or if your front-end isn't up to the job. It's also a disaster for anyone to whom "there's a module for that" is frequently used to solve the growing needs of sites.

    However, there's a reason the whole internet is moving in this direction: it makes amazing things possible, and it allows amazing developers to collaborate in ways that are not possible (or at least extremely difficult) when working on monolithic architectures.

  • Thanks for all the great comments so far. I am a bit busy right now due to some work related things that need to be solved before DevDays. Will join the discussion again during DevDays!

    @Josh: Hey! Yes, this is definitely meant to be a provocative title. "The Cake is a Lie" is more targeted at the overall hype and the misconception that it spreads that "It's all fluffy and easy now!".

    Of course it is possible, but people seem to forget that it is still extremely hard!

    I had some weird nightmares after I overheard a few people at a local event talking about how they are planning to use Angular with Drupal 8 instead of Twig.

    People seem to really think it's just that: Plug & Play. Nope.

  • Hello, nice article. However there are some points i don't get...
    - "Mutilingual" : yes we still can take advantage of drupal multilanguage system because we can serve differente resources depending on the language prefix in url, and we do have access to the "t()" function when generating resource.

    - "All modifications you make to your data schema or user interface are most likely going to require changes to your code on both sides of your application"

    Your output is formatted json, so even if you database changes, you may ouput the same json to keep compatibility. And if you have normal drupal html, you do have to change both html and controllers if you are relying on a field name ... so what's the difference ?

    - "It’s unlikely that you will have any linking between your backend and frontend making editorial tasks cumbersome."

    You can request only published articles, non-blocked users, what you want and need; nothing complicated here, and so drupal backend become a strenght here, coupled to angularjs or Ionic.

    "Application size and complexity"

    This is true for both php & js, there is some things to learn but Drupal was not an example until there : he was juste loading ALL the php files (except include attributes in hook_menu), and have until D8 a awfull, awfull autoloading system (registry in database causing some very bad troubles and white screen of the death)

    "however, this means that you are now relying on the browser for the rendering of your application. This is bad. Browsers change and so does Javascript."

    I really don't get this one ! :) So is the html : we all are relying on browser to render drupal front-end html, and html is changing, and that's great, so what's the point here ?

    I think one of the big point your are missing is that html-only front-end with some jquery is not sufficient anymore to build User interfaces today : user are expecting more and more all parts of the page to update automatically, and that's far more complex / time-consuming to build those kind of interfaces with htm / jquery today than in angular Js or such.

    Web has so much changed, nobody calls it "web 3.0" but that's where we are and html / jquery is no more the more efficient way to achieve this.
    In fact, javascript has so much changed that mots common methods of jQuery are NOT needed anymore.

    Today websites looks more and more native OS applications, that's the whole point, and that's why weed need more and more stuff like angular JS, active, react etc ...

  • Also, security. Being left handling raw field values on the client side means you have to re-invent formatters in JS

  • "Also, security. Being left handling raw field values on the client side means you have to re-invent formatters in JS"

    Or you may still use drupal formatters if you want to (in ouputted json)
    Or you mat use sanitizer library from your client library js, or type "bower install blabla'", nothing to reinvents here, there are plenty of valid and easy solutions, using drupal or not for your goal.