Building Big Shiny Apps — A Workflow 2/2

Second part of the blog transcription of the talk I’ve given during the eposter session of the rstudio::conf(2019).

Read the first part here.

This post is at the premise of our book: https://engineering-shiny.org/. You will find all complementary information in it.

Building Big Shiny Apps: step by step

Step 1: Designing

Don’t rush into coding. I know you want to, because it’s what we like to do and what we are good at. But before entering the coding marathon, take time to think about the application and the way it will be deployed and used. Take a pen and a piece of paper and draw the app. Talk about it with the people who will use the app, just to decipher what they actually need. Take a moment to talk with the IT. Here are some questions you can ask:

• “Who are the end users of the app?” — This will help you know if the end users are tech literate or not, and what they aim to achieve with the app.
• “How frequently will they use the app?” — The small details of the design & the UI of an app you use on a daily basis is more crucial than when the app is used once a month.
• “What level of complexity and personalization do the users really need?” — People writing app specifications sometimes want more functionalities than what is actually needed by the users.
• “What level of interactivity do you want, and to what extent is it central?” — People love interactive graphs and like when things automatically sync with each other. Yet these two can make the app slower, without any significant gain. For example, being reactive to a selectInput() or a sliderInput() can lead to too much computation: maybe the user will not succeed to choose the right input the first, second or third time… So let them do their choice, and add a button so that they can validate when they are ready.
• “How important is it that the app is fast?” — Should you spend a lot of time optimizing the little things?
• etc.

Asking questions, taking notes, and drawing the app help you have a good idea of what is expected and what you have to do now.

So, next step!

Step 2: Prototype

I like to go “UI first”. For two main reasons:

• Once the UI is set, there is no “surprise implementation”. Once we agree on what elements there are in the app, there is no sudden “oh the app needs to do that now”.
• A pre-defined UI allows every person involved in the coding to know which part of the app they are working on. In other words, when you start working on the backend, it’s way much easier to work on a piece you can visually identify and integrate in a complete app scenario.

So yes, spend time writing a front-end prototype in lorem ipsum. And good news, we’ve got a tool for you: it’s called {shinipsum}. The main goal of this package is to create random Shiny elements that can be used to draw a UI, without actually do any heavy lifting in the backend.

Hence, once you’ve got a draft of your app on a piece of paper, you can then move to the “ipsum-UI” stage: building the front-end of the app, and filling it with random Shiny elements, with functions like random_ggplot() or random_DT().

Another package that can be used to do that is {fakir}}. This package is designed to create fake data frames, primarily for teaching purpose, but it can be used for inserting data in a shiny prototype.

Step 3: Build

Now the UI and the features are set, time to work on the backend 🎉

This part is pretty standard — everybody can now work on the implementation of the functions that process the app inputs, in their own modules. As the UI, functionalities and modules have been defined in the previous steps, everyone (well, in theory) knows what they have to work on.

And also, as said before, there should be no “surprise implementation”, as the app has been well defined before.

Step 4: Secure

Securing your app means two things: testing, and locking the application environment.

So first, be sure to include tests all along the building process — just like any other R code. As the app is contained in a package, you can use standard testing tools for testing the business logic of your app — as said in the first part, it’s important to split the backend functions and algorithm from the user interface. That means that these backend functions can run outside of the application. And yes, if they can run outside of the app, they can be tested the standard way, using {testthat}.

When it comes to testing the front end, you can try the {shinytest} package from RStudio, if you need to be sure there is no visual regression all along the project development. {shinyloadtest}, on the other hand, tests how an application behaves when one, two, three, twenty, one hundred users connect to the app, and gives you a visual report about the connections and response time of each session.

One other tool I like to use is Katalon Studio. It’s not R related, and can be used with any kind of web app. How it works is quite simple: it opens your browser where the Shiny app runs, and record everything that happens. Once you stop the recording, you can relaunch the app and it will replay all the events it has recorded. And of course, you can specify your own scenario, define your own events, etc. It’s not that straightforward to use, but once you get a good grasp of how it works, it’s a very powerful tool.

Secondly, secure your app means that it can be deployed again any time in the future — in other words, you have to ensure you’ve got a proper handle on the required R version, and of the packages versions which are required to run your app. That means that you have to be aware that upgrading a package might break your app — so, provide an environment that can prevent your app from breaking when a package gets updated. For that, there is of course Docker, but also R specific tools like {packrat}, custom CRAN repositories or package manager.

Step 5: Deploy

Tools for deployment are not the subject of this blog post so I won’t talk about this in details (remember, we are talking about building 😉 ), but our two tools of choice are Docker & ShinyProxy, and RStudio Connect.

Building Big Shiny Apps: an introduction to {golem}

Ok, that’s a lot of things to process. Is there a tool that can help us simplify this workflow? Of course there is, and it’s called {golem}.

It can be found at https://github.com/ThinkR-open/golem

{golem} is an R package that implements an opinionated framework for building production-ready Shiny apps. It all starts with an RStudio project, which contains a predefined setup for building your app. The idea is that with {golem}, you don’t have to focus on the foundation of your app, and can spend your time thinking about what you want to do, not about how to do it. It’s built on top of the working process we’ve developed at ThinkR, and tries to gather in one place the functions and tools we’ve created for building applications designed for production.

When you open a golem project, you’ll start with a dev-history file, which contains a series of functions that will guide you through the whole process of starting, building, and deploying your app. The newly created package contains an app_ui.R and app_server.R waiting to be filled, and a run_app() function that will launch your application. Any new module can be added with golem::add_module(), a function that creates a new file with the required skeleton for a shiny module. As I said, you don’t need to think about the technical things 😉

You can also find a series of UI, server, and prod-related tools, functions for creating deployment scripts, and other cool stuffs. Check the README for more information.

Building Big Shiny Apps: the Tools

Package with {golem}

We believe that Shiny Apps are to be put into a package. Why? Because it allows them to be documented, tested, and can be installed in several environments.

Also, think about your last Shiny app that wasn’t in a package. It’s an app.R, maybe with a folder you’re sourcing and which contains functions (let’s say in a R/ folder). Maybe you’ve written some meta information (let’s call it a DESCRIPTION), and some tests you’ve put in a tests/ folder. Also, as you want to be sure to do the things right, you’ve put documentation in your functions. Do you see where I’m heading? Yes, you’ve written an R package.

Git

Friends don’t let friends work on a coding project without version control.

Shiny modules

Shiny modules are crucial tools when it comes to building big shiny apps: they allow to collaborate, to split the work into pieces, they facilitate testing, and they allow implementation of new features to be made more easily.

Prototyping with {shinipsum} and {fakir}

These two tools allow you to prototype a Shiny App and to go “UI first”. Learn more :

• https://github.com/ThinkR-open/shinipsum
• https://github.com/ThinkR-open/fakir

CI and testing

Testing is central for making your application survive on the long run. The {testthat} package can be used to test the “business logic” side of your app, while the application features can be tested with packages like {shinytest}, or software like Katalon.

Deploy

Docker, Shiny Proxy, RStudio Connect… Don’t just put your app on the server and let it live. Use tools that are specifically designed for deploying and scaling web applications. But this will be the topic of another blog post 😉

Want to know more? Keep an eye on our bookdown: https://engineering-shiny.org/.