On importance of simple, basic sanity testing and how Cypress fixtures can help

Testing an existing app can be very boring. We're all trying to get humans automated out of it and let the great tools and framework do the heavy lifting for us instead. Things like Cypress and Protractor do wonders in that respect.

Testing an existing app due to some sort of infrastructure change is even more boring - you're forced to verify your app just because something in the technology stack or delivery pipeline's changing. So when a change like that happens and the app under change does not have a battery of tests in place, what do you do?

I'd suggest doing a sanity ("smoke") test by doing a very simple URL fetch test of the top N pages of your site, looking to get 200 response codes. Depending on your website's traffic that "N" can be whatever makes the most sense. In one case I took top 900 pages as they had over a 1000 page views in the last month. In another I took pages with more than a 100 page views as the site's traffic was a lot less. Either way, I'm suggesting to get some data from your analytics engine and pick a list of most visited pages from there. In case of Google Analytics, looking at the Behaviour > Site Content > All Pages, adjusting "show rows" to show as many as needed and than Export > CSV (at the very top of that screen). That should give you a list of pages with their relative URLs.

Using Cypress.io fixtures to sanity test your site

Once we have the list of URLs that are popular, we can create a test fixture in Cypress where the URLs would be used to feed a single test.

To get started with this, take the CSV you got from analytics and leave only the URL column (call it url). Then, head out to something like freeformatter.com and convert your input CSV to JSON. You should get a JSON that looks like more or less like:

[
    {"url": "/page/1"},
    {"url": "/page/2"},
    {"url": "/page/another/a"},
    ...
]

I call this the "sitemap test" so the fixture for Cypress should be called cypress/fixtures/sitemap.json.

The actual Cypress test is cypress/integration/sitemap.js and it looks like:

/**
 * to test the general structure of the site and verify that the core set of 
 * pages works without a glitch - 200s, no 4xx or 5xx responses on all!
 */
context('General Sitemap sanity tests (ensure main pages / most popular pages are working)', () => {
    beforeEach(() => {})

    it('should open pages with a 2xx response and never show the 5xx error message', () => {
        // load the fixture with the URLs we're interested in
        cy.fixture('sitemap.json').then(sitemap => {
            for(let i in sitemap) {
                // do a Cypress HTTP Request
                cy.request({
                    url: sitemap[i].url,
                    followRedirect: false,
                    failOnStatusCode: false // we want to let the test run through the list of URLs and not stop
                  })
                    .then((resp) => {
                      // we could verify it's 200 
                      // expect(resp.status).to.eq(200)
                    })

            }
        });
    })
  })

At first I thought it may be useful to always check for getting 2xx response codes but then due to differences between QA/PROD it seemed appropriate to also just run through the list of tests and if occasional 404 happens simply ignore it. So perhaps the expect should be most useful to check for non-5xx. I'm still undecided about that last bit.

Either way, running through this large list of URLs gives a nice, fuzzy feeling of going through a basic sanity test for a big portion of the app and noticing quickly if something's not right due to the latest deployment push.