Cucumber lets you write automated tests in plain language

In my misspent youth I've tried to introduce Cucumber/Gherkin a couple of different times but it's something with a deceptive promise.

The preconditions are that you have domain experts or business people interested and willing to engage in writing or reviewing these tests. Unless you have this and it's something that those people are going to sustain when the going gets tough you're just making writing tests harder for no real benefit.

It sounds great on the label, but what you’ll end up with are single-use matchers for almost every line of your cukes, annoyance maintaining the regexes that map your cukes to the actual test implementation, and the “non-technical” people won’t be maintaining the cukes anyway. Sure, let them specify the cukes if they want, but then translate them into regular unit tests or specs and forget maintaining them.

Just like all the other no-code inventions out there, they fail to reckon with the fact that essential complexity isn’t a problem with programming language syntax.

I posted this comment in 2013 [1] and I stand by it 12 years later:

Cucumber tries to solve the problem of turning customer requirements into 'real code'. In exchange for that worthwhile benefit, it asks you to implement the most terrible, reg-ex based spaghetti code imaginable.

The problem is that it doesn't solve the original problem AT ALL. And then you are left with terrible reg-ex driven spaghetti code. Like the Jamie Zawinski saying, "now you have two problems".

The lesson here is that software development processes have to pass the 'human nature' test.

The software industry has largely abandoned waterfall development because it just doesn't work well in practice. It doesn't work because people don't know perfectly what they want before they build it. Agile processes usually are much more efficient because they are more closely aligned to how humans solve problems in the real world.

Cucumber suffers from the same issue of being disconnected with reality. In theory, you can find a customer who can give you perfectly written use cases and you can cut-and-paste those into your cukes. In practice, that never, ever works. So let's all stop wasting our time pretending it was a good idea now that it has been shown to not work.

[1] https://news.ycombinator.com/item?id=6411787#6412391

Good old cucumber. I remember a discussion over coffee with someone who swore by it back in the day (~2010). They forced customers to sit down and write these things together side by side. Kind of cringe in retrospect. Wonder what happened to them.

Business people don't really care about this stuff. Over the years I realize more and more that we engineers are naive in thinking that business side is concerned with edge cases and race conditions.

Just to give an analogy software people might get better, if you come to a lawyer because, say, you want to buy a house, you are not going to sit down with them and say "given I want to buy a house, when the seller hides water damage costing over $2000, I get to walk away from the deal". You just hope the lawyer is good and will protect you from various edge cases. You have a lot more to deal with than just closing paperwork. You probably are thinking about renovating, moving, getting inspection, etc.

Businesses are just like that with engineering. They don't want to sit down and meticulously analyze every possible edge case. They have other things to do. Especially when stakes are not that high. Most of these errors can probably be resolved with a phone call and a database edit.

I think this is probably for the best. A good engineer will make sure you're standing on a solid ground, and ask the right questions at the right time. They wouldn't need this amount of hand holding. Leave business time to focus on making deals, connections, organizing the whole operation to move forward, etc. Let them give you vague requirements, and crystallize them yourself. It's way better than a micro-managing business that thinks they know exactly how everything should be.

P.S. Also, I'm not sure why everyone is so hung up on regex = bad, it's not like switching to an AST-based language would've made anything better here. Regex is fine imo, just the entire concept isn't.

I wish the doctest approach won over the BDD approach. I still find myself recreating the doctest approach in miniature...

Specifically I try to make a nice string representation of state, and then tests match off that state.

So for instance I don't want to say expect(x.length).toBe(0) (one syntax, Cucumber goes further in that syntactic direction). Because if the length is 1 then 1!=0 is a very opaque failure. Or if you do expect(x[0].attr).toBe("y") then you've tested one thing but you have to have another test for list length, other attributes, etc. Often you'll leave the others out, expecting them to be ok, but sometimes they aren't...

Investing in a good, stable, matchable string representation will give you a great tool for any number of tests, and it's not just a great tool to verify but also to debug your tests.

I've always thought this looks cool but then you realize the whole thing is driven by regexs on the "plain english" and I recoil in horror

    Given /^a nice new bike$/ do
      expect(bike).to be_shiny
    end

have never actually tried it for that reason - it just seems worse in every way

This tool is awful. I hated it a decade ago and I still hate its successors in other languages.

It creates unnecessary work for something that's never to be seen by real stakeholders.

There is a new trend these days using LLM that is similar to Cucumber: Spec Driven Development using AI. You'd be left disappointed again.

Engineers should refuse to write Gherkin. It's not meant to be for them, it's meant to be for non-engineers to tell engineers (or the systems the engineers make) what they expect will happen. It looks like some flavour of user stories for this reason.

Engineers' responsibility begins at the "code behind" the gherkin layer. Someone needs to enforce this.

Sadly, IMO because gherkin looks like structured language (i.e "code"), and because that's what engineers do, they end up doing it. Either they tricked themselves into thinking they should, or the product team tacitly assumed it was a code thing.

If you're an engineer and you're writing gherkin, you should consider this a "code smell" and consider ditching it. Go for a spec tool one layer closer to your main domain. i.e. just write normal unit & integration tests.

I'm not saying "don't do BDD" or "don't do acceptance tests". You should do that! I'm saying "gherkin should allow product owners/managers to take ownership of reporting on how well the app meets their expectations". But most typically, they don't get it, or they don't want to do it.

I recently worked somewhere where we had dedicated devs-in-test. They wrote most of the e2e tests for the application. They wrote it in Gherkin, which meant that they _also_ wrote the JS/Selenium tests behind them. So they were basically writing two sets of scripts for each test. At some point they explored using a 3rd party cross-device testing platform (lambda test, maybe?) who had their own in-house test script that looked like Gherkin, so it needed a JS/TS code-behind to actually run the tests. So they did it all again!! Such a waste.

> Cucumber lets you write automated tests in plain language

No, it doesn't.

Or to be more precise: Cucumber lets you write your tests twice - once in something that isn't nearly as close "plain language" as it seems at first glance, and then again in a horrible mismash of regex spaghetti code that will make you wish you'd been born a thousand years before computers were invented.

I was forced to try to use it about 15 years ago and spent literally twice as much time debugging the testing framework as I did actually implementing code - velocity on actual feature development ground to a halt. I think the most hilarious part about this was that it was completely incapable of finding the most common class of problem we tended to run into at the time - cross-browser presentation issues.

One of the big problems you'll find if you try to use this trash is that nobody except a software engineer will be really able to write tests in a consistent enough way that you'll be able to implement them without a truly horrible regex soup: even if you can actually get a non-software-engineer to write cucumber tests for you (dubious), the non-engineer-types will write "given I'm logged in" for one test and then "given I have logged in" for another, and "given I gave gone through the login process" for a third. And it doesn't matter how many regexs you implement to try to cover all their variations in their language, or how clever your regexs are - the nature of natural language means there's near-infinite variations, and they'll come up with new and more annoying variations just when you start feeling confident. And that's before we even start talking about the ambiguity of natural language.

I did manage to find a decent use for cucumber-style language a couple of years ago, though - we started commenting our unit tests using this style:

  def test_something():
     """ test some thing """

     # given some prerequisite
     set_up_prerequisite()

     # when I perform some action
     perform_some_action()

     # then some condition should be true, because <reason>
     assert some_condition is True

I am very much for internal DSLs for functional testing, but have never seen the point of external DSLs (it feels wrong that anyone but the implementing team could or should create or maintain them). I passionately believe you should refactor and use abstractions in your test code thought, and in my experience that ends up looking quite like parameterised steps (again, in code rather than English preferably).

I've commented elsewhere on lack of buy-in from stakeholders as a huge pitfall.

On a technical level, cucumber is also at odds with the need for a test suite to be easily maintainable. What I mean is that each test (especially e2e tests) will want to do some setup/initialization. This is usually expressed as step definitions.

Over time, an undisciplined team may write several slightly different but effectively identical step definitions. They may also combine multiple steps into bigger steps because usually a spec writer doesn't want to exhaustively define every piece of setup, they just want to write "Given all 200 pieces of input data and mocks are magically in the right place..."

I was able to wrangle the specs into composability using Rule and Background blocks, but at that point we were just programming tests with a shitty layer over the actual code.

This sparks a thought experiment I've been having. In this world where llms can be thought of as the new layer of compilers, things like pickle are likely going to be the main unit of work for humans.

Only now instead of this developing brittle generated tests, it will instead be used by the llm as guidance to generate the actual code and tests.

Before people jump down my throat, I know we are nowhere near that today and I promise I'm not pitching this to my leadership because they would gobble it up too fast.

But for us engineers, I think there is an interesting space for thinking of llms as akin to garbage collection, a feature that allows us to abstract to a slightly higher level of thought. Yes we still need to know how to check under the hood, but this is looking like the right level of precision-flexibility ratio that llms thrive in

IME this allows the BAs who are writing ticket requirements to also write tests. It works. The dev may have to tweak the tests to get them to compile, but it an org that is willing to spend quite a bit of money on testing (e.g. public sector) this is a format that devs and BAs can read.

There's a reason why I call BDD a Billion Dollar Disaster. I've not yet seen a Cucumber or equivalent suite not permanently broken on a mature codebase. Typically a dedicated testing team will fight tooth and nail that this is what the customer needs, even though nobody from the business side has ever looked or wants to look at the test definitions or even the output. Then of course it can't match changes to app behaviour and some poor sod has to go in periodically to fix it up after the fact.

BDD in whatever implemenation is a gigantic waste of everyone's time.

You can easily build your own clear, simple DSL with Minitest, Pytest, or JUnit - no overthinking needed. Engineers can whip up more readable and reliable tests quickly. Since only engineers are writing tests, there's no need for unnecessary complications.

Aren’t syntaxes and semantics purposeful? Isn’t knowing the proper syntax and semantic structure part of how we design good software? I don’t understand the desire for “plain language” beyond the notion that ignorant people want to stay ignorant.

Personal opinion: I absolutely hate these "Given" "When" "Then" type tests. I have to learn a whole new language to write tests, I don't get on with them at all.

I work with engineers in Sweden, Poland, mainland China, the Philippines, and half the US team is Indian.

Even the US-born engineers cant always agree on how to communicate product ideas!

I'm a little bit astonished by the comments mentioning regex. I used Gerkins at work and there was no regex at all.

Once you write it, you can use the sentence for building other tests if you design it for being flexible and variable, i.e. design well enough your code.

Given I'm connected as ___ with password ___.

When I click on the element __.

Then the element ___ is present.

Then the element ___ is present.

[...]

For critical use case, it was enough. For more that's a bad idea: it take way too much maintenance cost.

I thought we might have killed the crap cucumber was trying a decade ago. It's horrible. You wind up writing more matching code to turn that plain text into a test flow than you wouls of you just wrote the darn test

I love cucumber but I never follow the pattern to (1) write the feature file, (2) generate partially implemented tests with said file, (3) finish implementing the tests + subject under test.

It's better to (1) write the test, (2) then later once all that's done, extract documentation in a human-friendly format.

As a software developer, I have enjoyed occasionally writing Cucumber/Gherkin. However, the roles that were _supposed_ to be writing it did _not_ enjoy it, either refusing to write it or writing unusably-brief and -malformed versions even after extensive training.

Interesting this is being brought up, I used to use this back in 2013 when developing Rails apps, I've been thinking about what a good "standard" spec language would look like in a post-LLM world. I wonder if the cucumber syntax could help here?

Deja vu.

Cucumber was all the rage in 2008, IIRC..

I think the promise of cucumber/gherkin is that one cannot write overly complex tests with them.

I’ve seen (somewhat) experienced engineers get lost in complex pytest environments that used a lot of mock objects. Over time, they lost clarity on what was being tested, how, and why.

The real value is to have an automated set of simple smoke tests.

Frankly, Cucumber serves a nonnegotiable reminder to me that the automated tests should be simple and straightforward.

For anything extremely complex, I’ll write unit tests instead of trying to craft ridiculous gherkin steps only used for a specific scenario. None of this “when it is midnight of a blood moon in a leap year and the second request attempt has timed out…”

Extra layers of busywork that nobody else wants to look at.

If your job is polishing a turd, make the turd shine like a cucumber. I have seen countless hours wasted on making Cuke/Gherkin "pretty" instead of accomplishing anything for the business.

Nothing made me more enraged than trying to express test steps via regex. I will never go down that road again.

I’ve worked with cucumber for a few yeats and only ever seen developers writing Gherkin.

To me it’s just another framework to learn - a failed abstraction. It’s always introduced by idealistic devs who regularly jerks off to conference talks.

Cobol reborn.

Fuck Cucumber.

Buried in the mess there are a couple of good concepts like it moves away for the 'busy-work' of atomic tests.

But shit like "living documentation" makes me want to vomit.

Are people dying quicker than we can teach this is bullshit? Or are people dying stupid?

The idea non-coders might ever be involved with Cucumber is also fucking retarded.

Last I used it, it was a mess, maybe it's better now. Part of that was those arrogant Selenium motherfuckers.

All this shit is why LLMs are talking over. Fucking retards with their stupid as fuck theory's about auto documentation and Selenium refusing to implement X (i forgot) on some BS 'principal'.

LLMs just get the job done, they might fuck shit up, but they are not arrogant autists about it, and LLMs don't care when they are called useless arrogant autists and double down.