How long have you been a software engineer?

I've found that going straight into hacking on something can end up biting me in the ass for big projects. I've also worked where we would do everything up front in a design document all the way down to functions, what args they take, and pseudocode for everything. The code would then take a day or two with nothing to redo.

A sufficiently detailed design document is essentially indistinguishable from code. How do you avoid the problems you encounter when writing code while writing the design document?

In my experience it is not one or the other. You can experiment first and then document, or vice versa. The point of documentation is "writing is thinking". The act of writing forces you to think through your ideas in ways that other modes of thinking do not. This is especially important when you are working on a team and need to share your ideas.

I agree that sufficiently detailed design documents are indispensable from code and should be avoided. However, clearly documenting the why, what and how of your work helps align everyone clearly. It also helps an individual clarify their ideas to themselves. It is hard to write clearly & succinctly. It takes time & practice, but I find it essential.

You know your programming language very very well, AND design for failure. What I do normally is first write the API of the library I want to use. Then I might start detailing things, which becomes some early documentation/API. Finally I try to implement it in code, and if I find a truly blocking code bit I restructure my assumptions about what is possible with my API and go back to the API/docs.

For example, at some point I knew I wanted to set localStorage and cookies values just like this:

    cookie.token = '1234';
    local.token = '1234';

I know enough Javascript, and the language is flexible enough, that I know that I could achieve that, either with getter/setters or with the more flexible Proxy(). So I continued writing the other methods first; how would it look ideally to "read" a value? To "delete" a value? Etc:

    console.log(cookie.token);
    delete cookie.token;
    // ...

I wrote few examples of each, put it all in some documentation, and then wrote some tests, following those documentation examples:

    expect(cookie.token).toBe(null);
    cookie.token = '1234';
    expect(cookie.token).toBe('1234');

And finally wrote the code for it. Later I wanted to add IndexedDB, BUT! That is async! My abstraction was broken, or was it? I could just modify a single method and everything else would work as expected:

    db.token = '1234';
    console.log(await db.token);  // <= added await here
    delete db.token;

I have the same reaction to this as I would to "a sufficiently detailed blueprint is essentially indistinguishable from a building". I'd be very interested to see a software design document so opinionated and prescriptive about implementation that it leaves no room for a software engineer to add value.

I think it was an extreme example but we didn't. The design phase is exactly when you should be formulating ideas and throwing away things that don't work. The theory is that this is easier to do before code is laid down. They were way too granular though so it was super painful even in design phase to get these design docs together.

I think the design phase in this instance required far too much detail. Some middle ground would probably be best imo.

You can’t necessarily, unless you write it at a higher level of abstraction* and trust the engineers are correct when they say it’s feasible to implement in a reasonable amount of time. Now your team is aligned and coordinated, even if the team consists of one person.

* Shape Up - https://basecamp.com/shapeup

> A sufficiently detailed design document is essentially indistinguishable from code.

Stealing this one. What a great quote!

I don't think it's about experience. It's just different approaches geared to the individual. I really dislike it when people assume that anything other than their way of working is down to experience.

I've been doing this for 10 years and I've tried the design doc first route and it's arduous for me. I have coworkers who absolutely love that route.

I generally just do it in parallel. Coding to me is like sketching. So I get down my ideas in code, and I document the design as I go. I switch between the two as each one informs the other.

Not the guy you asked, but personally I've been one long enough to know that iterative design/implementation often produces a better design than waterfall. In waterfall, you have to think of everything at the beginning, and you aren't smart enough. Nobody is.

In iterative, sure, you have a design (often informal and sketchy), and you try implementing it, and as you do you find out why certain aspects were a bad idea, and you change the design to fix those, and you find more things that were bad ideas, and you fix those, and so on. By the time you're done, you have either a nice clean design or a total pile of garbage, depending on whether you did real redesigns or just kludged some bandaids on.

Writing the documentation up front doesn't prevent iteration. But it means that you're going to have to re-write it every time you change the design. And it means that you're going to have to flesh out the initial design enough to write the documentation for it. This is somewhat wasteful.

I agree. I'm not condoning waterfall though. I only mean that planning has value.

I've had this bite me in the ass on many projects too, but I still agree with the parent. When I get an idea I get a rush of motivation to implement it. If, after that rush is over, all I have is a document, the motivation isn't coming back and the project isn't getting finished. Starting with a mostly-functional MVP means I can show it to people or use it myself, which gives me the extra motivation needed to make it less hacky/more maintainable/perform better/all the other things, that I could've thought about in the planning phase.

(I just went through this cycle in the last month or so actually - I got pissed off at the state of public transport apps after I missed a train, so I decided I should make my own. I made a buggy hacked-together web app, started using it daily and showing it to people. Every time I used it, it made me want to work on it more, so I did - rewrote the backend to cache and segment data properly, made the map run at a non-sideshow framerate, made the icons consistent, started building a native version... It's actually getting close to something I'd call "good" now. Knowing myself, this would never have happened had I started off with a specification or even a mockup)

What kind of systems do you develop and what stack do you use?

Software developers tend to impose their habits on everyone thinking the circumstances do not matter. But well, they do. A lot.

Both the article, you, and the GP are doing this. I've just read the first few comments, but I expect to find a lot more of it there.

Anyway, if the strictness isn't imposed by the problems you are solving, you may gain from pushing it into your tools and getting some experience with less strict procedures too. Just like the GP may gain from trying some more strict procedures for a while.

What exactly am I doing? I'm saying that I've experienced both extremes but I am not endorsing either of them. What works best obviously depends on circumstances.

I completely agree with your point here.

As I wrote in a previous comment for a different post: I'm currently developing an "experimental" piece of software, i.e. I am not even sure some features I want to add are possible, meaning I had to rewrite it a couple of times (and won't be the last). Having strict versioning, pinning, and documenting everything before it reaches a semi-stable state would discourage me from developing it altogether.

Depends on the project too.

Sometimes, writing up some code as a proof of concept is the best way to _inform_ the documentation, a way to gain clarity and understanding of the problem.

In other words, writing software doesn't have to be a linear process (rarely is). Sometimes you start with documentation if the problem is well understood. Sometimes you start with code to derisk the project. Most of the time, you bounce back and forth between the two.

if you can't explain what your code does, you don't know what your code does.

if you can say you know the level of detail you need to write for all the code you're ever going to produce, you're probably not writing very complex code.

It's interesting that you can replace "documentation" with "tests" in your argument and it's still an argument people make. It's double work you have to write tests and code! Yup, it is, kind of.

I'm not necessarily assuming which side of the argument is right -- for docs or tests.

Certainly "starting something new spontaneously just by starting to hack" can be useful and fun. And may not involve tests or docs. That's sometimes called a "spike" (although that's not exactly what the term "spike" was meant for, I think).

I find that generally after an initial period of spontaneous experimentation, getting a sense of what might work or not and what direction I might want to go--it's generally best to move to a less spontaneous development process. That will likely involve test coverage (whether strictly TDD or not). I think thats probably also the point to consider "test-first development" too.

Lots of programs fail adoption because the documentation just isn’t there, or of poor quality, and adoption can seriously be hampered by it.

It all depends on the style of project. If it’s your side project that tries out a lot of things it makes less sense to make docs, then say making a new start up core app or framework.

You're talking about something very different -- basically, building something for fun. The objective is to spend time building something, and it doesn't really matter what the thing being built is. This aligns with the intrinsic motivation very well.

The other scenario is that you're building something for others to use -- you have to understand their problem, design the solution so that it would make their lives easier, etc. With this, it makes sense to start at the soft end of things. The implementation, the actual building -- that's just drudgery at this point, something someone will do if they're paid enough.

So true, when I say in engineering school we had a subject called project management and we were taught that you have to write the documentation and everything first and it should be done in such a detail (including variable names and such) that you can give the project documentation to three different programmers and receive the exact same product from all three.

I always hated this approach because of the reasons you stated.

I love the quote from that German hacker Fefe: "Coding is an explorative process"

> write the documentation and everything first and it should be done in such a detail (including variable names and such) that you can give the project documentation to three different programmers and receive the exact same product from all three.

The only documentation that specific would be … the code!

Programmers seem to think that code is somehow its own documentation. It is not, no matter how readable it is.

Documentation is there to provide structure to the reader (no, the class hierarchy in your code and your build files do not do that) and to eschew the unimportant details. Code has a lot of unimportant details that provide cognitive load when reading it, but do not give insight. Documentation is about transferring insight.

These are very different things.

I don’t think any English language based documentation can achieve this:

> give the project documentation to three different programmers and receive the exact same product from all three.

Nit unless the language used is very code-like and pedantically logical to the nth degree.

That's not what documentation is for. That's not even really achievable by sharing code unless you are sharing a lot of metadata too (exact build setups, compute environments, etc.) or the code is very simple.

Documentation is there to _abstract away_ all of the specification that you don't need. That is very valuable if you are ever going to work with other people - it reduces their cognitive load - and it is valuable if you are working on something complex since it also reduces your cognitive load.

I agree with you, I just don't think this is possible with any kind of documentation in the classic sense:

> give the project documentation to three different programmers and receive the exact same product from all three.

Is like giving a play script to 3 performers, and expecting the same performance.

Software development is a creative art. People will deny it but it is true.

No...

The only documentation that specific would be the code... plus the documentation!

I'm totally with you there. I can't get down with TDD because the purpose and structure of the project come to me when I'm in the zone coding. If I worked like the author said I'd have a big document and then immediately encounter some glaring problem with it at coding time and have to go back to square one.

I think that other people's brains work differently, and more power to them, but we don't all work this way.