Emergent Development Principles
Top-down software engineering isn't always possible. The problem-space might not have been well explored, the platform may be new or volatile, or some development factor might not be knowable beforehand, such as: what designs will be "innovative" or "fun". Often a developer just won't know exactly how something should be best implemented until after diving in and hammering out a quick and dirty hack.
It's common that prior to a prototype becoming functional more elegant solutions become apparent. Since the way forward may have been obscured initially one's first working implementation may be poorly documented or in need of refactoring or optimizing. Management might also insist the kludge ridden hack is now good enough to ship, which causes the codebase to gather entropy in the form of ugly unmaintainable source code.
Below are a few emergent design methods I use to make more elegant software. While I use absolutisms herein for expediency, only the reckless should ignore exceptions.
Always Code it Twice
If you do it more than once you'll be more satisfied. Rather than attempt an implementation with the immediate goal of production ready software, plan to write code that you'll throw away and rewrite. Factor in enough time for a quick and dirty disposable implementation. If you know the code you're writing will be trashed you won't have to feel guilty for not properly documenting it or adhering to strict style guides. Disposable code can be [re]written faster, allowing the problem-space to be searched quickly. The primary benefit is reduced cost of failed experimental approaches.
Planning to rewrite should cost less time doing so. Exploratory approaches tend to be highly volatile; consisting of batches of code being scrapped and redone prior to completing a working unit. A dedicated rewrite pass means you spend less time polishing scrap since you'll only beautify code that's being kept.
Once a few hastily coded units are functional think on how they might be improved. Perhaps better names for interfaces will be obvious once completed. Should the functionality be split up, merged, or refactored? Maybe a familiar design pattern will have emerged without having set out to utilize it. A more performant algorithm might have been used in hindsight. In any case, the solution can be better implemented with a clearer goal in mind from the outset. Resist the urge to make such improvements until after something is working so that the whole solution can be polished at once.
When rewriting the improved code stub out all functions and document the API before doing the rest of the reimplementation. During the second pass it's better to do the things first that one usually winds up not having time to do upon completion. Having climbed past the dreaded tedium a race to the finish is downhill.
Alien Prototypes are Best
If possible, use a language that's incompatible with the production system to create the first-pass "reference" implementation. The end result may be too slow for the demands of one's production environment or missing needed bindings to other areas of the codebase. If not of an alien language then make the prototype sufficiently weird. The goal is to make it simply impossible to ship the kludgey first generation prototype code so that you're more likely to reimplement it. Additionally, the different language or strange approach might add a bit of valuable chaos...
Caveat: Beware that Management may suggest adding the alien language to the production environment; Make sure that it will take longer to do so than to rewrite the prototype. For this reason: Never prototype in a language you hate.
Try to Fail
One of the benefits of a multi-pass development approach is that it's OK to fail. However, the first attempt at a solution may still be daunting. One may find themselves bogged down by indecision over which approach to take first. Even the most skilled developer can be embarrassed by a failed attempt. One could simply be stumped and not see any decent way forward. If nothing seems promising don't make the promise; Actively break it by trying to fail.
Obviously, most deliberate attempts at failure will fail. One can then honestly declare, "I meant to do that", or "I didn't expect it to work, but it would have been cool if it did." Sometimes the failure will actually succeed or otherwise illuminate unforeseen possibilities. At the very least one will have gotten over the initial hurdle of a first attempt. The only way to go from failure is towards success.
On the other hand, some devs can make just about anything work. It's imperative that those of exceeding resourcefulness recognize when a solution is trying too hard to succeed. It may take some experience to learn when to pull out of a nosedive and deem the approach a failure. A good way to gain that experience is to try multiple approaches at once and thus create the possibility of comparison.
Iteration is Annealing
Iteration should not merely be incremental steps towards a goal. A two stage approach includes cleaning up prior quick and dirty code then fast stab at new functionality. Each iteration cycle is thus at least one step backwards followed by a leap forward.
In AI there's a process called annealing. It's a method by which a solver converges to a solution over several iterative steps. Annealing is quite similar to the iterating of software design and implementation. Realizing the similarity allows one to apply the insights gained in artificial intelligence research to the dev toolbox. Note that cybernetics also studies the flow of information in organizations and organisms not just machines.
Viewing iterative development as an annealing process allows us to shift our goal from completing an implementation bit by bit into mapping the geometry of a problem-space. Back-propagation and other annealing processes move parts of the solver's mind towards local minimums nearest to the desired result. Iterative development often follows a similar approach. However, we know from AI research that the locally obvious solution converged upon is frequently not the optimal solution. In software development we also find an acceptable solution may be right next to a much better solution just one sprint away. However, re-iteration towards the same nearest solution won't reveal new solutions.
Annealing is also a material science term for heating something such as metal or glass and cooling it slowly to alleviate internal stresses and strengthen it. Iteration should have similar effects on a codebase. Letting an area of code cool down properly allows bugs to be hammered out and more ideas for improvements to accumulate for simultaneous consideration.
Chaos is Key
When one begins separate runs of an iterative solver from similar initial conditions, such as our assumptions, the iterative annealing process will likely converge to the same solution. Rarely will the solver discover a new, potentially better, solution when starting at a point along the path a previous process took. We must find a different path. Thus, when trying to discover a better mousetrap one should cast off as many initial assumptions as possible about what a mousetrap is. This can be harder than it sounds since minds tend to gravitate towards knowns and away from unknowns. Humans tend to only try a few new things even when looking for a radically better solution.
It's obvious that different approaches can yield different solutions, but one may be surprised our resistance to try new things. Software devs especially tend to assume that trying to re-solve a solved problem is a waste of time. The dogmatic practice of reusing existing solutions can be our worst enemy when trying to make new discoveries. Don't be afraid to reinvent the wheel. Sports cars don't have wagon wheels. There is always room for improvement.
Originality is rare. Assumptions are not easily replaced. One surefire method to do so is by introducing chaos to the initial conditions while selecting a solution to attempt. If you really want your R&D processors to think outside their boxes, bring to experienced colleagues the advice you glean from dice, the I Ching, and etc. (pseudo)random oracles. Most times random suggestions are absurd but sometimes they're just crazy enough to work.
Don't Look Before You Leap
We're standing on so many giants' shoulders our heads are in the clouds and it's terrifying to hop down to earth. Utilizing only ancient design decisions can limit our options. In our modern era it's so quick and easy to perform a search and discover what others have done before us when confronted by a problem. Finding the common consensus is the fastest way to develop an incurable bias. A fresh and untainted mind is incredibly valuable, for it is much harder to unlearn than to learn. One who does not yet know how to solve a problem is more willing to entertain and even seriously attempt new ideas that others would deem absurd. Our greatest modern achievements were deemed absurd or impractical at some prior point, often by the common consensus of experts. The wise learn from mistakes of others, so progress depends on fools.
Rather than being a fool it's better to just act like one. Give some serious consideration to a few ideas that one would otherwise quickly dismiss. Don't foolishly attempt the impossible while running full tilt at an unknown windmill; Pretend to do so for a bit then stop and take inventory of the new situation. An intelligent mind can entertain an idea without adopting it, but a genius can implement those orphaned ideas and let them stand on their own merit.
Unknowns can be resources. When one does not know the solution to a common problem, but is aware that one exists, try to reason out a solution independently before looking to others. Worst case scenario: You come to appreciate existing methods more. If you reinvent a wheel then you've proved yourself as smart as the first person who did so. Best case scenario: You find a new solution possibly with different and better trade-offs than known methods. It's surprising how often a genius earned such a label while thinking they were rediscovering what everyone else already knew. Take Richard Feynman, for an example.
Adapt the Design to Fit the Features
In game development it's natural to get engrossed in a grand idea, building upon it specific details until it becomes one's beloved brainchild long before a prototype is ever seen. Business software also suffers from "Pet Project Syndrome". Overinvestment in an idea can be severely detrimental. No one wants to hear their brain-baby is ugly, or that it must be butchered for the project to survive. Refusing to change any design specifications is a recipe for disaster.
It doesn't matter how cool the game idea sounds, it must be playtested to find out if it will be enjoyable. Fun is an incredibly elusive prey. Innovation is likewise difficult to dream into being. What often works where the top-down approach fails is to prototype many small features, then use them as building blocks to assemble many designs for evaluation. One can not invent fun or innovation, they can only be uncovered through discovery or found by chance.
Great designers are like chemists. Rarely will they design a new element, but they frequently react various compositions of known elements. The combinations are filtered by experience then fired in the crucible of testing. If their prediction doesn't quite match the results they update the design of their experiments. Both distill as much value from accidental discoveries as they do from deliberate successes.