When starting out on a major research project in science or engineering, the first thing to do is to go through “the literature” (which usually means the peer-reviewed body of articles and published books, although internet stuff is becoming increasingly important) and try to figure out the current “state of the science” (we used to say “state of the art” but people are less inclined to use that expression than they used to be). From here we proceed to do new things which will hopefully advance the state of whatever field of endeavour we are operating in. As I stated in my master’s thesis:
In any investigation such as this the ideal goal is to come up with something truly novel, and many of such works emphasize their novelty to the denigration of those who have gone on before. While in some fields of endeavour this might be appropriate, in this case such sweeping novelty cannot be claimed. This work fits the mould as outlined by Pascal above: it takes the work that has been done before, advances it a step while realizing that there are many more steps before “perfection” is achieved.
But stepping back to those who have “gone before”, the scientific and engineering literature isn’t as transparent as one would like. In recent years fraud and misrepresentation of results has required any researcher to be careful as to what he or she believes. There are also situations where stuff that looks really good at one point in time get abandoned later for various reasons; we have to make sure our research takes a long sweep in time as well as topic. We also have the problem of “non-novel” papers, which are really rehashes of stuff figured out a long time ago but put back into the literature to give glory to someone else. These don’t do much for the originality reputation of their writers but, sometimes, can be useful, putting back into currency things which have “gotten lost in the shuffle” over the years.
But one serious problem that deserves some attention–and one that doesn’t get a lot of press–is the matter of “going dark” in the literature. An overview of the pattern of scientific and engineering advance is in order.
Generally speaking, in any given field there are “seminal papers” (usually more than one), which is where the field was kicked off. From there we have what comes after, which usually refers back to the seminal paper. In my field of pile dynamics, we have one paper that gets cited in just about everything written on the subject. From here the science and technology are developed and things advance. And then, without much fanfare, the literature “goes dark”.
That doesn’t mean that people stop publishing anything on a given topic. Far from it; however, it’s like a line from Hogan’s Heroes, when Gestapo Col. Hochtstetter tells Klink that he can make Hogan talk. Klink’s reply was, “You can make him talk. He just doesn’t say anything”. A lot of the literature is little better than fluff or promotion of a new idea without substantive detail on how these “new” improvements really work. The obvious question is why.
One reason is that the material is classified for military or national security purposes. Generally speaking, however, that literature doesn’t go dark as much as it’s dark to start with and it’s only later when things come to light.
Another reason is that the field has become inactive, usually temporarily. There are a number of reasons this happens. In my field, wave propagation in driven piles was discovered in the early 1930’s in Australia, and the English carried out some research later in the decade. (The Americans got into a food fight on the subject). But things went dark for a very big reason: World War II, which focused the participants on other matters, such as rational soil classifications and nuclear weapons. After that conflagration, things resumed and progressed to the current state.
In my experience, however, the biggest reason the technical literature goes dark is because of commercialisation. In the early stages, the research is the “property” of academic institutions, individuals and the government (especially since World War II) which funds it. In these conditions there is a relatively free exchange of ideas and expression of these ideas in articles and books. However, when technologies are commercialised (especially when it’s done by a relatively small number of organisations) things start getting proprietary, and then things start getting secret. Although it’s possible to have patents and copyrights to protect oneself in some cases, it’s not possible to copyright an idea; it’s easier to simply use trade secrets, even if those trade secrets are derivative from research from more “open” sources.
The fact that a technology can be commercialised is a good thing in that it shows that it works and is useful. Over time, however, it happens that organisations use institutional inertia and human habit–to say nothing of our tort system, which stifles innovation by punishing experimentation which can go wrong–to make their proprietary method a “standard” and keep its true nature under wraps to discourage its replacement or even improvement. In time this slows the advance of science and technology in ways that are not obvious to most people.
Researchers who set out to try to advance methods in areas which have “gone dark”–assuming they can get funding for their work in the first place–face a number of obstacles. First they must realise that beyond the dark literature are doubtless some improvements the nature of which are obscure. They may find themselves “reinventing the wheel” in an unavoidable way. If they get past that, they find that they lack the benefit of the learning curve which those who actually use the existing method. The road to advancement can be a perilous one under these circumstances.
But advancement is what science and engineering is supposed to be about, isn’t it?
vulcanhammer.net 