History

In the early days (the 1960s), guys like Rosenblatt and Widrow built fascinating, linear and mostly single-layer networks using lots and lots of transistors. The development took an embarrassing halt shortly afterwards after proof that these types of networks were fun, but rather useless. It was not until the 1980s that they became popular again, due to a paper Rumelhart published in the very influential book Learning Internal Representations by Error Propagation, in Parallel Distributed Processing: Explorations in the Microstructure of Cognition5.

This paper brought attention to things that made artificial neural networks surpass their believed limitations by introducing differentiable non-linearities6 and multi-layer networks. This had really been discovered already by some guys in the 1970s. Nobody really remembers these guys, though - for instance Werbos (1974), but he did it in his dissertation, and no-one really reads dissertations.