As Chris Crawford instructs: interactivity is ongoing conversation between at least two actors. In a semi-metaphorical breakdown he uses speaking, thinking, and responding as three basic steps that are exchanged in a dynamic, continuing way. In more concrete terms these seem to stand for Actor A's output; input to Actor B who processes which and generates output which in the interface between the actors becomes Actor A's input which is processed and then produces a new output and the cycle repeats itself. In the current sense this involves digital and analog sensors and processors, which can either interface with like, or, with an organic system— such as a vertebrate's CNS.
Playtime with my cat or calling a friend on the phone or playing a computer at chess is interactivity with wildly varying degrees of physical engagement. As Crawford remarks, a tree branch falling and your reaction to it are not interactive— unless you lived in the Wizard of Oz universe where you could have a retaliatory apple throwing fight. It's a simple and good point, however any interactivity experience with an actual tree would be on a timescale that Crawford might find unsuitable if one goes by his reference to the first commercial computers which took hours for a perceptible response. Plants have recently broached the scientific conversation as active, aware organisms and are capable of interacting with each other in decent time (e.g. potent interplant and species pheromones or an auditory/vibratory response system to predators like aiphids) - it's more about what other kinds of inputs we can detect; and in turn use to build corresponding sensor interfaces. To speak to Crawford's other point, it's true that a traffic-schematic children's rug, or a fridge door light sensor is not interactive, since it's a monosyllabic exchange which ends and never changes and is only ever initiated by a human actor.
The physical aspect of interactivity is vital when considering the mind and body as a cohesive entity— which Western culture, crippled and maladapted to reality as it is— often ignores. The degrees of interactivity Crawford and Victor each outline essentially relate to the input sensor resolution, and the depth, breadth, and dynamism of the processing, as it is adapted to and exemplified by the natural evolution of the animal body (humans) as a basis for the logic and processes of interactivity. Interactive technology is perhaps best seen as glove that fits the mind and body as a unified hand. A human tongue has some millions of tastebuds - a complex and dynamic set of sensors. Hands, nose, eyes, and other nerve termini are sensors/output receptors. To create the ideal interactive physical experience you might create a system, or an artificial "actor" with a higher resolution of sensors to keep up with the conversation with the entire, complete organic/animal/human actor - a complete organism as it were.
If we look at a physical interactivity-continuum of the possible, Bret Victor seems to say that the plethora of screens we pour ourselves into is basically the antithesis of physical interactivity, and physical interactivity is it - as our bodies are an extension of our brains, and there is much more left to imagine and actuate. Bret Victor's 'Rant' on the current fad of screen swiping as the main mode for interactivity is spot on, as well as disturbing (e.g. "finger-blindness")— especially relevant as the heavily brand-incorporated AppleWATCH presentation video was unveiled yesterday. On the one hand, it may pull the consumer populace farther from physical interactivity with their world and into a myopia of wristbound tiny screens - creating widespread stilted, narrow skeletal and social positioning. On the other hand, as it is motion-oriented and equipped with various inbuilt sensors— it seems like a rich tool for developers, makers, and everyday consumers to build upon in expanding the interactive sphere.
My understanding from life, my narrow sample core of human experience, knowledge and thought before me, and as recently as supplemented by Crawford and Victor's writings, tell me that physical interactivity has the potential to bring body and mind together, and to bring this body-mind-entity in closer contact with the reality of the planet and human and animal systems around us - a compensation for recent millenia of 'progress' which has brought us so far to start from the beginning again. Physical interactivity could be viewed as a rejection or workaround or heavy supplementation of the mostly verbal interface which acts as a barrier between us and the truth of concrete reality, consciousness, mindfulness, balance, and pure experience (Attribution: science fiction, ITP atmosphere, the zeitgeist, yoga, my brain, Crawford, Victor, et al.). Good physical interactivity, given our body is a collective of physical termini, can help us learn (play is learning), communicate, community build, explore, express, empathize, and problem-solve because it feeds our brain-body with rich dynamic information and in turn receives as much as it can handle and process and act on from the brain-body. Let us be in touch with the planar surface of perceptonium which we share with all self-aware matter.