On and off over the past decade, I would meet Derek Bailey in the same Chinese restaurant in Dalston, north London. As well as being a wonderful raconteur, the Yorkshire-born guitarist regularly blew holes in convenient wisdoms sitting smugly on some shelf in my head. His provocativeness was not oneupmanship, or a parade of erudition; it was the way his brain was wired. He had done the same for musicians and listeners all over the world for 40 years or more as a free-player and a freethinker, a Frank Zappa for the world of spontaneous performance.

Bailey, who has died aged 75 of complications from motor neurone disease, was a guru without self-importance, a teacher without a rulebook, a guitar-hero without hot licks and a one-man counterculture without ever believing he knew all the answers - or maybe any at all. With his passing, the world has lost an inimitable musician and an implacable enemy of commercialised art.

Bailey once described his friend John Zorn, the American avant-garde composer and improviser, as "a Diaghilev of contemporary music" for his catalytic influence. But he could as easily have been describing himself. He worked with performers as different as free-jazz piano legend Cecil Taylor, cool school saxist Lee Konitz, Harlem bop-and-swing hoofer Will Gaines, naked Japanese improvising dancer Min Tanaka, fusion guitar star Pat Metheny and the drum virtuoso Tony Williams. In later years, he collaborated with Japanese art-of-noise rock band the Ruins, and - when he had already passed 70 - with young drum and bass DJs.

Singlemindedly devoted to unpremeditated improvisation, Bailey published a book on the subject in 1980 called Improvisation: Its Nature and Practice in Music. Twelve years later, it led to Jeremy Marre's revealing Channel 4 four-parter On the Edge: Improvisation in Music, an ambitious venture that Bailey both scripted and presented. The project tracked the improvising impulse through the most radical interpreters of Mozart, the methods of the organist at the Sacré Coeur, Paris, in baroque music or the blues, and in locations from the Hebrides to the Ganges.

Bailey was born to George and Lily Bailey, in the Abbeydale district of Sheffield. His father was a barber, his uncle a professional guitarist who gave the boy his first instrument and some haphazard lessons. By a process of osmosis from musicians he met, sustenance from odd jobs, record-listening (bebop guitar pioneer Charlie Christian was his early model) and some later self-education in theory and arranging, Bailey became a pro on the UK dance-band and studio circuit in the early 1950s. By 1965, he was playing Blackpool seasons for Morecambe and Wise.

By that time, he had begun rehearsing regularly with two adventurous younger players in Sheffield - classical percussionist turned jazz drummer Tony Oxley and bassist (later to become classical composer) Gavin Bryars. The three formed the group Joseph Holbrooke (named after an obscure British composer whose work they never played), and, from 1963 to 1966, its jazz beginnings in John Coltrane and the Bill Evans Trio were crossbred with ideas from John Cage, Stockhausen, serialism, Oxley's labyrinthine rhythm variations, and much more. Gradually, the group moved from jazz into a non-idiomatic approach - free-improvisation.

From 1966, Bailey began visiting the Little Theatre Club, a West End bolthole where the drummer John Stevens ran all-comers' sessions and young improvisers (including Evan Parker, Trevor Watts and Paul Rutherford), jazz virtuosi (Dave Holland, Kenny Wheeler) and contemporary classical players like Barry Guy gathered. With various versions of Stevens' Spontaneous Music Ensemble, Oxley's sextet, the Music Improvisation Company (electronics, percussion and Parker's sax) and the trio Iskra 1903 (with trombonist Rutherford and bassist Guy), Bailey began to build a completely new vocabulary for the guitar.

Though he never abandoned the conventional instrument, he was mixing warped chordal ideas, serialism's lateral melodies, Cage's elevation of silence, pedal-operated electronics and a brittle attack borrowed from percussionists. From 1970, he also ran the Incus Records label, first with Oxley and Parker, then with his partner (and later third wife) Karen Brookman - their Hackney flat is still the Incus HQ.

Bailey's Diaghilev qualities came to the fore in 1976, when he began his Company project, an improvisers' festival that involved 400 players each year up to 1994 in Britain, the US and Japan, with Zorn, Lee Konitz, saxist Steve Lacy, classical violinist Alexander Balanescu, bassoonist Lindsey Cooper and composer/saxist Anthony Braxton among those taking part. He also invited dancers, performance-artists, electronica-specialists and avant-rockers to join in, with the artists deciding who would improvise with who.

He likened improvisation to spontaneous relationships and conversation - full of accidental harmonies, misunderstandings, passion and indifference. Though a sophisticated instrumentalist himself, he did not mind playing with people who had comparatively few skills; something interesting might always happen. He worked with bassist Bill Laswell and drummer Tony Williams in the trio Arcana in 1995, and collaborated with Pat Metheny and two percussionists on The Sign Of Four in 1996.

He described that encounter to me thus: "The equipment I use I bought in Canal Street 15 years ago. Pat's sitting in the middle of what looks like the console of a 747, with four guitars and a distortion unit that could be used for dispersing mobs. There were two guys with huge percussion kits, and I'm making a lot of noise, and then he switches this thing on, and it's like there's three dogs playing around a little, and suddenly an elephant lands on top of them."

Yet for all that raw-noise energy, Bailey continued to be a delicate acoustic improviser, often unaccompanied or in duets. Just in time, he was caught by the ideal biographer, Ben Watson, in the book Derek Bailey and the Story of Free Improvisation. And, though his combativeness never left him, he seemed to take heart from the musical eclecticism and dissolution of idiomatic differences he had done so much to encourage.

"The kids don't mind whatever it is these days," he told me once. "Maybe there's a lot of stuff out there now that is by its nature odd. But they seem to be able to take anything. Which is great to somebody like me. I find it very comfortable. In an uncomfortable sort of way." Karen survives him, as does Simon, the son of his second marriage.

Richard Williams writes: The least typical recording Derek Bailey ever made also turned out (not that he would have appreciated the compliment) to be one of the great jazz recordings of the last 40 years. Titled simply Ballads, and recorded in 2002 for John Zorn's Tzadik label, it consisted of solo guitar meditations on 14 songs from the standard repertoire, including Laura, Body and Soul, What's New, Stella by Starlight and You Go to My Head.

Although this was the last project one might have expected from a professed enemy of composed music, it was no surprise to discover that in these songs - their musical and emotional contours long since flattened by overuse - Bailey found brand new angles and meanings, thanks to the application of his highly personal imagination and unique instrumental language. Extraordinary renditions, indeed, and utterly spellbinding.

By the time he recorded another solo CD for Tzadik, entitled Carpal Tunnel, three years later, his refined technique had all but disappeared. No longer able to grasp a plectrum with his right hand, he adapted by striking the strings with his thumb. The album's title came from the condition, carpal tunnel syndrome, that was said by doctors to explain his reduced dexterity. In fact, it marked the onset of the motor neurone disease from which he died.

In these pieces, the spiky elegance of Ballads is replaced by a halting delicacy reminiscent both of Japanese koto music and of the last paintings of Willem de Kooning, when illness had robbed the great abstract expressionist of the power to do anything other than trace a haunting shadow of the shapes and colours that had once burst from the canvas.

· Derek Bailey, improvising guitarist, born January 29 1930; died December 25 2005