Art that works through protocols, the shared rules and standards behind networks and blockchains. It treats the protocol itself as a medium.
I think there's an interesting distinction to be made between artists that are making art on top of a protocol, for instance NFT artists. They are using the blockchain protocol to deploy a digital artwork, versus the more protocol as a medium, which in my view at least is closer to protocol art, which is you're actually putting the creative endeavor in the design of the protocol.
Primavera De Filippi, Strange Rules: Protocol Art at the Venice Biennale
Protocol art is a little more opinionated than that; our systems/protocols are set up to enact a certain kind of interaction and outcome—a little more like an AI agent than an arbitrary feedback system.
Mat Dryhurst, Mat Dryhurst on Becoming Infinite