Grammatical categories are just tools to decompose a language into very simple, independent processes and rules that can be studied separately. But the actual language is much more convoluted than just a vocabulary and some universally applicable grammar rules.
Let's take the English word "can". Its infinitive either doesn't exist at all, or it takes the highly irregularly derived form "to be able to". (The first approach to describing the situation is traditionally preferred by grammarians.)
Non-passive binyanim are normally described as word derivation processes (because they are semantically non-transparent: you can't fully predict what a stem means just from the binyan and the root; and because they are non-productive: the stems are fixed, speakers of Hebrew aren't deriving new ones on the go, and if they do, they risk a misunderstanding). Infinitive formation is normally described as an inflection process, and that surely suggests that you should be able to create an infinitive from any verb, including pu'al and huf'al. Alas, like with modal verbs of English (see the example of "can" above), those binyanim are grammatically defective, and there's no infinitive nor imperative.
OK, so far I'm just saying "those words don't form an infinitive, live with that". However, I think that I just found a hint of an answer to your question in Doron's draft paper Binyanim - form and function:
The marked binyanim,
the causative [hifˈil/hufˈal] and the intensive [piˈel/puˈal], express the thematic role of the verb's subject: cause
and agent respectively. The simple binyan [paˈal/nifˈal] functions as default and is neutral as to the
Well, by definition, an infinitive is the word form which does not have an expressed subject. So it should better be neutral as to the subject's role (or thematic relation). But let's not stop here.
That same article analyzes each binyan as to the grammatical voice. In the active voice (pa'al, pi'el, hif'il), the subject is the agent, the object is the patient; in the passive voice (pu'al, huf'al), the reverse mapping of the subject/object roles to those thematic relations applies; in the middle voice (nif'al, hitpa'el) the mapping is neither of those (the several interpretations of the middle voice given in the article are interesting in themselves, but not relevant to the question at hand).
Now Doron says something about the passive voice in Modern Hebrew worth contemplating:
All verbs derived by the non-active binyanim are intransitive [(unary - not requiring any object)]. But there is an important difference between the intransitivity of the middle binyan and that of the passive binyan. While the subject argument of the corresponding active verb can be totally obliterated in the derivation of the middle verb, it always implicitly participates in the derivation of the passive verb [pu'al/hufˈal]. Moreover, this implicit participant (which can also be expressed explicitly as an ידי על ʿal-yede(y) 'by' phrase) is agentive,
irrespective of the thematic role of the subject in the corresponding active binyan.
I believe that it is this implicit agentive participant of the sentence structure which is semantically blocking formation of imperatives and possibly even infinitives.
It seems to me more obvious in case of the imperative. If the imperative is used as a command or request, as it usually is, the addressee is supposed to become the agent of the requested action. If I have to accept that the imperative in pu'al or huf'al also has that "non-obliterable" additional agentive role, we would be ending up with a middle voice construction where the addressee is simultaneously the subject and the object of the imperative; but pu'al and huf'al are not middle voice binyanim. Nif'al is.
Taking the root כּתב, you can create a passive imperative using nif'al just fine: היכתב!, "get written!", or rather "sign up!"; you are commanding the addressee to be both the agent and the patient of the writing action, a middle voice, as opposed to a passive voice construction.
It seems to me that infinitives of pu'al or huf'al would have an analogous problem only in some specific constructions, for example as part of complex imperatives (infinitive form dependent on an imperative form). So what I'm offering is just a hint of why these binyanim and not others are more likely to be defective in this particular way; there may be some much stronger reasons I'm missing, including phonological or purely historical ones.
And indeed: in many languages including English, both imperatives and infinitives are possible in constructions traditionally labelled as passive voice; not just a "sign up!", but also a genuine, if a little bizarre, "get written, you overdue unfinished edit!".