John Stuart Mill on Public and Private Actions

I am always intrigued when I see John Stuart Mill arguing for less freedom than I expect. While expansive in the degree of freedom he argues for behind closed doors, In paragraph 7 of On Liberty “Chapter V: Applications,” paragraph 7, he argues for much tighter limitations on what can be done in public:

Again, there are many acts which, being directly injurious only to the agents themselves, ought not to be legally interdicted, but which, if done publicly, are a violation of good manners, and coming thus within the category of offences against others, may rightfully be prohibited. Of this kind are offences against decency; on which it is unnecessary to dwell, the rather as they are only connected indirectly with our subject, the objection to publicity being equally strong in the case of many actions not in themselves condemnable, nor supposed to be so.

I find this troubling when one considers that many actions for which it is contested whether they are injurious or not. And doing something in public as performance art can be a form a free speech. 

On the other hand, I don’t see how to make society work without some significant limitations on what one can do in public that go beyond any limitations on what one can do in private. 

I am actually quite puzzled about how to draw the line in a principled way between things that should be allowed both in public and private and those that should be allowed only in private. And it occurs to me that any serious classroom discussion of On Liberty should wrestle with this question.