James Joyce flouted all the rules and convention and he is regarded as a literary genius. So Is spelling, grammar and punctuation really important? Or is it enough that what you write is understood by the reader?
Communication is the reason to write. How you are understood within that writing is where the use (or non-use) of grammar come in to play.
This question asks for an answer that is going to be more opinion than fact. But I'll tackle it and see where the answer goes.
Before diving in (and since I've never gulp read Joyce) I thought I'd throw in a link for the wikipedia article about him: https://en.wikipedia.org/wiki/James_Joyce
Writing, as in any other artistic pursuit, is all about the transmission of information, of communication. The artist/writer/performer/whomever is doing what they are doing to convey information - whether that be to inform, or entertain, or whatever. The writer writes to communicate with the reader. So in my opinion, the writer seeks above all else to communicate.
How that communication is received by the reader is another issue entirely. We can take examples from different writers and their books and see how it is that they write in order to communicate. Does a writer seek to be understood in a "normal" way, using proper sentence structure, spelling, grammar, and all the other rules of writing? And most of the time, the answer is an unqualified "yes" - that those rules are in place to help the writer and reader find a common ground so that what the writer is trying to convey and what the reader is getting out of that writing are as close to being the same as possible.
There are times when a writer will put in bits and pieces where the rules are bent, and every so often totally thrown out the window and these times are used to contrast against the grammar rules we are anticipating or expecting.
That said, every so often a writer will come along and break the mould and redefine what literature is and what it should look like. Jack Kerouac did that in his stream of consciousness style - little to no grammar marks in the writing at all. And as is pointed out in the question, Joyce broke many rules in his writing as well.
The next question is - Did it work? Read some of the thoughts in the Wiki article and you'll see that while Joyce himself is regarded as brilliant, not all of his individual works are seen in that way by all people. Finnegans Wake seems to be a work that catches a lot of criticism while Ulysses seems to catch a lot of praise.
And so, in closing, I believe the writer must use whatever devices are available to make him or herself known to the reader. That said, most of the time proper grammar is the well-worn route to that end. But it needn't be the only route.
In the end, it's each person's perspective on the writing, both the writer and the reader, who will decide what it is that is being conveyed. Just keep in mind, dear writer, that there is only one of you and potentially many many readers out there.
Joyce had a message that required him to change the rules.
You do realize the strength of ideas needed so that readers understand you despite idiosyncratic spelling and free creation of words. That strength is rare. Joyce had it. Extrapolating might be risky :) But the best answer I could give you is this. Try it. Let us know how it goes.