What does it mean when an apostrophe is used to demonstrate dialogue instead of quotation marks in novel or story?
Single quotes get used to help differentiate quotes within quotes.
Sometimes it happens that someone will be speaking, which means the text is inside quotation marks, and that person within their talking will quote someone else. How do you keep track of what is what? Single quotes (which looks exactly like an apostrophe).
Let's take a look at that:
Geddy and Lee are walking along a street. Geddy looked up and saw a poster for a Shakespearean Festival, and said "It's amazing that there are still Shakespearean Festivals". Lee responded "It's not at all amazing. Just think about all that he influenced. Even my favourite band, Rush, references him. 'All the world is indeed a stage and we are merely players...' " [italicized so you can see where the single quotes are].