My thoughts exactly.
It seems a person incapable of proper grammar (like a baby) has some concept or thought it wants to express, but can't because it doesn't know the words etc.
These language models seem to know the words and the grammar etc, but lack a underlying concept they want to express.
There are systems that derive 'thought-vectors', but I'd be interested going the other way: somehow create such a 'thought-vector' and generate text to express that thought.
I don't know how to construct a 'thought-vector' of any concept though.
I have been thinking about this kind of thing too. What if there was some way to feed your condensed thoughts into such a model and it writes a paper/blog post/article?
Essentially, one should be able to use these models to "interpolate" the writing around the raw meaning/content. Typing assistance (think Grammarly) already allows you to refine finished writing to be more in line with what some language model expects, but imagine if it actually generated most of the text for you, based on small bites and chunks you throw at it.
If we get to large scale text generation like that, we are all going to have to become even better skimmers due to how the meta language will evolve.
So take your standard press release. We know about two thirds of it is just fluff. In other words, we are accepting the mass of fluff as one word in our language, it translates to ‘ignore’.
What might some valid sources of data for this be? Perhaps comparisons of Simple English Wikipedia to the standard English Wikipedia? We'd need a side-by-side comparison of condensed information and a fluffed up piece.
These language models seem to know the words and the grammar etc, but lack a underlying concept they want to express.
There are systems that derive 'thought-vectors', but I'd be interested going the other way: somehow create such a 'thought-vector' and generate text to express that thought.
I don't know how to construct a 'thought-vector' of any concept though.