ryanheise wrote:I've updated the script to display the word position in the frequency list (e.g. "10" means "10th most frequent word). It will show a blank if the word is not in the corpus (which again is clipped off at 10,000 words due to resource constraints on the server).
Here is a summary of my findings. My concern yesterday about the large values I was getting for mean and variance were completely unfounded, since mean is "self-adjusting" so it remains a practical size, and since variance needs to be unlimited, and since your list has a maximum size (10,000), anyway. I added a 4th example that was intended to be very sophisticated text with a high measure of sophistication, and its large mean that was produced confirmed my impression.
TEXT EXCERPTS CHOSEN:
(1) "Go, Dog. Go!"
http://iepclass.weebly.com/uploads/1/1/ ... astman.pdf(2) "Alice's Adventures in Wonderland," Chapter IV
https://www.owleyes.org/text/alices-adv ... chapter-iv(3) U.S. Constitution Preamble
https://constitutioncenter.org/interact ... n/preamble(4) photonics article abstract
https:https://pubs.acs.org/doi/10.1021/acsphotonics.0c01530e
----------
UNREFINED
(1a)
SUBMITTED TEXT:
Now all the dogs get out. And now look where those dogs are going! To the tree! To the tree! Up the tree! Up the tree! Up they go to the top of the tree. Why? Will they work there? Will they play there? What is up there on top of that tree? A dog party! A big dog party! Big dogs, little dogs,
RETURNED TEXT:
Now all the dogs get out. And now look where those dogs are going! To the tree! To the tree! Up the tree! Up the tree! Up they go to the top of the tree. Why? Will they work there? Will they play there? What is up there on top of that tree? A dog party! A big dog party! Big dogs, little dogs,
CALCULATOR OUTPUT:
Variance s2 = 2557945.9
Standard Deviation s = 1599.358
Count n = 64
Mean x¯¯¯ = 620
Sum of Squares SS = 161150590
(2a)
SUBMITTED TEXT:
It was the White Rabbit, trotting slowly back again, and looking anxiously about as it went, as if it had lost something; and she heard it muttering to itself, “The Duchess! The Duchess! Oh my dear paws! Oh my fur and whiskers! She'll get me executed, as sure as ferrets are ferrets! Where can I have dropped them, I wonder!” Alice guessed in a moment that it was looking for the fan and the pair of white kid gloves, and she very good-naturedly began hunting about for them, but they were nowhere to be seen—everything seemed to have changed since her swim in the pool, and the great hall, with the glass table and the little door, had vanished completely.
RETURNED TEXT:
It was the White ?????? ?????? slowly back again and looking anxiously about as it went as if it had lost something and she heard it ?????? to itself The Duchess The Duchess Oh my dear ?????? Oh my fur and ?????? ?????? get me executed as sure as ?????? are ?????? Where can I have dropped them I wonder Alice ?????? in a moment that it was looking for the fan and the pair of white kid gloves and she very good ?????? began hunting about for them but they were ?????? to be seen everything seemed to have changed since her ?????? in the pool and the great hall with the glass table and the little door had vanished ??????
CALCULATOR OUTPUT:
Variance s2 = 10967712
Standard Deviation s = 3311.7536
Count n = 122
Mean x¯¯¯ = 1860.9344
Sum of Squares SS = 1327093100
(3a)
SUBMITTED TEXT:
We the People of the United States, in Order to form a more perfect Union, establish Justice, insure domestic Tranquility, provide for the common defense, promote the general Welfare, and secure the Blessings of Liberty to ourselves and our Posterity, do ordain and establish this Constitution for the United States of America.
RETURNED TEXT:
We the People of the United States in Order to form a more perfect Union establish Justice insure domestic ?????? provide for the common defense promote the general Welfare and secure the Blessings of Liberty to ourselves and our Posterity do ?????? and establish this Constitution for the United States of America
CALCULATOR OUTPUT:
Variance s2 = 4445830.1
Standard Deviation s = 2108.5137
Count n = 50
Mean x¯¯¯ = 1017.74
Sum of Squares SS = 217845670
(4a)
SUBMITTED TEXT:
We report the experimental realization of periodically perforated plasmonic metasurfaces capable of integrating several key functionalities, such as light-to-surface plasmon coupling, controllable beam-splitting, wavelength filtering and routing, high resolution differential wavelength measurement, and vectorial displacement sensing.
RETURNED TEXT:
We report the experimental realization of periodically ?????? ?????? ?????? capable of integrating several key ?????? such as light to surface ?????? coupling ?????? beam splitting wavelength filtering and routing high resolution differential wavelength measurement and ?????? displacement sensing
CALCULATOR OUTPUT:
Variance s2 = 17950831
Standard Deviation s = 4236.8421
Count n = 39
Mean x¯¯¯ = 4653.0513
Sum of Squares SS = 682131580
----------
REFINED
The types of words I filtered out of all samples were:
all articles
all conjunctions
all prepositions
all pronouns
all possessive adjectives
all auxiliary verbs
(1b)
ORIGINAL TEXT:
Now all the dogs get out. And now look where those dogs are going! To the tree! To the tree! Up the tree! Up the tree! Up they go to the top of the tree. Why? Will they work there? Will they play there? What is up there on top of that tree? A dog party! A big dog party! Big dogs, little dogs,
SUBMITTED, FILTERED TEXT:
Now all dogs get out. now look where dogs going! tree! tree! tree! tree! Up go top tree. Why? work there? play there? What is up there top tree? dog party! big dog party! Big dogs, little dogs,
RETURNED TEXT:
Now all dogs get out now look where dogs going tree tree tree tree Up go top tree Why work there play there What is up there top tree dog party big dog party Big dogs little dogs
CALCULATOR OUTPUT:
Variance s2 = 3926854
Standard Deviation s = 1981.6291
Count n = 38
Mean x¯¯¯ = 1031.5263
Sum of Squares SS = 145293600
(2b)
ORIGINAL TEXT:
It was the White Rabbit, trotting slowly back again, and looking anxiously about as it went, as if it had lost something; and she heard it muttering to itself, “The Duchess! The Duchess! Oh my dear paws! Oh my fur and whiskers! She'll get me executed, as sure as ferrets are ferrets! Where can I have dropped them, I wonder!” Alice guessed in a moment that it was looking for the fan and the pair of white kid gloves, and she very good-naturedly began hunting about for them, but they were nowhere to be seen—everything seemed to have changed since her swim in the pool, and the great hall, with the glass table and the little door, had vanished completely.
SUBMITTED, FILTERED TEXT:
was White Rabbit, trotting slowly back again, looking anxiously went, lost something; heard muttering, “Duchess! Duchess! Oh dear paws! Oh fur whiskers! get executed, sure ferrets are ferrets! Where can dropped, wonder!” Alice guessed moment looking fan pair white kid gloves, very good-naturedly began hunting, were nowhere be seen—everything seemed changed since swim pool, great hall, glass table little door, vanished completely.
RETURNED TEXT:
was White ?????? ?????? slowly back again looking anxiously went lost something heard ?????? Duchess Duchess Oh dear ?????? Oh fur ?????? get executed sure ?????? are ?????? Where can dropped wonder Alice ?????? moment looking fan pair white kid gloves very good ?????? began hunting were ?????? be seen everything seemed changed since ?????? pool great hall glass table little door vanished ??????
CALCULATOR OUTPUT:
Variance s2 = 14622024
Standard Deviation s = 3823.8755
Count n = 65
Mean x¯¯¯ = 3313.2923
Sum of Squares SS = 935809520
(3b)
ORIGINAL TEXT:
We the People of the United States, in Order to form a more perfect Union, establish Justice, insure domestic Tranquility, provide for the common defense, promote the general Welfare, and secure the Blessings of Liberty to ourselves and our Posterity, do ordain and establish this Constitution for the United States of America.
SUBMITTED, FILTERED TEXT:
People United States, order form more perfect Union, establish Justice, insure domestic Tranquility, provide common defense, promote general Welfare, secure Blessings Liberty Posterity, do ordain establish Constitution United States America.
RETURNED TEXT:
People United States order form more perfect Union establish Justice insure domestic ?????? provide common defense promote general Welfare secure Blessings Liberty Posterity do ?????? establish Constitution United States America
CALCULATOR OUTPUT:
Variance s2 = 10642754
Standard Deviation s = 3262.3234
Count n = 30
Mean
¯
x
= 2302.7
Sum of Squares SS = 308639870
(4b)
ORIGINAL TEXT:
We report the experimental realization of periodically perforated plasmonic metasurfaces capable of integrating several key functionalities, such as light-to-surface plasmon coupling, controllable beam-splitting, wavelength filtering and routing, high resolution differential wavelength measurement, and vectorial displacement sensing.
SUBMITTED, FILTERED TEXT:
report experimental realization periodically perforated plasmonic metasurfaces capable integrating several key functionalities, light-surface plasmon coupling, controllable beam-splitting, wavelength filtering routing, high resolution differential wavelength measurement, vectorial displacement sensing.
RETURNED TEXT:
report experimental realization periodically ?????? ?????? ?????? capable integrating several key ?????? light surface ?????? coupling ?????? beam splitting wavelength filtering routing high resolution differential wavelength measurement ?????? displacement sensing
CALCULATOR OUTPUT:
Variance s2 = 14843239
Standard Deviation s = 3852.6924
Count n = 30
Mean x¯¯¯ = 6044.4333
Sum of Squares SS = 430453920
----------
UNREFINED MEANS:
(1a) 620
(2a) 1861
(3a) 1018
(4a) 4653
REFINED MEANS:
(1b) 1032
(2b) 3313
(3b) 2303
(4b) 6044
Each mean can be interpreted as the measure of "sophistication" or "unintelligibility" of that text excerpt.
Since the 4 text excerpts were ordered in what was believed to be increasing sophistication, (1) through (4), the fact that the means also usually increase in that order is strong evidence that this method of measuring text sophistication works.
In general, the more refinement that is done to the data (preprocessing to "sharpen the data"), the higher the means will be, because the filtered out words tend to be the words with high frequency, which leaves behind only words with low frequency, which means words with high values. This is why the means for the second set of 4 are larger than the first set of 4. The programmer can use any given degree of preprocessing, including none at all, as a standard of measurement. There exists some maximum value of each of these means, one that corresponds to perfect preprocessing, but that maximum would be very difficult to produce or to estimate.
High variance values (not shown in the summary) occurred in excerpts (2) and (4), especially in (2), and signal the presence of some anomaly in the consistency of sophistication of the text excerpt.