Chronology Current Month Current Thread Current Date
[Year List] [Month List (current year)] [Date Index] [Thread Index] [Thread Prev] [Thread Next] [Date Prev] [Date Next]

Re: Entropy (very long)



Regarding Chuck B.'s observation of the use of the terms 'entropy' and
'disorder' in an AIP Physics News Update (# 469):

It's no wonder that some HS teacher continue to promulgate such
blatantly dangerous notions as entropy being related to disorder.

SOMEBODY working for the AIP has similar misconceptions!

e.g.
...
From the perspective of thermodynamics, controlling an object means reducing its
disorder, or entropy. Lowering the disorder of a hot gas, for example, decreases the
^^^^^^^^ ^^^^^^^
number of possible microscopic arrangements in the gas. This in turn
removes some of the uncertainty from the gas's detailed properties.
...

I personally see nothing objectionable with the usage of these terms in
the article. I suspect that Chuck is reacting to one or more previous
rants by others (the initials L. P. come to mind most prominantly) on
this list about the danger of identifying thermodynamic entropy and the
d-word. The others would have to speak for themselves in this regard,
but I see little conceptual danger in the association as long as the term
'disorder' is used in an imprecise and nontechnical sense, and it is
understood that the disorder to be associated with (thermodynamic)
entropy is a disorder *at the level* of the microscopic states of the
macroscopic thermodynamic system. It is possible, I suppose, that an
unsuspecting student might bring to the conceptual table ideas of the
relevant disorder possibly being at the level of a macroscopic
arrangement of macroscopic parts of a thermodynamic system. If so then
the student would be liable to be confused. For instance, a thoroughly
shuffled deck of cards has essentially the same thermodynamic entropy
as a well-ordered deck (assuming, of course, both decks have the same
temperature and are subject to the same external macroscopic
environment). OTOH, a cold deck of cards has less thermodynamic entropy
than a warm one regardless of how the decks were shuffled or not.

The terms, 'information', 'disorder', 'order', 'information',
'complexity', 'uncertainty', (even 'entropy'), etc. can all be used in
both an imprecise and colloquial as well as a precise and technical
manner. But the same goes for such words as 'energy', 'power', 'force',
'work', 'color', 'black', 'white', 'flavor', 'heat', 'flow', etc.
Physics/physicists makes/make a point (that I, personally, like) of
trying to coopt short mundane and vernacualar terms and to imbue them
with precise technical meaning when inventing the necessary disciplinary
jargon. It helps to give the conceptually difficult discipline an
aura of approchability. Physics is hard enough as it is to learn
and understand without having to wade through a morass of big
unfamiliar terms. This is in marked contrast with the tendency of some
other disciplines to adopt a technical vocabulary that is replete with
long, pretentious, and unfamiliar words to encode a range of technical
concepts that are intellectually simple at heart. It seems that such
disciplines try to make up for with obscure jargon what they lack in
intellectual depth.

A down side, though, of the physics tendency to use a technical
vocabulary composed of extant simple vernacualar words is that it is an
invitation for the learners to associate previously learned outside
imprecise everyday meanings with the words that physics has declared to
have a single precise technical meaning. This is a recipe for creating
and sustaining lots of misconceptions in the minds of the learners at
entry levels. At higher levels the learners have, by then, become well
aware of the need to use the newly introduced technical terms in only the
technically defined way, and to eschew any previous connotations the
simple word might have previously had in the non-physics world.

Another problem that sometimes arises in the terminology of physics is
that the same common word may be occasionally adopted for multiple
technical meanings. This, too, can lead to confusion for the student.
For instance the term 'heat' is used quite differently in the next
two sentences. "The latent heat for the phase change for the substance
is 750 KJ/kg." "The Joule paddle-wheel experiment shows that
1 cal. of heat is equivalent to 4.18 J of work." (Aside to J. G.: Both
usages of the h-word here were as nouns, not verbs. Sorry.) This
practice of assigning multiple technical meaning to words is more common
in physics than I would like. It is sort of like using the same letter
in mathematical equations to signify more than one quantity. In both
instances context is the key to proper understanding.

But, back to the constellation of terms: 'disorder', 'entropy',
'uncertainty', 'random', 'complexity', 'information', etc. These terms
are words that are often used in precise single-meaning technical ways
in a given physics/information theory/probability theory context by a
given author. But different authors sometimes permute the assignments
of the meanings to the words. This, too, is unfortunate and can lead
to much confusion. I think part of the problem here is that information
theory is a new enough field that its jargon has not yet become 100%
standardized.

In the specific cases of the terms 'entropy' and 'disorder', I would
*not* use them as synonyms. I would consider the concepts they
signify as related, but not identical, to each other. What relates them
is that they are both measures of (missing) information. What
distinguishs them is that they are *different* such measures.

My general definition of entropy (as a info theoretic/Bayesian
probability theoretic concept) is that it is the average minimal amount
of further information necessary to determine with certainty the precise
outcome of a random process governed by a given probability distribution
when a sample is drawn from that distribution by the process given that
the only prior information about the process is contained in the
assignment of probablities in the distibution. The mathematical
form of this is: S = Sum{i, p_i*log(1/p_i)} where the index i in the
sum labels the individual outcomes for the distribution, and p_i is the
probability for outcome i, and all such outcomes are summed over. The
base to which the logarithm is taken in the formula determines the units
in which the entropy is to be measured. If the base of logarithms is b
then the amount of information represented by S is the number of
symbols necessary to convey the needed information when the information
is encoded using a set of symbols whose cardinality is b. Thus, if the
base b is 2 then the entropy is measured in bits; if b = 256 = 2^8 then
S is measured in bytes; if b = 10 then S is in decimal digits; if
b = 2^8192 then S is measured in kbytes, etc. Since the entropy S is
just the average number of needed information bearing symbols, it is
fundamentally a dimensionless concept (that is nonnegative in value).

This definition implies that since entropy is an *average* over a
distribution it is a statistical concept that is a property of
statistical ensemble of possibilities, (i.e. the probability distribution
itself). It is not a property of individual outcomes or states drawn
from the universe of possibilities. The entropy is a functional on the
space of probability distributions. It is not a function of an
individual given realization or sample drawn from such a distribution.

In a specifically thermodynamics/stat. mech. context I would define
thermodynamic entropy as the average minimal amount of further
information necessary to determine the exact microscopic state of a
macroscopic thermodynamic system given only the precise specification of
system's macroscopic state. In this special case the universe of
possibilities is the set of all microscopic states consistent with the
macroscopic description. The relevant probability distribution over that
universe is the set of probabilities for the realization of the
individual microscopic states under the conditions imposed by the
macroscopic state. Since the macroscopic state determines (in a
well-defined Baysesian sense) the distribution of possible microscopic
states we see that thermodynamic entropy is a function of the system's
macroscopic state. In order to measure the thermodynamic entropy in
conventional units such as J/K we can think of the base of logarithms as
being b = e^(1/(1.38065 x 10^(-23)))) = 10^(3.14558 x 10^22). This is
equivalent to taking the base b to be the Naperian base e and multiplying
the resulting logarithm by Boltzmann's constant k = 1.38065x10^(-23) J/K.

Since entropy is really dimensionless this means that Boltzmann's
constant can be thought of a conversion constant between energy
measured in joules and energy measured in kelvins. (This is sort of like
how in relativity c is a conversion constant that converts between
spacetime displacements measured in meters and those measured in
seconds). This makes the concept of thermodynamic temperature
intrinsically an energy concept (albeit an intensive one). Specifically,
we can (for mathematical convenience in doing the calculus of thermal
physics) consider the base b to be the Naperian constant e and consider
the entropy to be measured in so-called "nats" where the conversion is
1 bit = ln(2) nat. If this is done then a temperature of 1 K really
means the intensive (energy/per entropy units) of 1.38065x10^(-23) J/nat.
In these units the value of Boltzmann's constant is the pure number 1.
The triple point temperature of pure water is then 3.77138x10^(-21) J/nat.
Also, since thermodynamic temperature is defined by the partial
deriviative 1/T == dS/dE (where the variation for this partial derivative
is taken quasi-statically under the conditions of no macro-work
being done) we see that this derivative makes temperature have
units of energy/entropy, i.e. J/nat.

Using such a unit system the ideal gas law is just P*V = N*T where N is
the number of particles present in the gas and T is the absolute
temperature measured in J/nat. The average translational kinetic energy
E_trans of a classical particle in thermal equilibrium with an environment
at temperature T is then E_trans = (3/2)*T. In the the dimensional
analysis of these last couple of formulas the dimensionless 'nat' unit
evaporates in exactly the same way the dimensionless 'radian' unit
evaporates in the dimensional analysis of the formula v = r*[omega] when
r is a radius distance and [omega] is an angular velocity in rad/s.

What about the term 'disorder'? What is it? this term is not
nearly as standardized in meaning in the literature as the term
'entropy' is. My own *personal* definition for disorder presupposes
a prior definition of the term 'complexity'. Following Chaitin and
Kolmogorov, the 'complexity' of an object is the number of symbols in
describing the shortest possible algorithm capable of *completely*
reproducing or reconstructing the object. Thus, the complexity
represents the length of the most compact complete set of instructions
for exactly assembling the object.

Now consider some (usually complex) object that is made from or with
'parts' (i.e. subsystems) which are assembled together to make the whole
object of interest. Suppose these 'parts' can be arranged in a multitude
of ways to make a whole ensemble of possible composite systems. The
total complexity of these composite composite systems depends to some
extent on just how the parts are arranged. Consider the complexities of
all such composite objects. This set of complexity values has a minimal
value. The composite object(s) that has(have) this minimal possible
complexity value over the whole ensemble is/are said to be the
most-ordered object(s). The difference between the actual complexity of
each of the ensemble members and this minimal complexity value defines
(by my definition of the term) the 'disorder' of each such composite
object. The disorder in a given composite object is how much more
complex that object is than the most-ordered object(s) than can be made
of the same consituent parts. Thus, the complexity of a most-ordered
object represents essentially the complexity of the parts themselves and
of all the necessary connections to produce a composite object, and the
disorder in a given composite object represents the further amount of
information needed to arrange the parts in the manner found in that
particular composite object. The disorder is thus a measure of the
information needed to *arrange* the preexistant parts into the whole
composite object. The disorder is thus a measure of how complex a given
*arrangement* of the parts is.

As a concete example consider a deck of playing cards arranged in a
stack. The most ordered state is the stack (or stacks) that have the
minimal total complexity. This minimal complexity is *huge* if our
construction algorithm is detailed enough to exactly specify the exact
placement of every subatomic particle in the stack. It would not
be nearly so large it only explained how to make and stack the
cards from some preexistant formula of fixed amounts of paper pulp and
ink. The disorder for any given stack of the cards is the minimal
amount of extra information (beyond that already required to make the
most-ordered stack in the first place) needed to arrange the cards in
the sequence in which they happen to be found.

These definitions mean there are three important differences between the
concept of disorder and the concept of thermodynamic entropy. First, the
thermodynamic entropy involves the information for the specification of
the microscopic state in terms of the actual microscopic degrees of
freedom in the system present, whereas the disorder is a property
tha operates at the higher level of a macroscopic arrangement. Second,
the thermodynamic entropy is an *average* statistical property property
of an ensemble (of possible microstates) and is not defined for
individual members of the ensemble, whereas the disorder *is* a property
of *each individual* arrangement. Third, the amount of information
averaged over in the entropy is merely the information necessary to
merely pick out the individual case from an enumerated list of the
possibilities, whereas the amount information used in the disorder is a
complexity measure and that represents the amount of information
necessary to actually *reconstuct* (not just pick out from a list) the
actual arrangement.

I'm sorry about how long this post ended up becoming, and apologize to
any reader who made it this far for taking up so much or your time. I
hope you may have found something interesting to think about in it.

David Bowman
David_Bowman@georgetowncollege.edu