XML, why Mma is popular, was...Re: [Maxima] Lunch with George Carrette

At the risk of some repetition, let me attach the history
of Macsyma as I see it. I know the history after 1986 first-
hand, and I have tried to incorporate as much about the
earlier period as I can. Please send me any further 
corrections. (I may not have include all that have been
sent to me, for which I apologize.)

Regarding 'marketing glitz', Wolfram found the right way to 
exploit the mismanagement of Macsyma by M.I.T. and Symbolics. 
He started by focusing on notebook interface and graphics, 
and then on numerics, while Mathematica's symbolic math was 
still very weak. Wolfram correctly assessed what most people 
wanted, and what he should do first before tackling the 
extremely demanding task of building a first-class Symbolic 
math engine. Given his background in physics, he understood
what scientists wanted, but he did not understand at that
time how to build a great symbolic engine, nor did he 
understand what engineers wanted, so that WRI still does not
do so well in that segment. (I now work for MathWorks which 
does understand what engineers want, and it is by and large
very different.)

As you can read below, the failure of Macsyma required the 
failure of academic, government, commercial, and philanthropic 
Sectors -- quite a stunning record.


Richard Petti

=============================================================

                         Macsyma Saga

--------
1. History
--------

----
1.1 M.I.T.
----

Macsyma was developed at M.I.T. from 1968 to 1982 with funds from 
the U.S. Defense Advanced Research Projects Agency and some from the 
Department of Energy. The government lost interest in Macsyma around 
1977 when numerical analysts persuaded the government than numerical 
libraries on supercomputers were better suited to performing the 
engineering computations needed by the defense establishment.

In the late 1970s the professor in charge of Macsyma at M.I.T. wanted 
to lead a company to commercialized Macsyma while he remained a 
professor at M.I.T. M.I.T. has never permitted this sort of arrange-
ment, and would not approve the plan.  Ever since that time, M.I.T.
has had an ambivalent policy toward the success of Macsyma. (For 
example, around 1982, an M.I.T. professor close to Macsyma told 
Charles Johnson, the owner of IMSL (the leading numerical software 
vendor of the 1970s and 1980s), that Macsyma was obsolete and that 
IMSL should not try to commercialize it.) 

Richard Fateman worked on Macsyma in the early 1970s and took a 
position in the Computer Science department at U.C. Berkeley around 
1976(?). He wanted access to the Macsyma sources to continue his
work but M.I.T. denied the request. This was the opening round
in a series of bitter disputes over Macsyma that involved M.I.T.,
the U.S. Department of Energy, U.C. Berkeley, and the U.S. National
Labs, mostly Los Alamos. At that time in the U.S. there were broader
policy disagreements about whether universities could privatize 
research results obtained on government research contracts. (By the 
end of the 1980s, government policies resolved this issue in the 
affirmative.)

----
1.2 Symbolics, Inc.
----

Symbolics acquired an exclusive license to Macsyma in 1982. The 
Symbolics effort did not go well for three reasons. First, Symbolics 
was so focused on selling Lisp machines that it gave inadequate funding 
to Macsyma and initially impeded the contractually required ports of 
Macsyma to other types of workstations. Secondly, the Macsyma group 
lacked business leadership and had a lot of internal quarrels. Thirdly, 
some government agencies, notably the Department of Energy and the 
national labs, refuse to buy from Symbolics, and began pouring 
significant resources into an alternative government version of Macsyma,
called 'DOE Macsyma.' At this time, Wolfram's first symbolic product, 
SMP, was seriously eating into Macsyma's market, though it was a 
markedly inferior product.

I joined Symbolics in July 1986. Over the next 18 months we greatly
improved the output of the development department, doubled sales, 
improved customer relations (which had been pretty bad), and drove SMP
out of business. However, Symbolics' main business was sinking due
to poor strategy and management problems. So instead of giving Macsyma 
more headcount to continue growing, Symbolics cut the Macsyma 
headcount.

In 1987-88, the Macsyma group tried to build a PC Macsyma with Gold 
Hill Lisp, which proved to be too unstable. (Symbolic had killed its 
Lisp compiler project for standard computers in order to avoid 
competing with Lisp machine sales. They also refused to cooperate 
with Sun to make Lisp available on Sun workstations for the same 
reason.) When Wolfram's Mathematica was released in summer 1988, 
Macsyma could not respond with a PC product because we lacked the 
manpower, and because there were no good Lisp compilers on PCs.  
We responded with a product in August 1989 that lacked notebooks, 
and was very slow numerically. (The Lisp community at Symbolics 
believed that numerical analysis was an old technology that was not 
important for Lisp, so they declined to invest significant resources 
to improve the inadequate speed of numerical computations of their 
Lisps.)

By 1989, it was clear that Symbolics would implode due to poor
strategy and bad management, and that they would take Macsyma
with them. I could not assemble a buy-out team to free Macsyma
from Symbolics due to lack of cooperation from M.I.T.  After 
trying to explain to management how to fix the main Symbolics
business (which got the new president angry), I left Symbolics.

----
1.3 Macsyma Inc.
----

In April 1992 Russell Noftsker and I founded Macsyma Inc. and 
acquired the Macsyma business from Symbolics. Macsyma's market
share in symbolic math software had fallen from 70% in 1987 to
1% in 1992. While the market was growing fast, Macsyma sales
in 1991 and early 1992 were falling.

Macsyma Inc. made enormous improvements in Macsyma, culminating
in Macsyma 2.0.5 in early 1995. 
o On Wester's large test of symbolic math, this product scored 
  10% better than Maple and 15% better than Mathematica. Instead 
  of being very slow, it was on average faster than Mathematica 
  and almost as fast as Maple. 
o Macsyma 2.0.5 had a much better notebook interface than either 
  competitor.
o All reviewers unanimously agreed that it had the best help 
  system in the industry (including hypertext, function templates, 
  and later Mathtips natural language query). 
o Although Macsyma 2.0.5 was still very slow at numerics, it had 
  a greatly strengthened portfolio of numerical analysis and 
  linear algebra routines.  In 1996 we added LAPACK to solve 
  the worst speed problems in numerical linear algebra.

By 1993, market growth had slowed and the market had standardized
on Mathematica and Maple. The competitors had development staffs
that were 4-8 times our size, so it was quite hard to gain the
lead in so many areas at once -- but we did it anyway. 

Our backers were wealthy hobbyists who would not give us adequate 
funds to make up for the past problems. Indeed, when we announced 
Macsyma 2.0 in the fall of 1994, our primary backer cut nearly 
all funding, saying he wanted to see if we could market the 
product on our own, after he had paid to develop it. So we spent 
virtually nothing on marketing during the year after we achieved 
the greatest level of product superiority.


-------------------------------------
2. Search for Life After Macsyma Inc.
-------------------------------------

----
2.1 Foundation
----

In late 1997 I had virtually nailed down an endowment of roughly
$50 million to enable Macsyma to revolutionize the interaction 
of mathematics and software.

o We persuaded our potential benefactor that automated 
  computation was the emerging intellectual technology in 
  mathematics; and that, while axiom-sets-and-mappings mathematics
  is extremely valuable, symbolic math software offers greater 
  opportunities to yield major results over the next few decades.

o Our donor told us he definitely planned to fund this plan.

o We prepared rough programs and budgets, contacted many 
  academics to explore interest, and we explored legal and tax 
  implications of various structures.

In the spring of 1998 a leading pure math department persuaded our 
potential donor (an alum) that the best minds only work on pure 
math.  So the frontier of automated operational mathematics lost 
to the mature activity of supporting bright guys to discover more 
theorems.

I view this result as a victory of tactical competence over 
strategic effectiveness.

----
2.2 FSU
----

In spring 1999 I tried to persuade Florida State to adopt Macsyma
as a powerful addition to their small program in symbolic 
mathematics software.

o The project leader, Mika Seppala, told FSU that supporting 
  Macsyma as an open academic math system with ample resources 
  would eventually make FSU the world center of math software.

o The dean of science agreed. He is charged with long-term 
  development of the school of science.  We agreed that the 
  best way to improve their standing is to beat the established 
  leaders into the emerging growth fields, and this looked like 
  an attractive opportunity.

o The mathematics faculty strongly rejected the proposal. They 
  were unwilling to see resources diverted to something other
  than pure mathematics, since their resource base was already
  tight due to difficulties which FSU, as a second-tier math 
  department, has winning research grants. 

So the initiative collapsed.

----
2.3 Sale of Macsyma Inc.
----

Macsyma was acquired in 1999  by Tenedos LLC, a small holding
company. Tenedos is controlled by an ex-government agent, who 
hoped to get government funds from the defense and intelligence 
agencies to do something with Macsyma. As of 2003, Tenedos has 
not raised funds to to anything with Macsyma, so it lies dormant.


------------
3. Diagnosis
------------

----
3.1 Products and Technology
----

Macsyma's most damaging product problem was slowness in numerical
analysis. Whenever a customer considered Macsyma for adoption in
a major commercial or government or educational program, the
severity of this problem killed Macsyma's chances. This problem 
had two causes.
o Lisp systems had slow numerical analysis. Lisp's support for 
  uniform garbage collection and run-time type checking (and
  possibly other developer-centric features) required software
  indirection that slowed arithmetic and basic array operations. 
  Lisp developers focused on "artificial intelligence" and they 
  considered numerical analysis of little consequence for Lisp.
o Macsyma built all matrices at user level from list structures 
  which are terribly slow for linear algebra. For those matrix
  operations that were performed internally on arrays, the 
  conversion between lists and arrays was itself very slow. 
The slowness of numerical linear algebra was too big a problem 
to tackle, given our tie to Lisp and the seriousness of other 
problems that needed attention after Symbolics milked the product 
two times in the 1980s. So I hoped I could build a product that 
was best at everything and had all the basic numerical analysis 
but was slow at numerics.

----
3.2 Societal Support Structures 
----

The most basic reason long-term for the decline of Macsyma was 
that the academic and government mathematical and scientific 
communities took virtually no interest in symbolic math software. 
The academic mathematical community is trapped in an historical 
lacunae of excessive abstraction that will become a backwater 
within mathematics in the 21st century (or else most of mathematics 
will become a backwater in the sciences).

Philanthropists who might have helped Macsyma did not, for two
reasons. 
o Very little philanthropy goes to the hard sciences today except 
  for life sciences, because these fields cannot make a compelling
  case to philanthropists. I believe symbolic math software should
  be the exception -- it can rejuvenate the crown intellectual
  jewel of science by injecting revolutionary doses of 
  computational power.
o The main philanthropic investor we were fortunate enough to 
  find eventually took the advice of the academic mathematics 
  community, so he funded a mathematics foundation whose programs 
  reflect the strategic failures mentioned above in the academic 
  mathematics community.

----
3.3 Macsyma Team
----

The Macsyma team that started at M.I.T. became riven with factions 
starting in the late 1970s. Not only did this precluded cooperation, 
but it aligned some major institutions against one another, which 
scared off investors who might have tried to get involved. 


> -----Original Message-----
> From: maxima-admin at math 
> [mailto:maxima-admin@math.utexas.edu] On Behalf Of Stavros Macrakis
> Sent: Thursday, October 30, 2003 2:37 PM
> To: 'Richard Fateman'
> Cc: maxima@www.ma.utexas.edu; 'Michael Reimpell'
> Subject: RE: XML, why Mma is popular, was...Re: [Maxima] 
> Lunch with George Carrette
> 
> 
> > The key lesson of Mathematica's success is, I think,
> > disturbing. That is, marketing and glitz was more important 
> > than efficient, reliable and comprehensive mathematical 
> > computation. Technical achievement was not critical since 
> > most came years later.
> 
> I find the lesson instructive, but not disturbing.  It is not 
> enough to have a good (or even the best) product.  You must 
> convince potential users to try it, and then adopt it.  You 
> must make the product not only be good, but appear good.  
> It's probably a good idea to throw away the first draft (SMP 
> was Wolfram's first attempt to improve on Macsyma, 
> Mathematica his second -- was Macsyma rearchitected in the 
> meantime?). You must ship on appropriate platforms.  You 
> bootstrap whatever success you do have into evolving both the 
> product and the marketing.
> 
> I would be fascinated to learn the full story, but I get the 
> impression that Mathematica's success comes as much from 
> Macsyma's marketing errors as from Mathematica's marketing triumphs.
> 
>       -s
> 
> 
> 
> _______________________________________________
> Maxima mailing list
> Maxima@www.math.utexas.edu 
> http://www.math.utexas.edu/mailman/listin>; fo/maxima
> 
>