You could approach this 'syntactically', that is, pull out the corresponding
coefficients and solving (as laid out by Leo), or you could approach it
'semantically', as follows:
(%i1) solve(x*y=a*(x+y)^2+b*(x-y)^2,[a,b,x,y]);
(%o1) [[a =
-(%r17*%r19^2+(-2*%r17-1)*%r18*%r19+%r17*%r18^2)/(%r19^2+2*%r18*%r19+%r18^2),
b = %r17,x = %r18,y = %r19]]
(%i2) aval: part(%o1,1,1,2);
(%o2)
-(%r17*%r19^2+(-2*%r17-1)*%r18*%r19+%r17*%r18^2)/(%r19^2+2*%r18*%r19+%r18^2)
(%i3) daval: diff(aval,%r18,1,%r19,1); /* d(solution)/dxdy
(%o3)
2*(%r17*%r19^2+(-2*%r17-1)*%r18*%r19+%r17*%r18^2)/(%r19^2+2*%r18*%r19+%r18^2)^2
-2*(2*%r19+2*%r18)^2*(%r17*%r19^2+(-2*%r17-1)*%r18*%r19+%r17*%r18^2)
/(%r19^2+2*%r18*%r19+%r18^2)^3-(-2*%r17-1)/(%r19^2+2*%r18*%r19+%r18^2)
+(2*%r19+2*%r18)*(2*%r17*%r19+(-2*%r17-1)*%r18)/(%r19^2+2*%r18*%r19+%r18^2)^2
+(2*%r19+2*%r18)*((-2*%r17-1)*%r19+2*%r17*%r18)/(%r19^2+2*%r18*%r19+%r18^2)^2
(%i4) solve(%,%r17); /* for what value of b is it 0? */
(%o4) [%r17 = -1/4] /* result does not depend on x and y, ok */
2009/11/8 Ulf Ekstr?m <uekstrom at gmail.com>
> solve(x*y=a*(x+y)^2+b*(x-y)^2,[a,b]);
>