There is a specification for graphics and for plotting in Mathematica.
My understanding is that the plotting program produces a graphics object
(based on the function(s) being plotted as well as parameters,) and the
display program produces a pretty picture (based on the graphics object as
well as additional parameters).
Looking at Mathematica's design might be helpful, seeing as how you seem
to be coming up with something similar, anyway. Maybe you can avoid some
pitfalls.
RJF