Vanguard Software
Forecasting & Planning Solutions
Home Products Solutions Services Customers Partners Company
 
Applying science to business management

contents.gifindex.gifprev1.gifnext1.gif

precision

Format: precision( digits )

Arguments: (int) digits Optional number of significant digits (1-14)

Returns: (int) Current setting for precision

Description: Precision sets the maximum error tolerated in approximated primitives integral, root, and minimize. The time required for these primitives to converge is highly dependent on the desired precision. If no argument is supplied, precision returns the current setting without changing it.

Precision is expressed as the number of significant digits. That is, the number of digits that contain meaningful information. For example, if the precision is set to three and an approximated primitive is solving for PI (3.141592...), the result will be 3.14--the last digit will never be off by more than +/-1.

All of the following numbers are expressed to three significant digits: 12.3, 0.000123, 1.23e-12, 123000. Note that the number of significant digits is different from the number of decimal places displayed.

Examples: precision(3) = causes subsequent integral, root, and minimize operations to be carried out to three significant digits

precision = current setting for precision

See Also: integral, root, minimize

  Demand Planning  |  Financial Forecasting
Forecasting Software  |  Strategic Planning Software  |  Decision Tree Software
Decision Support Software  |  Expert System Software  |  Online Surveys

Contact Us  |  Terms of Use  |  Privacy Policy  |  Site Map
Vanguard Software
© 2011 Vanguard Software Corporation
Print this page   Email this page to a friend