Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

too low precision for double data points #395

Open
2 of 8 tasks
DNKpp opened this issue Feb 21, 2024 · 0 comments
Open
2 of 8 tasks

too low precision for double data points #395

DNKpp opened this issue Feb 21, 2024 · 0 comments

Comments

@DNKpp
Copy link

DNKpp commented Feb 21, 2024

Bug category

  • bug - compilation error
  • bug - compilation warning
  • bug - runtime error
  • bug - runtime warning
  • bug - logic error

Describe the bug

The precision used for converting the double data values from binary to textual representation seems to unreasonable low. That's at least the case in the line::data_string function, where it is constantly set to 10 (see line.cpp#L166).
This leads to data points not showing up in plots with logarithmic axes scaling.

Locally I bumped this straight to 16 (std::numeric_limits<double>::digits10yields 15),, which works as intended (at least for me).

Steps to Reproduce
Well, create a loglog plot with very small double data values.

Output

imprecise output

image
That's part of the cmd dump, sent to gnuplot
image

fixed output

image
That's part of the cmd dump, sent to gnuplot
image

Platform

  • cross-platform issue - linux
  • cross-platform issue - windows
  • cross-platform issue - macos

Environment Details:

  • OS: Windows
  • OS Version: 11
  • Compiler: msvc
  • Compiler version: 17.9.0

Additional context

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant