README.md
1# `parse_number_fxx` Test Data
2
3This repository contains test data for `parse_number_fxx` implementations (for
4`fxx` being `f16`, `f32` or `f64`), also known as `StringToDouble`, `strtod`,
5`atof`, etc. These convert from an ASCII string to a 16-, 32- or 64-bit value
6(IEEE 754 half-, single- or double-precision floating point).
7
8Most of the `data/*.txt` files were derived by running
9`script/extract-numbery-strings.go` on various repositories or zip files,
10listed further below. Their contents look like:
11
12 3C00 3F800000 3FF0000000000000 1
13 3D00 3FA00000 3FF4000000000000 1.25
14 3D9A 3FB33333 3FF6666666666666 1.4
15 57B7 42F6E979 405EDD2F1A9FBE77 123.456
16 622A 44454000 4088A80000000000 789
17 7C00 7F800000 7FF0000000000000 123.456e789
18
19For example, parsing `"1.4"` as a `float32` gives the bits `0x3FB33333`.
20
21In this case, the final line's `float16`, `float32` and `float64` values are
22all infinity. The largest finite `float{16,32,64}` values are approximately
23`6.55e+4`, `3.40e+38` and `1.80e+308`.
24
25For each line of these `data/*.txt` files, the `f16`, `f32` and `f64`
26hexadecimal digits and the ASCII string subslices are:
27
28- When column indexes start at 0: `[0..4]`, `[5..13]`, `[14..30]` and `[31..]`.
29- When column indexes start at 1: `[1..5]`, `[6..14]`, `[15..31]` and `[32..]`.
30
31The first half (the high 16 bits) of the `f32` hexadecimal digits are also
32known as the `bfloat16` format.
33
34
35## Data
36
37In the `data` directory:
38
39- `freetype-2-7.txt` was extracted from [Freetype](https://www.freetype.org/)
40 2.7
41- `google-double-conversion.txt` was extracted from
42 [google/double-conversion](https://github.com/google/double-conversion)
43- `google-wuffs.txt` was extracted from
44 [google/wuffs](https://github.com/google/wuffs)
45- `ibm-fpgen.txt` was extracted from IBM's
46 [IEEE 754R test suite](https://www.research.ibm.com/haifa/projects/verification/fpgen/test_suite_download.shtml)
47- `lemire-fast-double-parser.txt` was extracted from
48 [lemire/fast\_double\_parser](https://github.com/lemire/fast_double_parser)
49- `lemire-fast-float.txt` was extracted from
50 [lemire/fast\_float](https://github.com/lemire/fast_float)
51- `more-test-cases.txt` was extracted from this repository's manually curated
52 collection of [more test cases](./more-test-cases)
53- `tencent-rapidjson.txt` was extracted from
54 [Tencent/rapidjson](https://github.com/Tencent/rapidjson)
55- `ulfjack-ryu.txt` was extracted from
56 [ulfjack/ryu](https://github.com/ulfjack/ryu)
57
58
59### remyoudompheng/fptest
60
61The `data/remyoudompheng-fptest-?.txt` files were created by running `go test
62-test.run=TestTortureAtof64` in the
63[remyoudompheng/fptest](https://github.com/remyoudompheng/fptest) repository
64(with the following patch), running the resultant `TestTortureAtof64.txt` file
65through `script/extract-numbery-strings.go` and then using `sed` to split what
66would be a 189 MiB file into multiple (million line) files:
67
68```diff
69diff --git a/torture_test.go b/torture_test.go
70index 87ba7e7..59887ff 100644
71--- a/torture_test.go
72+++ b/torture_test.go
73@@ -1,8 +1,11 @@
74 package fptest
75
76 import (
77+ "bufio"
78 "bytes"
79+ "fmt"
80 "math"
81+ "os"
82 "strconv"
83 "testing"
84
85@@ -124,6 +127,11 @@ func TestTortureShortest32(t *testing.T) {
86 }
87
88 func TestTortureAtof64(t *testing.T) {
89+ tmpFile, _ := os.Create("/tmp/TestTortureAtof64.txt")
90+ defer tmpFile.Close()
91+ tmpWriter := bufio.NewWriter(tmpFile)
92+ defer tmpWriter.Flush()
93+
94 count := 0
95 buf := make([]byte, 64)
96 roundUp := false
97@@ -140,6 +148,7 @@ func TestTortureAtof64(t *testing.T) {
98 t.Errorf("could not parse %q: %s", s, err)
99 return
100 }
101+ fmt.Fprintf(tmpWriter, "%s\n", s)
102 expect := x
103 if roundUp {
104 expect = y
105```
106
107
108## Users
109
110Programs that use this test data set:
111
112- `script/manual-test-parse-number-f64.cc` in
113 [google/wuffs](https://github.com/google/wuffs)
114