Data is corrupted in static large objects (SLO swift API) when using nginx as a GET cache.
|Reported by:||Owned by:|
|uname -a:||Linux host 3.10.0-862.9.1.el7.x86_64 #1 SMP Mon Jul 16 16:29:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux|
nginx version: nginx/1.14.0
built by gcc 6.2.1 20160916 (Red Hat 6.2.1-3) (GCC)
Compiled by yourself with ngx_http_slice module
We found that data is corrupted in static large objects (SLO swift API) when using nginx as a GET cache.
Nginx used ngx_http_slice plugin.
Nginx download file as a slices using the "Content-Range" header.
At first it seemed to be missing a few bytes at the end of the full object.
Their number was one less than the number of slices for which the file was downloaded.
But the size of the final file was correct!
The analysis showed that in each slice all the headers are correct, but at the end of the data one extra byte!
It is clear that initially this is a ceph bug and we would not have noticed it if nginx compared the header and the number of received bytes.
Apparently nginx adds slices to the buffer without checking the match of the header and the data actually received,and the last slice either does not fit completely into the buffer or is cut off.
Bug in ceph tracer https://tracker.ceph.com/issues/37765