cache line size wrongly detected on virtual instance

[Antoine Musso]

I found out nginx is reporting the wrong cache line size on a virtual system.

On startup it reports:

Restarting nginx: nginx: [emerg] could not build the types_hash, you should
increase either types_hash_max_size: 1024 or types_hash_bucket_size: 32

This is caused by /etc/nginx/mime.types having a key which is 36 bytes long:

echo -n application/vnd.google-earth.kml+xml |wc -c
36

In src/core/ngx_cpuinfo.c , the ngx_cpuinfo() reports my CPU has having a 32 bytes cache line which is then used as the default value for types_hash_bucket_size and thus trigger the startup error.

Looking at the ngx_cpuinfo() code, the values are hardcoded. For GenuineIntel the values yields are:

family 5 -> 32
family 6 -> 32 or 64 depending on model
family 15 -> 128

My virtual machine is based on QEMU which reports a family of 6, a model of 2 but a cache_alignement of 64 :

$ cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 2
model name : QEMU Virtual CPU version 1.0
stepping : 3
microcode : 0x1
cpu MHz : 2659.998
cache size : 4096 KB
fpu : yes
fpu_exception : yes
cpuid level : 4
wp : yes
flags : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm up rep_good nopl pni vmx cx16 popcnt hypervisor lahf_lm
bogomips : 5319.99
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:

I do not have that issue on a real server which reports:

processor : 7
vendor_id : GenuineIntel
cpu family : 6
model : 30
model name : Intel(R) Xeon(R) CPU X3450 @ 2.67GHz
stepping : 5
microcode : 0x4
cpu MHz : 2659.886
cache size : 8192 KB
physical id : 0
siblings : 8
core id : 3
cpu cores : 4
apicid : 7
initial apicid : 7
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips : 5319.72
clflush size : 64
cache_alignment : 64
address sizes : 36 bits physical, 48 bits virtual
power management:

The workaround is to hardcode types_hash_bucket_size to 64.

It would be nice to enhance the detection system to rely on the cache_alignement has reported by cpuinfo on linux.

The ngx_cpuinfo() seems to have the same issue in nginx 1.4.1 and 1.5.0.

[Antoine Musso]

Our bug is ​https://bugzilla.wikimedia.org/show_bug.cgi?id=48210

[Maxim Dounin]

Missing semicolon in mime.types? In mime types hash keys are extensions, not type names, and 32 is enough for default mime.types as shipped with nginx.

As for "incorrect" detection of cache line size on fake emulated CPU - we don't really care, as optimal cache line size isn't know in this case anyway, any number will do.

[Antoine Musso]

mime.types being used

[Antoine Musso]

diff of 1.1.19 mime.types file and the one from debian

[Antoine Musso]

I have attached the mime.types we are using which comes from the Ubuntu package. It has a few differences with the one provided in Nginx 1.1.19.

I also provided a unified diff (ignoring whitespaces) between the mime.types from 1.1.19 and Debian.

[Maxim Dounin]

Ok, I see why mime.types in question doesn't load. Here what goes on:

1. On 64-bit platforms each hash element is at least 16 bytes long (2 pointers), and 32-byte hash bucket can only hold one hash element (only 24 bytes are usable for elements as 8 bytes are per-bucket technical data).
2. The mime.types in question have collisions up to 1044 buckets, but this more than allowed default (types_hash_max_size defaults to 1024). Hence the startup failure.

I don't think this is a bug, and tuning types_hash_max_size/types_hash_bucket_size as nginx suggests is a right solution. On the other hand, we may want to do something to make such failures less likely. Keeping this open for now.

[Antoine Musso]

I guess I will report a bug to Ubuntu/Debian so they raise the type_hash_maxi_size according to the mime.types they provide.

I am wondering how you determine the 1044 buckets number? That could be used in the Debian package to ensure the default configuration as a sane value.

Finally, would it make sense to adapt the hash size automatically ?

[Maxim Dounin]

The 1044 number was obtained via unifdef'ing appropriate logging in ngx_hash.c:ngx_hash_init() and running with larger types_hash_max_size. Not something usable for automatic testing.

As for automatic hash size adoption - it's already there, but the default limit (types_hash_max_size 1024) isn't enough for a particular mime.types. I think of setting types_hash_bucket_size to min(64, cacheline) or something similar to always allow at least 2 elements per hash bucket.

[Antoine Musso]

Would it make sense to request the Debian package to set a default types_hash_max_size of 2048 ?

[Eric Bréchemier]

I ​noticed the same issue with server_names_hash_bucket_size and long domains names in server_name directive, on a VPS from digitalocean.com.

​/proc/cpuinfo shows:

processor : 0 
vendor_id : GenuineIntel
cpu family : 6
model : 2
model name : QEMU Virtual CPU version 1.0 
(...)
clflush size : 64
cache_alignment : 64 
(...)

[Maxim Dounin <mdounin@…>]

In 5a65b9d8bc2bc6d104a17ba0460c2855398ab9c2/nginx:

Adjusted default value of types_hash_bucket_size (ticket #352).

The ngx_cacheline_size may be too low on some platforms, resulting
in unexpected hash build problems (as no collisions are tolerated due
to low bucket_size, and max_size isn't big enough to build a hash without
collisions). These problems aren't fatal anymore but nevertheless
need to be addressed.

[Maxim Dounin]

With c348dea081fb it is no longer fatal if an optimal hash can't be built due to collisions (now it results only in a warning, with bucket_size ignored), and 5a65b9d8bc2b adjusts types_hash_bucket_size to be 64 even on platforms with 32-bit cache line size. These two changes ensure that types_hash can be built using defaults regardless of a platform used, and more or less independently of a mime types list.