I’m having terrible performance when building machines that use MDADM in some cases. The symptom being the deployment takes FOREVER, on the order of hours for a single machine.
When logging into a deploying machine and running cat /proc/mdstat I get this:
ubuntu@server:~$ cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md1 : active raid1 sdb3[1] sda2[0]
5843968 blocks super 1.2 [2/2] [UU]
resync=DELAYED
md0 : active raid1 sdb2[1] sda1[0]
244059136 blocks super 1.2 [2/2] [UU]
[==>…] resync = 14.1% (34638208/244059136) finish=17.4min speed=200064K/sec
bitmap: 2/2 pages [8KB], 65536KB chunk
md2 : active raid10 sdn[11] sdm[10] sdl[9] sdk[8] sdj[7] sdi[6] sdh[5] sdg[4] sdf[3] sde[2] sdd[1] sdc[0]
5625360384 blocks super 1.2 512K chunks 2 near-copies [12/12] [UUUUUUUUUUUU]
[>…] resync = 0.0% (5435712/5625360384) finish=3157261.0min speed=29K/sec
bitmap: 42/42 pages [168KB], 65536KB chunk
unused devices:
Notice the build speed on md2, 29K/sec what the heck is going on?
Ephemeral kernel:
uname -a
Linux use1-oscomp-76 5.4.0-52-generic #57-Ubuntu SMP Thu Oct 15 10:57:00 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
The curtin config for the host related to storage is:
storage:
config:
- grub_device: true
id: sda
model: KINGSTON_SKC6002
name: sda
ptable: msdos
serial: REDACTED
type: disk
wipe: superblock - grub_device: true
id: sdb
model: KINGSTON_SKC6002
name: sdb
ptable: gpt
serial: REDACTED
type: disk
wipe: superblock - device: sdb
flag: bios_grub
id: sdb-part1
number: 1
offset: 4194304B
size: 1048576B
type: partition
wipe: zero - id: sdc
model: Micron_5200_MTFD
name: sdc
serial: REDACTED
type: disk
wipe: superblock - id: sdd
model: Micron_5200_MTFD
name: sdd
serial: REDACTED
type: disk
wipe: superblock - id: sde
model: Micron_5200_MTFD
name: sde
serial: REDACTED
type: disk
wipe: superblock - id: sdf
model: Micron_5200_MTFD
name: sdf
serial: REDACTED
type: disk
wipe: superblock - id: sdg
model: Micron_5200_MTFD
name: sdg
serial: REDACTED
type: disk
wipe: superblock - id: sdh
model: Micron_5200_MTFD
name: sdh
serial: REDACTED
type: disk
wipe: superblock - id: sdi
model: Micron_5200_MTFD
name: sdi
serial: REDACTED
type: disk
wipe: superblock - id: sdj
model: Micron_5200_MTFD
name: sdj
serial: REDACTED
type: disk
wipe: superblock - id: sdk
model: Micron_5200_MTFD
name: sdk
serial: REDACTED
type: disk
wipe: superblock - id: sdl
model: Micron_5200_MTFD
name: sdl
serial: REDACTED
type: disk
wipe: superblock - id: sdm
model: Micron_5200_MTFD
name: sdm
serial: REDACTED
type: disk
wipe: superblock - id: sdn
model: Micron_5200_MTFD
name: sdn
serial: REDACTED
type: disk
wipe: superblock - devices:
- sdc
- sdd
- sde
- sdf
- sdg
- sdh
- sdi
- sdj
- sdk
- sdl
- sdm
- sdn
id: md2
name: md2
ptable: gpt
raidlevel: 10
spare_devices: []
type: raid
- device: sda
id: sda-part1
name: sda-part1
number: 1
size: 250051821568B
type: partition
uuid: d616f9ef-e3a5-41c8-a1d5-c3f08312c8a6
wipe: superblock - device: sda
id: sda-part2
name: sda-part2
number: 2
size: 5989466112B
type: partition
uuid: bd14c40e-1bd0-4599-b14d-69573f6ded62
wipe: superblock - device: sdb
id: sdb-part2
name: sdb-part2
number: 2
size: 250051821568B
type: partition
uuid: fd3fff0a-ce4a-44d7-81f2-d2a2d884d64a
wipe: superblock - devices:
- sda-part1
- sdb-part2
id: md0
name: md0
raidlevel: 1
spare_devices: []
type: raid
- device: sdb
id: sdb-part3
name: sdb-part3
number: 3
size: 5989466112B
type: partition
uuid: 555d06e8-64c6-404b-a808-a2acd220c828
wipe: superblock - devices:
- sda-part2
- sdb-part3
id: md1
name: md1
raidlevel: 1
spare_devices: []
type: raid
- device: md2
id: md2-part1
name: md2-part1
number: 1
offset: 4194304B
size: 5759970574336B
type: partition
uuid: a6b4001c-f7b5-4e3a-af66-afb3957d5b7f
wipe: superblock - devices:
- md2-part1
id: raid10_vg
name: raid10_vg
type: lvm_volgroup
uuid: 2740a009-e529-435b-8d03-31f1f5514a13
- md2-part1
- id: raid10_vg-nova
name: nova
size: 5470823645184B
type: lvm_partition
volgroup: raid10_vg - fstype: ext4
id: md0_format
label: ‘’
type: format
uuid: 21abd547-3d0c-4fab-84b8-f9621781ab06
volume: md0 - fstype: swap
id: md1_format
label: ‘’
type: format
uuid: 7919b895-e94e-4e3b-b4f2-d9f3d62ebf35
volume: md1 - fstype: xfs
id: raid10_vg-nova_format
label: ‘’
type: format
uuid: 4dd85880-6f1b-4b8a-8fa2-9f3944efca43
volume: raid10_vg-nova - device: md0_format
id: md0_mount
options: ‘’
path: /
type: mount - device: raid10_vg-nova_format
id: raid10_vg-nova_mount
options: ‘’
path: /var/lib/nova
type: mount - device: md1_format
id: md1_mount
options: ‘’
type: mount
version: 1
swap:
size: 0B