8.0: After changing innodb_undo_directory, Server Fails to Start: Plugin initialization

APPLIES TO:

MySQL Server - Version 8.0 and later
Information in this document applies to any platform.

SYMPTOMS

Server fails to start, with a vague message in the log:
 

2020-03-08T13:32:53.741553Z 0 [System] [MY-010116] [Server] mysqld (mysqld 8.0.19-commercial) starting as process 31728
2020-03-08T13:32:53.991350Z 1 [ERROR] [MY-012930] [InnoDB] Plugin initialization aborted with error Invalid Filename.
2020-03-08T13:32:54.407234Z 1 [ERROR] [MY-010334] [Server] Failed to initialize DD Storage Engine
2020-03-08T13:32:54.407432Z 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed.
2020-03-08T13:32:54.407547Z 0 [ERROR] [MY-010119] [Server] Aborting

 If you add "log-error-verbosity=3" to my.cnf and try starting it,  we see the root cause of a problem:

2020-03-08T13:36:14.679954Z 1 [Note] [MY-012203] [InnoDB] Directories to scan './;<DIR>'
2020-03-08T13:36:14.680059Z 1 [Note] [MY-012204] [InnoDB] Scanning './'
2020-03-08T13:36:14.680854Z 1 [Note] [MY-012204] [InnoDB] Scanning '<DIR>'
2020-03-08T13:36:14.690814Z 1 [Note] [MY-012208] [InnoDB] Completed space ID check of 6 files.
2020-03-08T13:36:14.691522Z 1 [Note] [MY-012955] [InnoDB] Initializing buffer pool, total size = 128.000000M, instances = 1, chunk size =128.000000M
2020-03-08T13:36:14.698394Z 1 [Note] [MY-012957] [InnoDB] Completed initialization of buffer pool
2020-03-08T13:36:14.699659Z 0 [Note] [MY-011952] [InnoDB] If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2020-03-08T13:36:14.756196Z 1 [Note] [MY-013086] [InnoDB] Starting to parse redo log at lsn = 20157993, whereas checkpoint_lsn = 20158111
2020-03-08T13:36:14.791631Z 1 [Note] [MY-013083] [InnoDB] Log background threads are being started...
2020-03-08T13:36:14.792129Z 1 [Note] [MY-012532] [InnoDB] Applying a batch of 0 redo log records ...
2020-03-08T13:36:14.792249Z 1 [Note] [MY-012535] [InnoDB] Apply batch completed!
2020-03-08T13:36:14.792577Z 1 [Note] [MY-012905] [InnoDB] Cannot create <DIR>/undo_001 because ./undo_001 already uses Space ID=4294967279! Did you change innodb_undo_directory?
2020-03-08T13:36:14.792714Z 1 [ERROR] [MY-012930] [InnoDB] Plugin initialization aborted with error Invalid Filename.
2020-03-08T13:36:15.199887Z 1 [ERROR] [MY-010334] [Server] Failed to initialize DD Storage Engine
2020-03-08T13:36:15.200363Z 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed.
2020-03-08T13:36:15.200657Z 0 [ERROR] [MY-010119] [Server] Aborting
2020-03-08T13:36:15.200699Z 0 [Note] [MY-010120] [Server] Binlog end
2020-03-08T13:36:15.200855Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'MyISAM'
2020-03-08T13:36:15.200906Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'InnoDB'
2020-03-08T13:36:15.200962Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'CSV'

CHANGES

 The my.cnf value of innodb_undo_directory was set, or changed.    

CAUSE

The existing undo_001 and undo_002 files are still in the original location (data directory).   These two files have to reside in the innodb_undo_directory location (or if that's not set,  in the datadir).

The manual page at

MySQL :: MySQL 8.0 Reference Manual :: 15.6.3.4 Undo Tablespaces

Says

The default undo tablespaces (innodb_undo_001 and innodb_undo_002) created when the MySQL instance is initialized must always reside in the directory defined by the innodb_undo_directory variable. If the innodb_undo_directory variable is undefined, default undo tablespaces reside in the data directory. If default undo tablespaces are moved while the server is offline, the server must be started with the innodb_undo_directory variable configured to the new directory.

SOLUTION

Either move the undo_001 and undo_002 from their existing location to the location specified by innodb_undo_directory,   or set the innodb_undo_directory to point to where the undo_001 and undo_002 are located.

If you move the existing undo_001 and undo_002 into the new location specified by innodb_undo_directory,  then server starts up:

2020-03-08T13:38:28.907401Z 1 [Note] [MY-012532] [InnoDB] Applying a batch of 0 redo log records ...
2020-03-08T13:38:28.907525Z 1 [Note] [MY-012535] [InnoDB] Apply batch completed!
2020-03-08T13:38:28.907744Z 1 [Note] [MY-013252] [InnoDB] Using undo tablespace '<DIR>/undo_001'.
2020-03-08T13:38:28.911700Z 1 [Note] [MY-013252] [InnoDB] Using undo tablespace '<DIR>/undo_002'.
2020-03-08T13:38:28.917928Z 1 [Note] [MY-012910] [InnoDB] Opened 2 existing undo tablespaces.
2020-03-08T13:38:28.918153Z 1 [Note] [MY-011980] [InnoDB] GTID recovery trx_no: 3087
2020-03-08T13:38:29.022061Z 1 [Note] [MY-012923] [InnoDB] Creating shared tablespace for temporary tables
2020-03-08T13:38:29.022561Z 1 [Note] [MY-012265] [InnoDB] Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2020-03-08T13:38:29.919556Z 1 [Note] [MY-012266] [InnoDB] File './ibtmp1' size is now 12 MB.
2020-03-08T13:38:29.924187Z 1 [Note] [MY-011825] [InnoDB] Scanning temp tablespace dir:'./#innodb_temp/'
2020-03-08T13:38:30.463870Z 1 [Note] [MY-013018] [InnoDB] Created 128 and tracked 128 new rollback segment(s) in the temporary tablespace. 128 are now active.
2020-03-08T13:38:30.464247Z 1 [Note] [MY-012976] [InnoDB] 8.0.19 started; log sequence number 20158111
2020-03-08T13:38:30.501572Z 1 [Note] [MY-011089] [Server] Data dictionary restarting version '80017'.
2020-03-08T13:38:30.641153Z 1 [Note] [MY-012357] [InnoDB] Reading DD tablespace files
2020-03-08T13:38:30.642077Z 1 [Note] [MY-013516] [InnoDB] DD ID: 4 - Tablespace 4294967279, name 'innodb_undo_001', './undo_001' is moved to '<DIR>/undo_001'
2020-03-08T13:38:30.642238Z 1 [Note] [MY-013516] [InnoDB] DD ID: 5 - Tablespace 4294967278, name 'innodb_undo_002', './undo_002' is moved to '<DIR>/undo_002'
2020-03-08T13:38:30.642579Z 1 [Note] [MY-012356] [InnoDB] Validated 8/8 tablespaces
2020-03-08T13:38:30.646715Z 1 [Note] [MY-012173] [InnoDB] Committed : 2
2020-03-08T13:38:30.739532Z 1 [Note] [MY-010006] [Server] Using data dictionary with version '80017'.
2020-03-08T13:38:30.780768Z 0 [Note] [MY-010902] [Server] Thread priority attribute setting in Resource Group SQL shall be ignored due to unsupported platform or insufficient privilege.
2020-03-08T13:38:30.848592Z 0 [Note] [MY-012487] [InnoDB] DDL log recovery : begin
2020-03-08T13:38:30.848804Z 0 [Note] [MY-012488] [InnoDB] DDL log recovery : end
2020-03-08T13:38:30.849083Z 0 [Note] [MY-011946] [InnoDB] Loading buffer pool(s) from <PATH>/ib_buffer_pool
2020-03-08T13:38:30.855588Z 0 [Note] [MY-012922] [InnoDB] Waiting for purge to start
2020-03-08T13:38:30.890964Z 0 [Note] [MY-011946] [InnoDB] Buffer pool(s) load completed at 200308 15:38:30

There's a feature request filed to make this problem easier to identify in future.

Bug 31005361 - TOO VAGUE: PLUGIN INITIALIZATION ABORTED WITH ERROR INVALID FILENAME