He realizes that while he still remembers the outline of thing that happens in the Six Trials he also notices that he does remember the details
It is like the memory about the Six Lives in the Six Sabers is slowly being erased as he grows in power.
He remembers Azul and he remember the trials but he didn't remember the process.
Azief could understand why. After all, that life is not really his life and those memories is not really his memories.
As such, it will naturally fade leaving him only the memory and the result of his act. But just because he didn't remember every details don't mean he doesn't remember anything
Though, he also suspected that this loss of memory is also another gift from Azul. It might seem contradictory. How could him forgetting about his experiences in the Six Trials help him?