"The process to upgrade a VMFS-3 volume to VMFS-5 is very straightforward. Should the VFMS-3 volume be a SAN volume and shared to many hosts, it is critical that the other hosts are updated to ESXi 5, as it is not readable by ESXi 4.x hosts or earlier. VMFS-3 was pretty slick in that it is forward and backward compatible, but VMFS-5 requires ESXi 5 or higher.
Within the host configuration of the vSphere Client, browse to the storage configuration section. There, the Upgrade To VMFS-5 link is available for the datastore. "