Both. How quickly a server can send a webpage with images (even if they're small) is directly proportional to the storage mediums seeks times. The worse the seek times, the less 'responsive' a website feels. Hard drives are a terrible location to keep your metadata.
The server scan will search for the files, look them up and grab metadata, and then store that metadata in the metadata location. If your metadata location is the same spot as your movie, it will cause some major thrashing, and will significantly increase the scan time for jellyfin. Essentially, it gets bogged down trying to read and write lots of tiny files on the same drive, the absolute worst case scenario for a hard drive to have.
If the movies are on a hard drive, and the metadata on an ssd (or even just a different hard drive) the pipeline will be a lot less problematic.