You're both almost right...
When you delete a file, and it creates a hole, the hole isn't just left there. It could be filled with part of a file. Here's an example.
You have three files named A, B, and C, and they're organized on your disk like this:
You then delete B so you have this:
But then you created D, which is 5 clusters in size, and you end up with this:
Then you edit C and make it larger, so you end up with this:
So now when you want to open D, you have to skip across C. After months of saving, deleting, creating, changing, deleting, saving, etc, you end up with one fragmented disk!
Defragmenting your disk doesn't increase speed as much as is thought. Fragmentation only affects files that are not accessed using the index method. So the most used files on your disk (virtual memory for example) are not affected because it doesn't read the file sequentially, but randomly (like your ram.. random access memory, only it's a file). If you edit a lot of video, or deal with very large files that you created, then you should definitelly defrag.
BUT, be warned, defragging involves taking data off your disk, putting it into memory, then writing it back to the disk. So there is definitelly the chance of losing data. Let's say the defrag has just moved a file into memory, then your power goes out. That data is most likely gone forever. So be sure to do a backup of important files before you defragment!