OD&D gets flak about how hard it was to piece the game rules together from the three booklets. Some have commented that Gary was objectively horrible at writing rules. I think that is total bunk. Yes, OD&D could have been clearer and better organized, but consider that this is the FIRST role playing system ever published. Even today, D&D and many others follow the broad grouping of material.
- Alignment. This has nothing to do with psychological foundations of behaviour, nor with the complexity of needs and motivations that drives people to their actions. It's frankly so dumb, Gary had to spend lots and lots of effort in trying to defend it, and considered it a mistake himself to have published it as it was. It is interesting that currently there seems to be a trend to even remove alignment as a given from "evil" races like Orcs, to make the game more inclusive and tolerant. Exabercating the problems with alignment was that he based a lot of spells and abilities on alignment, making it mechanically relevant and hard to ignore.
- Human-focus with restrictions on classes and maximum level for demihumans. He believed that in a world where demihumans were not so restricted, it would be hard to justify human dominance, and he felt unable to believably describe non-human societies. But elven kindoms abound, from Dragonlance to the Forgotten Realms, as do Dwaven kindoms under the mountains, in lots of campaingns, and nobody does care. He even had humanoid empires of hobgoblins and orcs himself in his World of Greyhawk. The average gamer does not have the same respect for the verisimilitude of the milieu that military history and wargame Gary had. People love to play races that make them feel "special", in 5e Tieflings are one of the most popular choices of all, and the number of renegade good drow played to the image of Drizz't by angst-ridden teenager is legion. You also easily can find reasons to justify human dominance if you want to, for example just assume humans breed much faster than the other races. On a mechanical level, you just need to give humans more flexibilty or some other compensation to make them mechanically as attractive as demihumans, be it skills or feats, as was done in 5e.
- Particularism. He wanted different classes and races to feel different. So demihumans could not take just any class. Humans could not dual class. Why? Players love to construct their characters and have "unusual" combinations that belie the archetype and make their characters (and thus, them) feel more special. And if you do want to play the archetypical gruff dwaven fighter, you still can, and this can be encouraged by pre-made example characters. 3e showed that there is no harm to allow any combination of class and race, and no harm to allow anyone to take levels in any class to create dual or even triple or quadruple class characters -- you paid the price of going broad by not getting access to the higher class-level related abilities and feats that you got from staying focused. (The only problem was that many classes were front loaded with a lot of key abilities on level one, but once you understand that, you can address it in class design). Getting rid of these restrictions also simplified things.
- Different experience level progression ladders by class, and in OD&D even different listed top levels by class. Unnecessarily complicated. Even if wizards were very weak in the beginning, and uber-powerful towards the end, in keeping with Conan and Dying Earth, what was the point? It may have been because he allocated XP to wizards for spells cast, and with more and more slots, the number of XP gained that way per day would grow non-linearly, while fighters had no such method. But that could have been simply fixed by not allocating XP in such a fine grained manner, just give group based XP as is done today. Again, I think his repect for the "reality" of the game world got the better of him here.
- No core mechanic. For the reasons above, every system used differnt dice, tables, and resolution, sometimes you have to roll over or high, sometimes under or low to get a good outcome, sometimes percentile dice are used, sometimes d20, sometimes d6, d12, 2d6, 2d4 etc. 3rd Edition showed how this can all be cleaned up with a core mechanic of d20 rolls over target, high rolls being good.
- Attributes. While his selection was a good as any, or maybe better than many, he made no good use of these attribute values initially. Also it again made no sense why only fighters would get a given benefit from say, high strength - even if you could justify it, it made things unnecessarily complicated, as 3e who treated all the same would show. In particular the ridiculous Stength 18/00 for fighters. I get it that he wanted to strenghten fighters so they would hit better and deal more damage and be stronger than other humans that lacked training. But why not just give fighters a +1 bonus then on Strength (or something like that), even if that brought them out of the normal range of untrained humans? Also, each attribute gave benefits at different cutoffs. Again, 3e cleaned all that up with standard boni and mali for a given value, no matter what attribute.
- Saving throws by souce class with own table. Firstly, this did not extend well as new monsters and effects were introduced, but that is a minor quibble, you can state that the slime lord's touch that liquefies flesh needs a save against petrifiction as easily as you can say it needs a Con save. However, this again is just unnecessary extra ballast and yet another table to look up. It worked great for other systems like DSA or 5e to just base the saves on the character attributes, and if you wanted class or race to count, you could give certain saves a bonus based on those.
- Rules that are nonsensical by in-world logic just to make life difficult for PCs. This includes rules that treat monsters differently from PCs:
- Dungeon doors, that never are stuck for monsters, but by default are stuck for PCs and need to be forced open. There is absolutely no logic in this -- maybe this could happen for a single door, that needs a certain "knack" (slight lifting or such) to be openend, but certainly not all of them.
- Monsters being able to see in the dark, unless they serve a PC. Again, there is no logic at all in this. Either the creature does have "infravision" as darkvision was called back then, or not. Both of these make no sense in in-game logic and are purely "gamey" treatments to give an advantage to monsters over PCs that disrupts the suspension of disbelief.
- Scrolls "vanishing" over time if they are not cursed and not read, to ensure scrolls are read and cursed scrolls get triggered.
- Low-Level deadliness. Gaining experience and improving your character towards the next level was a core driver that made the game addictive, and likely a great part of its success. It still is used today by games everywhere, wether on computers or in paper RPGs, so it is hard to call levels per se a design mistake. However the vunerability of first level characters was too high. You could die from a single hit. Sure, it was not as time consuming back then to make up a new character. Still it is a bad beginner experience to create a character just to see him being wasted by an unlucky roll. Gary in later years started characters on level 2 or even level 3 to adjust for this. DSA started them with 20 or 30 hit points, at similar damage dice.
- Classes. While this makes it simple to play a given archetype, non-cookie cutter character concepts are either impossible to represent with such a system, or unnaturally complicated to approximate by dual- or multiclassing, which also was not generally allowed in original D&D and AD&D. He himself admitted, that an approach based on bundles of abilities would be a better system. This is not so much an issue when you play this the first time, but it will become an issue once you want to be a bit more creative about your character.
- Lack of skills. You can get around this by assuming certain classes have certain skills or abilities, and roll against the most relevant character attribute, as he did in his home game, but this was nowhere formalized part of the rules. He later admitted that using skills, and defining archetypes by skills was a better approach to character concepts.
- Save or die rolls. I'm not sure on these any more. I used to hate them, it always felt unfair that your character carefully played over months and years should perish to a random unlucky roll. Now I think these capture some of the excitement and danger by stripping away the saftey padding hp provide at higher levels, and characters should have access to magic like raise dead, cure poison, stone to flesh, or gold to pay for them. In 5e, poison and petrification monsters lose a lot of their terror and become just another opponent.
- Level Drain. I loved that as a GM, and players obviously hated it. I think the dread of losing levels and permanently setting the character back without it outright, was a great way to make greater undead terrifying. And in the very simple OD&D, losing a level was not that hard to note on your sheet. The problem was that administrative overhead of this, once the rules got more complex in AD&D and you had to undo all the additional abilities you gained with levels. There are complications -- how many hp would you lose? Did you need to record how many you last gained, or just roll again? In later editions, less onerous approaches like lowering the hit point maximum or attributes were taken, which were also less terrifying, because they did not lower your overall fighting ability -- a mage with less hp still could cast all his spells, and they could be reverted with curative magic. In 5e, this again has been further nerfed and padded to a mere nuisance, with characters being able to recover after just a night's sleep. I think a practical approach would be something like petrification, a permanent effect that can be undone with some spell that has a costly material component.